The Definitive Guide To

The Definitive Guide To
tm tm
Windows Application and Server Backup 2.0

HOME OF BACKUP 2.0
Don Jones
The Definitive Guide to Windows Application Server and Backup 2.0
Don Jones
IntroductiontoRealtimePublishers
by Don Jones, Series Editor
Forseveralyearsnow,Realtimehasproduceddozensanddozensofhighqualitybooks thatjusthappentobedeliveredinelectronicformatatnocosttoyou,thereader.Weve madethisuniquepublishingmodelworkthroughthegeneroussupportandcooperationof oursponsors,whoagreetobeareachbooksproductionexpensesforthebenefitofour readers. Althoughwevealwaysofferedourpublicationstoyouforfree,dontthinkforamoment thatqualityisanythinglessthanourtoppriority.Myjobistomakesurethatourbooksare asgoodasandinmostcasesbetterthananyprintedbookthatwouldcostyou$40or more.Ourelectronicpublishingmodeloffersseveraladvantagesoverprintedbooks:You receivechaptersliterallyasfastasourauthorsproducethem(hencetherealtimeaspect ofourmodel),andwecanupdatechapterstoreflectthelatestchangesintechnology. Iwanttopointoutthatourbooksarebynomeanspaidadvertisementsorwhitepapers. Wereanindependentpublishingcompany,andanimportantaspectofmyjobistomake surethatourauthorsarefreetovoicetheirexpertiseandopinionswithoutreservationor restriction.Wemaintaincompleteeditorialcontrolofourpublications,andImproudthat weveproducedsomanyqualitybooksoverthepastyears. Iwanttoextendaninvitationtovisitusathttp://nexus.realtimepublishers.com,especially ifyouvereceivedthispublicationfromafriendorcolleague.Wehaveawidevarietyof additionalbooksonarangeoftopics,andyouresuretofindsomethingthatsofinterestto youanditwontcostyouathing.WehopeyoullcontinuetocometoRealtimeforyour educationalneedsfarintothefuture. Untilthen,enjoy. DonJones
Don Jones
IntroductiontoRealtimePublishers.................................................................................................................i Chapter1:Introduction.........................................................................................................................................1 ThePhilosophyofBackup...............................................................................................................................2 WhyBackup1.0IsNoLongerEnough.......................................................................................................3 BackupWindows............................................................................................................................................3 NonContinuous..............................................................................................................................................5 JusttheDataNottheApplication.........................................................................................................5 DisasterRecoveryIsTooInflexible........................................................................................................6 Backup1.0:TheVerdict...............................................................................................................................7 BackupBasics........................................................................................................................................................7 WhyBackUp?...................................................................................................................................................7 WhatDoYouBackUp?.................................................................................................................................8 WhenDoYouBackUp?.............................................................................................................................10 WhatTypeofBackupWillYouUse?...................................................................................................11 . WhereDoYouStoreBackups?...............................................................................................................13 ShouldYouTestBackups?.......................................................................................................................13 AreYouKeepinganEyeonYourBackups?......................................................................................14 ApproachestoBackingUp............................................................................................................................16 FileBasedBackups.....................................................................................................................................16 ImageBackups..............................................................................................................................................17 ApplicationSpecificIdeas........................................................................................................................18 OurBackup2.0WishList..............................................................................................................................21 WhatsAhead......................................................................................................................................................22 Chapter2:12HorrorStoriesWeThoughtWeHadaBackup!......................................................24 . Corruption:NotJustforPolitics.................................................................................................................24 CrackberryWithdrawal.................................................................................................................................25 ExchangeFailure...............................................................................................................................................26 ii
Don Jones
MigratingtheClusterOrNot...................................................................................................................28 . VirtuallyExchange...........................................................................................................................................29 ForWantofaCheckBox...............................................................................................................................30 . WeightLossPlan...............................................................................................................................................32 So,640KBIsntReallyEnoughforEveryone?......................................................................................32 SQLServerSyndrome.....................................................................................................................................33 PatchProblem....................................................................................................................................................35 VirtualHotSpares............................................................................................................................................36 IsitaServeroraPhotocopier?...................................................................................................................37 HorrorStoriesandtheLessonsTheyTeachUs..................................................................................39 ComingUpNext.................................................................................................................................................40 Chapter3:WholeServerBackups.................................................................................................................41 TheTechnicalDetails......................................................................................................................................41 TraditionalTechniques.............................................................................................................................42 NativeSolutions................................................................................................................................................43 WindowsBackup/WindowsServerBackup.................................................................................43 . VolumeShadowCopy/PreviousVersions......................................................................................46 TraditionalNonNativeSolutions.............................................................................................................48 . ProblemsandChallenges..............................................................................................................................49 IntheOldDays...................................................................................................................................................50 BackupTechniques.....................................................................................................................................50 RestoreScenarios........................................................................................................................................50 DisasterRecovery........................................................................................................................................51 BackupManagement..................................................................................................................................51 RethinkingServerBackups:AWishList................................................................................................52 NewandBetterTechniques....................................................................................................................52 BetterRestoreScenarios..........................................................................................................................55 iii
Don Jones
BetterDisasterRecovery..........................................................................................................................55 EasierManagement....................................................................................................................................56 Greatfor............................................................................................................................................................57 DomainControllers.....................................................................................................................................57 InfrastructureServers...............................................................................................................................58 WebServers...................................................................................................................................................58 PublicKeyInfrastructure.........................................................................................................................59 ComingUpNext.............................................................................................................................................59 Chapter4:ExchangeServerBackups...........................................................................................................60 NativeSolutions................................................................................................................................................60 ProblemsandChallenges..............................................................................................................................63 ABitAboutHowExchangeServerWorks........................................................................................63 WhyExchangeServerBackupsCanbeTricky................................................................................64 IntheOldDays...................................................................................................................................................65 BackupTechniques.....................................................................................................................................65 RestoreScenarios........................................................................................................................................66 DisasterRecovery........................................................................................................................................69 BackupManagement..................................................................................................................................69 RethinkingServerBackups:AWishList................................................................................................70 NewandBetterTechniques....................................................................................................................71 BetterRestoreScenarios..........................................................................................................................72 BetterDisasterRecovery..........................................................................................................................73 EasierManagement....................................................................................................................................73 ExchangeSpecificConcerns........................................................................................................................74 CCR.....................................................................................................................................................................74 IndividualItemRecovery.........................................................................................................................74 DeDuplication..............................................................................................................................................75 iv
Don Jones
DataCorruption............................................................................................................................................76 SearchandeDiscovery.............................................................................................................................76 ComingUpNext.............................................................................................................................................77 Chapter5:SQLServerBackups.......................................................................................................................78 NativeSolutions................................................................................................................................................78 HowSQLServerWorks.............................................................................................................................79 HowSQLServerNativeBackupWorks..............................................................................................80 ProblemsandChallenges..............................................................................................................................81 IntheOldDays...................................................................................................................................................81 BackupTechniques.....................................................................................................................................82 RestoreScenarios........................................................................................................................................83 DisasterRecovery........................................................................................................................................85 BackupManagement..................................................................................................................................86 RethinkingServerBackups:AWishList................................................................................................86 NewandBetterTechniques....................................................................................................................86 BetterRestoreScenarios..........................................................................................................................87 BetterDisasterRecovery..........................................................................................................................88 EasierManagement....................................................................................................................................90 SQLServerSpecificConcerns.....................................................................................................................91 Sprawl...............................................................................................................................................................91 LogTruncation..............................................................................................................................................92 SingleObjectRecovery,Corruption,andOffLocationRestores............................................93 ComingUpNext.............................................................................................................................................94 Chapter6:SharePointServerBackups........................................................................................................95 NativeSolutions................................................................................................................................................95 Versioning.......................................................................................................................................................96 RecycleBin.....................................................................................................................................................96 v
Don Jones
SharePointDesignerandSTSADMImport/Export.......................................................................96 SQLServerBackup......................................................................................................................................97 CentralAdministration.............................................................................................................................98 ProblemsandChallenges..............................................................................................................................99 IntheOldDays.................................................................................................................................................101 BackupTechniques...................................................................................................................................101 RestoreScenarios......................................................................................................................................102 DisasterRecovery......................................................................................................................................104 BackupManagement................................................................................................................................104 RethinkingSharePointServerBackups:AWishList......................................................................105 NewandBetterTechniques..................................................................................................................105 BetterRestoreScenarios........................................................................................................................107 BetterDisasterRecovery........................................................................................................................111 EasierManagement..................................................................................................................................111 SharePointSpecificConcerns...................................................................................................................111 SingleObjectRecovery...........................................................................................................................111 ProtectingDependencies........................................................................................................................112 ComingUpNext...........................................................................................................................................112 Chapter7:VirtualizationServerBackups.................................................................................................113 NativeSolutions..............................................................................................................................................113 ProblemsandChallenges............................................................................................................................113 IntheOldDays.................................................................................................................................................116 BackupTechniques...................................................................................................................................116 RestoreScenarios......................................................................................................................................119 DisasterRecovery......................................................................................................................................120 BackupManagement................................................................................................................................121 RethinkingServerBackups:AWishList..............................................................................................121 vi
Don Jones
NewandBetterTechniques..................................................................................................................122 BetterRestoreScenarios........................................................................................................................124 BetterDisasterRecovery........................................................................................................................127 EasierManagement..................................................................................................................................128 VirtualizationSpecificConcerns..............................................................................................................129 Performance................................................................................................................................................129 . GranularRecovery....................................................................................................................................129 GuestsandHosts........................................................................................................................................129 ComingUpNext...........................................................................................................................................130 Chapter8:OtherConcernsandCapabilities............................................................................................131 DisasterRecoveryandBareMetalRecovery.....................................................................................131 RecoveryTimes..........................................................................................................................................134 RecoveryLocations...................................................................................................................................135 WhyWaitforaDisaster?........................................................................................................................136 HighAvailability,InstantRecovery,andReplication......................................................................136 Hardware......................................................................................................................................................137 Clustering......................................................................................................................................................137 ReplicationforRedundancy..................................................................................................................138 ReplicationforOffsiteRecovery.........................................................................................................139 DataRetentionHereCometheLawyers...........................................................................................141 ComplianceandSecurityHereCometheAuditors......................................................................142 IntegratingBackup2.0withBackup1.0..............................................................................................145 BackupsasaFormofMigration...............................................................................................................146 ComingUpNext...........................................................................................................................................147 Chapter9:KeepingYourBackupsStorageArchitecture................................................................148 WhereBackupsGoStorageinBackup1.0.......................................................................................148 WhatBackedUpDataLooksLike......................................................................................................148 vii
Don Jones
TapeDrives...................................................................................................................................................150 DiskStorage.................................................................................................................................................151 RethinkingStorageforBackup2.0.........................................................................................................152 WhatBackedUpDataShouldLookLike.........................................................................................152 PhysicalStorage:Tapes,SANs,NAS,theCloud,andMore.......................................................153 RethinkingHierarchicalBackupStorage.........................................................................................154 SavingSpace.....................................................................................................................................................161 DeDuplicationofData............................................................................................................................161 CompressionofData................................................................................................................................163 WorkingwithBackedUpData.................................................................................................................164 ComingUpNext...........................................................................................................................................164 Chapter10:WhenEverythingFails:WhatsYourDisasterRecoveryPlan?.............................165 . OldSchoolDisasterRecoveryPlanning...............................................................................................165 BareMetalRecoveryandServerRebuilds.....................................................................................167 OffSiteRecovery.......................................................................................................................................171 Backup2.0sDisasterRecoveryApproach..........................................................................................172 BareMetalDisasterRecovery.............................................................................................................172 . VirtualizationasaDisasterRecoveryStrategy............................................................................173 . . RestoringApplicationstoSomeplaceElse.....................................................................................173 ColdRecovery.....................................................................................................................................174 WarmRecovery.................................................................................................................................175 HotRecovery.......................................................................................................................................177 DisasterRecoveryasaMigrationStrategy..........................................................................................179 ComingUpNext...........................................................................................................................................180 Chapter11:RedesigningBackup:IsItReallyWorthIt?....................................................................181 . LetsReview:WhatisBackup2.0?..........................................................................................................181 ABackup2.0ShoppingList........................................................................................................................187 viii
Don Jones
WhatsInvolvedinaRedesign?................................................................................................................188 CalculatingtheRealCostofYourGrandfathersBackupStrategy............................................191 TechnicalandBusinessAdvantagesofBackup2.0.........................................................................193 WholeServers.............................................................................................................................................193 Exchange.......................................................................................................................................................194 . SQLServer.....................................................................................................................................................194 SharePoint.....................................................................................................................................................194 Virtualization...............................................................................................................................................194 IsaRedesignThatExpensive?..................................................................................................................194 TacklingtheOfficePolitics.........................................................................................................................195 Conclusion:ThisIsNotYourFathersBackupPlan.........................................................................196 ComingUpNext...........................................................................................................................................197 Chapter12:TalesfromtheTrenches:MyLifewithBackup2.0.....................................................198 Backup1.0FiredOurAdministratorandBackup2.0PromotedMe.....................................198 TheCureforCrackberryWithdrawal....................................................................................................201 ExchangeFailureSuccess...........................................................................................................................202 . VirtuallyOnline...............................................................................................................................................203 Wait,WeGotThat?........................................................................................................................................206 StoptheSpindles!OrWhatever...........................................................................................................207 . EnoughIsNeverEnoughUntilItIs......................................................................................................207 SQLServerSolution.......................................................................................................................................208 Hotfixes,GoneCold........................................................................................................................................210 DeDuplicationDiscovery...........................................................................................................................211 ItsaBackupandanArchive!....................................................................................................................213 VirtualNirvana................................................................................................................................................214 Conclusion.........................................................................................................................................................215
ix
Don Jones
Copyright Statement
2010 Realtime Publishers. All rights reserved. This site contains materials that have been created, developed, or commissioned by, and published with the permission of, Realtime Publishers (the Materials) and this site and any such Materials are protected by international copyright and trademark laws. THE MATERIALS ARE PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. The Materials are subject to change without notice and do not represent a commitment on the part of Realtime Publishers or its web site sponsors. In no event shall Realtime Publishers or its web site sponsors be held liable for technical or editorial errors or omissions contained in the Materials, including without limitation, for any direct, indirect, incidental, special, exemplary or consequential damages whatsoever resulting from the use of any information contained in the Materials. The Materials (including but not limited to the text, images, audio, and/or video) may not be copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any way, in whole or in part, except that one copy may be downloaded for your personal, noncommercial use on a single computer. In connection with such use, you may not modify or obscure any copyright or other proprietary notice. The Materials may contain trademarks, services marks and logos that are the property of third parties. You are not permitted to use these trademarks, services marks or logos without prior written consent of such third parties. Realtime Publishers and the Realtime Publishers logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. If you have any questions about these terms, or if you would like information about licensing materials from Realtime Publishers, please contact us via e-mail at info@realtimepublishers.com.
Don Jones
Chapter1:Introduction
Thefirstbackuptechnicallywasaround1951,whenthefirstgenerationofdigital computingappearedintheformofUNIVACI.Thebackups,suchastheywere,werethe punchcardsusedtofeedinstructionstothemassivemachine.Oncecomputersbeganto usemoreflexibleformsofstorage,reeltoreelmagnetictapebegantoreplacepunchcards. In1965,IBMintroducedthefirstcomputerharddrives,althoughthroughthe1970s,these devicesremainedimpracticallyexpensivetouseforbackups.Floppydiskscameintousein 1969an8inchmonsterstoringjust80kilobytesofdata.Recordablecompactdisks becameavailableintheearly1990s,andflashdrivesbecamecommonintheearlypartof the21stcentury.Shockingly,magnetictapethesecondoldestformofbackupstorageis stillinusetoday.Figure1.1showsatimelineofdatabackupstorage(excerptedfrom www.backuphistory.com),andyoucanseethattapeisstillaliveandwellandhasbeen foralmost50years.
Figure1.1:Backupstoragetimeline. Theresaninterestingparalleltobedrawnhere:Despitenumeroustechnicaladvancesin storage,wecontinuetorelyononeoftheoldestmediumstostorebackups.Thesame appliestoourbackuptechniquesandprocedures:Despiteadvancesinhowweperform backups,wetendtostillusethesamedecadesoldtechniques,albeitwrappedupinpretty newtools.
Don Jones
Throughoutcomputinghistory,backupshavebeenpractical,simpleprocedures:Copya bunchofdatafromoneplacetoanother.Complexitiesarisewithalwaysondatalikethe databasesusedbyExchangeServerandSQLServer,andvarioustechniqueshavebeen developedtoaccessthatformofinusedata;however,backupshaveultimatelyalways beenaboutafairlysimple,straightforwardcopy.Evenmagnetictapemuchmore advancedthaninthe1960s,ofcourseisstillaprimaryformofstorageformany organizationsbackups. IcallitBackup1.0essentiallythesamewayweveallbeenmakingbackupssincethe beginningoftime,withtheonlymajorchangesbeingthestoragemediumweuse.Although manybrightengineershavecomeupwithclevervariationsontheBackup1.0theme,its stillbasicallythesame.AndIsayitsnolongerenough.Weneedtorethinkwhywedo backups,andinventBackup2.0anewwaytobackupourdatathatmeetstodays businessneeds.Surprisingly,manyofthetechniquesandtechnologiesthatsupportBackup 2.0alreadyexistwejustneedtoidentifythem,bringthemtogether,andstartusingthem.
ThePhilosophyofBackup
Letsstartwiththequestion,Whydowebackup? Isupposethefirstanswerthatcomestomindissimpleenough:Sothatwedontloseany data.Butthatsnotactuallyanaccurateanswer.Personally,Idontbackthingsupjustso theywillneverbelost;IbackthemupsothatIcancontinueusingthem.Thatsasubtle difference,butanimportantone.Ifyoureonlyconcernedaboutneverlosingdata,Backup 1.0isprobablysufficient:Copyyourdatatoalongtermstoragemediumprobably magnetictapeandstickitinavaultsomewhere.Youcouldcallitarchivingandbemore accurate,really.Butmostorganizationsarentasconcernedaboutarchivingastheyare aboutmakingsurethatdataremainsavailable,whichmeansyounotonlyneedabackup butalsoameansofrestoringthedatatousability.Sotherealanswer,formost organizations,ismorecomplicated:Sothatourdataremainsavailabletousallthetime. ThatswhereBackup1.0thebackuptechniquesandtechnologiesweveallusedforever andarestillnativetooperatingsystems(OSs)likeMicrosoftWindowscanreallyfail. Makingacopyofdataisonething;puttingthatcopybackintoproductionuseisoftentoo slow,toocomplicated,andtoomonolithic.AndthatswhereBackup1.0failsus.Aswestart consideringBackup2.0,andwhatweneedittodo,weneedtobearinmindourreal purposeforbackingup.Ultimately,wedontcareaboutthebackinguppartverymuchat allwecareabouttherestorepartalotmore.
Don Jones
WhyBackup1.0IsNoLongerEnough
Ourdecadesoldbackuptechniquesarenotsufficientanymore.Theymaybegreatfor creatingbackupsalthoughinmanycases,theyarentevengoodforthatbuttheydonot excelatbringingtherightdatabackintoproductionasquicklyaspossible.Despite advancesinspecializedagents,compressednetworktransmissions,andsoforth,werestill justmakingacopyofthedata,andthatdoesntalwayslenditselfwelltorestoringthedata. Why?
BackupWindows
Oneproblemistheneedforbackupwindows,periodsoftimeinwhichourdataisntbeing usedveryheavily,sothatwecangrabaconsistentcopyofit.Consistencyiscriticalfor backups:Allthedatainanygivencopyneedstobeinternallyconsistent.Wecantbackup halfadatabasenowandtheotherhalflaterbecausethetwohalveswontmatch.Asour datastoresgrowlargerandlarger,however,gettingafullbackupbecomesmoreandmore difficult. MicrosoftsTerraServer,whichstoresandprovidesaccesstosatellitephotographsforthe entireUnitedStates,hasadatastoreinexcessof1terabyte,andevenwithfairlyadvanced backuphardware,itstilltakesalmost8hourstobackitallup.Thatsatotaldata throughputof137GBperhourbutifthatdatawereinconstantuse,itwouldbecomeless practicaltomakeacompletecopyonaregularbasis. Asaworkaround,wecommonlyusedifferentialorincrementalbackups.Theseallowusto grabalotlessdataallatonce,makingiteasiertomakeourbackupcopy.Theproblemis thattheseignoretherealreasonwemadethebackupinthefirstplacetoenableusto restorethatdata.ConsideracommonbackupapproachthatusesSQLServersnative backupcapabilities: Sunday,fullbackup Everyweekdayevening,adifferentialbackup Everyhourduringthedayfrom8amto5pm,alogbackup
Thoseweeknightdifferentialsgrabeverythingthathaschangedsincethelastfullbackup (asopposedtoanincremental,whichgrabseverythingthathaschangedsincethelastfull orthelastincremental).IfsomethinggoeswrongonFridayat4:05pm,theresalotofdata torestore: LastSundaysfullbackupwhichwillbefairlylarge Thursdaysdifferentialwhichwillalsohavegrownquitelarge ThelogbackupsfromFridayat8am,9am,10am,11am,noon,1pm,2pm,3pm,and 4pmthatsninefilesinall,althougheachwillbefairlysmall
Figure1.2illustratesthesedifferentbackuptypes.
Don Jones
Data Files
Full database backup
Log Files
Differential backup
Transaction log backup
Figure1.2:SQLServerbackuptypes. Thatsatonofworkandatonofwaitingwhiletapesandharddrivesspin,restoringall thatdata.Sure,wewonthavelostmuchjust5minutesworthofworkbutonalarge database(say,aterabyteorso),youcouldeasilybewaitingfor16hoursormore. Andthatsifthebackupsallwork.Tapedrivesandevenharddrivesarenotimmuneto corruption,errors,andfailures,andoneofthemostcommonstoriesintheworldisthe administratorwhorealizedthatthebackuptapeswerenogoodandrealizeditwhile tryingtorestorefromoneofthem.Weallknowthatweshouldtestourbackups,but honestly,doyoudoit?Outofmorethan300consultingclientsIveworkedwithinthepast 10orsoyears,oneofthemhadaregularlyscheduledplantodotestrestores.One.Less thanonepercent.Why? Well,theonecustomerwhodidregularlyscheduledtestrestoreshadadedicated administratorwhodidalmostnothingelse.Afulltestrestoreoftheirenvironment,using Backup1.0styletechniquesandtechnologies,wouldrequiretheentireITteamabouta weektoperform.Thatoneadministratorcouldtestrestorevarioussystems,oneatatime, overthecourseofamonthandthenstartover.Ithinkthatprettymuchanswersthe questionaboutwhysofewpeopletesttheirbackups.Itsasimple:Toomuchdataforfull backupsresultsinworkaroundssuchasdifferentialsandincrementalsthatcontributeto lengthyrestoretimes,whichiswhyweneverbothertotestandwhy,whentherubber hitstheroadandweneedthosebackups,nobodyishappyabouthowlongittakestograb theneededdata.
Don Jones
NonContinuous
Backup1.0hasanothermajorweakpoint:Itsalwaysapointintime.Asnapshot.Non continuous,inotherwords.ConsideroneapproachtobackingupActiveDirectory(AD), whichIseealotofmycustomersusing: FullbackupofeverydomaincontrollersSystemStateontheweekends.Thisisa quick,fairlysmallbackupevenanexceptionallylargedomaincanbebackedupin afewminutes. Ononeortwodomaincontrollers,atwicedailybackupoftheSystemState.Again, thisoperationisquick.
Theproblemisthatyoualwaysstandtoloseahalfdaysworthofworkbecauseyoure onlytakingsnapshotstwiceaday.Whatifyoujustimportedacoupleofhundrednew usersintothedomain,addedthemtogroups,andassignedthemfilepermissionson variousfileservers?Losingthatworknotonlymeansyouhavetostartover,italsomeans youvegotorphanSecurityIdentifiers(SIDs)floatingaroundonallthosefilepermission AccessControlLists(ACLs).Youllnotonlyhavetorepeatyourwork,youllalsohaveto cleanupallthoseACLs. BusinessestendtodesignbackupstrategiesaroundtheconceptofHowmuchdataarewe willingtolose,andhowmuchworkarewewillingtorepeat?Theresoftenalotless considerationaboutHowquicklydowewanttorestorethedata?andmuchtomy irritation,almostnobodythinkstoanswer,Wedontwanttoloseanydataorrepeatany work!Backup1.0hasconditionedustoacceptdatalossandrepeatedworkasinevitable, andsowedesignbackupschemesthattradeoffbetweentheinevitablelossofdataandthe amountofbackupresourceswewanttodevote.Frankly,theattitudethatIhavetoaccept datalossandrepeatedworkisnonsense.Icantbelievethat,adecadeintothe21stcentury, wereallsocomplacentaboutthatattitude.
JusttheDataNottheApplication
AnotherproblemwithBackup1.0isthatweoftentendtojustbackupdatadatabases, files,SystemState,orwhateverwerarelybackuptheapplicationsthatusethedata.I tookabriefsurveyonmyblogatConcentratedTech.com,andabout95%ofthe respondentssaidtheydontbackupapplicationsbecausetheykeeptheoriginal installationmediasothattheycanalwaysreinstalltheapplicationifnecessary. Really?Letsthinkaboutthat:MicrosoftExchangeServertakesabout45minutestoan hourtoinstallproperly.Thenyouhavetoapplythelatestservicepackanotherhalfhour orsoandanypatchesreleasedsincetheservicepackcallthatanother20to30minutes. Thenyoucanstartrestoringthedata,whichmaytakeanotherfewhours.Ifyoure rebuildinganentireserver,ofcourse,youllhavetostartwithreinstallingWindowsitself, anditsservicepackandpatches,whichwilladdanother2or3hourstotheprocess.The total?Maybeafullworkday.Andforsomereason,peoplefindthatacceptablebecause Backup1.0isallaboutarchiving,really,notrestoring.
Don Jones
Onevalidcounterargumentisthatmostrestorationsareforjustthedata,orevenapartof thedata(likeasingledatabasetable,orasingleemailmessage),andarentafullon disasterrecoveryrebuild.Well,okaybutdoesthatmeanitsstillacceptableforafullon disasterrecoveryrebuildtotakeafullday?Typicallynot,andthatswhysome organizationswillimagetheirservers,usingsoftwarethattakesasnapshotoftheentire harddriveandoftencompressesittoasmallersize.Usedforyearsasadeployment technique,itworkswellforbackingupanentireserver.Forbackinguptheserverbutoften notrestoringit.Snapshot,orpointintime,imagestaketimetoproduce,andtheserver mayevenhavetobeshutdowntomakeanimagemeaningyoullneedafrequent maintenancewindow.Atraditionalsnapshotimagewontcontainthelatestdata,soeven afterrestoringtheimage,youstillhavetorelyontraditionalbackupstogetyourmost recentdataback.Itjustamazesmethatweaccepttheselimitations.
DisasterRecoveryIsTooInflexible
Closelyrelatedtothepreviousdiscussionisthefactthatpeopledobackupsfortwo differentreasons.Reasonone,whichIthinkisprobablymorecommonlycitedasareason tobackupistorestoresmallpiecesofdata.Youvedoubtlessdonethis:Restoredasingle filethatsomeonedeletedbyaccident,oranemailmessage,oradatabasetable.Nearly everyonehasdealtwiththis,anditisntdifficulttosellthisreasontomanagementwhen acquiringbackuptechnologies. ThesecondreasonisforwhatIcalledfullondisasterrecoveryintheprevioussection. Thisiswhenanentireserveror,goodnesshelpyou,anentiredatacenterislost,and hastoberestoredonsiteoratadifferentlocation.Unfortunately,thislevelofdisasteris actuallyquiterare,anditsatoughsellformanagementiftheorganizationhasnt encounteredthistypeofdisasterinthepast. TheultimateproblemisthatBackup1.0technologieslendthemselvestooneortheother scenariosbutnotusuallytoboth.Inotherwords,ifyouhaveaproductthatdoesgreat baremetalrecovery,itmaynotdosingleitemrecoveryaswell.Someproducts compromiseanddoanokayjobatbothbackinguptheentireservertoenablebaremetal recovery(andoftenprovidingbootableCDsorothertechniquessothatyoucaninitiatea baremetalrecovery),andthenkeepingaseparateindexofeverybackeduppieceofdata tomakesingleitemrecoveryeasier.Frankly,Ivenotusedmanysolutionsthatdoagreat jobatbothtasksandthefactthattheyreallessentiallysnapshotbasedstillmakesthem prettylimited.Ineverwanttohavetoagreethatlosingacertainamountofworkis acceptable.
Don Jones
Backup1.0:TheVerdict
Ifyourejustarchivingdata,Backup1.0isprettyawesome.Itstartstofail,though,when youneedtorestorethatdata,andwanttodosoinanefficientmannerthatenablesboth singleitemandfullondisasterrecoveryrestoration.Thesnapshotorientednatureof Backup1.0meansyourealwaysatriskoflosingsomework,andthatsnapshotoriented naturealsoimposesrigidrequirementsformaintenanceandbackupwindowswindows thatmightnotalwaysbeinthebusinessbestinterests. Soletsrethinkbackup.Iwanttogobacktobasicsandreallydefinewhatbackupsshould do.ConsiderthisdefinitionawishlistforBackup2.0.
BackupBasics
IhavenodoubtthatyoureaprettyexperiencedadministratororITmanager,andyou mightnotthinkthatbackupbasicsisaparticularlyenticingsection.Bearwithme.Illtry tokeepeachofthefollowingsectionssuccinct,butIreallywanttostepbackfromthe existingtechnologiesandtechniquestofocusonwhatpeopleandbusinessesnot softwarevendorswanttheirbackupprogramstodo.Wevebeendoingbackupsmoreor lessthesamewayforsolongthatIthinkitsbeneficialtojustforgeteverythingweve learnedanddoneandstartoverwithoutanyassumptionsorpreconceptions.
WhyBackUp?
Wevecoveredthistopicprettywell,butletmestateitclearlysothattheresnoconfusion: Backupsshouldpreventusfromlosinganydataorlosingany work,andensurethatwealwayshaveaccesstoourdatawith aslittledowntimeaspossible. Thatstatementimpliesafewbasicbusinesscapabilities: Whenaproblemoccurs,wewanttoexperienceaslittledatalossaspossible Weneedtobeabletorecoverdataasquicklyaspossible Weplaceequalimportanceonrecoveringasinglebitofdataaswedoindealing withacompletedisaster
Don Jones
ThestatementalsomeansafewthingstraditionallyassociatedwithBackup1.0probably arentacceptable: Snapshotsthatgrabonlyacertainpointintimeimagearelessdesirable Anysystemthatisweightedtowarddisasterrecoveryortowardsingleitem recoveryislessdesirableweneedbothcapabilities Anysystemthatrequireslengthy,multisteprestoreprocessesislessdesirable Backupsthatdonotlendthemselvestosomeformofphysicallyprotectedstorage arelessdesirable Backupsthatrequirehoursandhourstocompletewillrequirehoursandhoursto restorebothofwhicharelessdesirable
Sogivenwhywebackup,wecantakeafreshlookatwhatwebackup.
WhatDoYouBackUp?
EventherelativelyprimitivebackupsoftwareincludedwithWindowsServer2003(and priorversionsofWindows)understoodthatyoudontalwaysneedtobackupeverything everytimeyourunabackup.Figure1.3showshowthatutilityallowedyoutoselectthe itemsyouwantedtobackuporrestoreauserinterface(UI)duplicatedinsomeformby mostcommonlyusedbackupsoftware.
Don Jones
Figure1.3:SelectingwhattobackuporrestoreinWindowsBackup. Sowhatdoyoubackup?Onanygivenserver,youhavemanychoices: Backuptheentireservereveryfileoneverydisk Backupjustdatasharedfiles,applicationdatabases,andsoforth Backupapplicationsandtheirdataincludingtheapplicationsexecutablefilesand settings BackuptheOSanditssettingsbutnotanyapplicationfilesoranydata
Don Jones
Thepermutationsarepracticallylimitless.Ifyouresimplyafterarchivingcreating backups,thatisthenanythingthatgrabsthedataisfine.Infact,justgrabbingthedatais probablyallyouneedtodoifallyouwanttodoiscreateapointintimesnapshotfor archivalpurposes.Sure,youcouldbackuptheentireserverandifyourgoalistobeable tohandleafullondisasterorrecoverindividualitems,thenbackinguptheentireserver wouldprovidebothcapabilities.Butbackinguptheentireserverwouldprobablytakealot longer.Itmightnotevenbeentirelypossiblebecausetherewillalwaysbesomeopenfiles, runningexecutables,andotheritemsthatBackup1.0stylebackuptechniquescantgetto. Somaybebackinguptheentireserverisntreallypractical.Afterall,wecanalways reinstalltheOSandanyapplicationsfromtheiroriginalmedia,right? Wait,asecondletsgobacktowhywerebackingup: Backupsshouldpreventusfromlosinganydataorlosingany work,andensurethatwealwayshaveaccesstoourdatawith aslittledowntimeaspossible. Okay,thisstatementclearlyindicatesthatweneedtograbthedata,butthatlastphrase haveaccesstoourdatawithaslittledowntimeaspossibleaddssomethingimportant. Inordertoaccessourdata,weneedtheOSandassociatedapplicationstobeupand running!AdatabackupisuselesswithoutanapplicationandOStorestoreitto.Ivealready explainedwhyrebuildingaserverusingtheinstallationmediaissoslowyouhaveto performlengthyinstalls,theninstallservicepacksandpatches,andthenrestoreyourdata. Thistellsme,then,thatourbackupmustbeoftheentireserver.Afterall,Imnotjusthere toarchivemydataIalsoneedtobeabletorestoreitquickly,andgetaccesstoitquickly, andthatmeansIneedtorestoretheOSandanyapplicationsquickly.Soregardlessof practicalityfornowImgoingtosaythatbackinguptheentireserveristheonlywayto go.Ijustneedtofigureouthowtodoitquickly.
WhenDoYouBackUp?
HowoftenwillIbemakingbackups?UnderBackup1.0,thiswasarealquestion.Often,you mighttakeafullbackupduringaneveningorweekendmaintenancewindow,thengrab smallerbackupsmorefrequently.WhenIstartedoutinIT,IwasanAS/400operator. Everyevening,wemadetapebackupsofourmostimportantdatafilesfromtheAS/400;on weekends,weranafullbackupoftheentiresystem.Backingupthedatafilestookafew hours,andwehadtodoitintheeveningafterprettymuchalltheworkforthedaywas finishedbecausethedatawasunavailablewhilethebackupswererunning(infact,we prettymuchkickedeveryoneoffthesystemwhilebackupswerebeingdone).Theweekend fullbackupscouldtakeafullday,andeveryonehadtobeofflinethen. ButmaintenancewindowsareaBackup1.0concept,soletsdisregardthem.Infact,lets reviewourwholereasonforbeinghereonemoretime: Backupsshouldpreventusfromlosinganydataorlosingany work,andensurethatwealwayshaveaccesstoourdatawith aslittledowntimeaspossible.
10
Don Jones
Rightthereismyanswer:fromlosinganydataorlosinganywork.Thattellsmethe Backup1.0methodofpointintimesnapshotsisuselessbecausenomatterhowoftenIm makingincrementalordifferentialbackups,Imstillgoingtobeatriskforlosingsomework ordata,andthatsnotacceptable. Soifyouask,Whenwillyoumakebackups?IhavetoanswerAlways.Literally continuousbackups.Infact,theindustryalreadyhasatermforit:continuousdata protection.Althoughspecifictechniquesvary,youcanthinkofthisveryroughlyas beingsimilartoRAID1drivemirroring.InaRAID1array,asFigure1.4shows,thedrive controllerwritesblocksofdatatotwo(ormore)harddrivesatonce.Disk2isacomplete, blocklevelbackupofDisk1.RestoringisfastsimplyswapthetwoifDisk1goesbellyup.
Figure1.4:RAID1diskmirroring. Ofcourse,RAID1isgoodforacertaintypeofscenariobutitisntpracticalforallsituations, anditdoesntmeetallourbackuprequirements.Forone,serverdisksarestillpretty expensive.Inaserverusingalargenumberofdisks,mirroringeveryoneofthemisnt practical.SomeorganizationswillsetupaRAID5arrayandthensetupasecondarrayto mirrorthefirstbutthatcanbeincrediblyexpensive.Afurtherproblemisthatthebackup diskscoexistwiththeprimarydisks,sothebackupdisksarestillatriskfordamagedueto fire,flood,andotherphysicalthreats.Further,amirrorisonlyabackupforthecurrent conditionoftheprimarydisk:Youcantrollbacktoapreviouspointintime.Somirroring aloneisagreattoolforcertainsituations,butitisntacompletebackupsolution.Whatitis, though,isagoodideaforhowtomakecontinuousbackups.Wejustneedtoleveragethe techniqueabitdifferently.
WhatTypeofBackupWillYouUse?
Fullbackup?Differential?Incremental?IhavetosaynoneofthesebecausethoseBackup 1.0termsareassociatedwithpointintimesnapshots,andwerenotgoingtousethose. Instead,weregoingtouseaBackup2.0technique,borrowingfromtheRAID1conceptof blocklevelmirroring.Partofcontinuousdataprotection,wellcallthisblocklevelbackups. Figure1.5showshowitmightwork.
11
Don Jones
Figure1.5:Agentbasedblockbackups. Here,asoftwareagentofsomekindrunsontheserver.Asblocksarechangedondisk,this agenttransmitsthoseblockstoabackupserver,whichstoresthem.Thatservermight storeblocksformanydifferentservers.Theagentwouldlikelytapintoaverylowlevelof theOStoaccomplishthis.Thebenefits: Wecanhavenearlyrealtimebackupsofallchanges,astheyhappen Wegettheentireserver,notjustthedata Wecanstoremorethanjustthecurrentblocks;infact,wecanstorechangesasfar backaswewant,meaningwecanrestoreanygivenfilewhichconsistsofmultiple diskblockstoanypointintimewewant Wecanrestoretheentireserverbysimplywritingallthelatestbackedupblocksto aservereithertheoriginalserverorareplacement Intheeventofcorruptedblocks,wemightstillbebackingthoseupbutwevealso gotolder,noncorruptversionsofthosesameblocks,sowecanpotentiallyrepair filesthathavebecomecorruptedtotheirmostrecent,noncorruptedpointintime
Thisispowerfulmagic,butitsareality:Todaysmarketincludessolutionsthatfollowthis technique.ItdeliversBackup2.0:continuousbackupsthataredesignedforrestores,notfor archiving.
12
Don Jones
WhereDoYouStoreBackups?
Doyouneedoffsitestorageofyourbackups?Probablyyes.In1996,ParisbasedCredit Lyonnaishadafireintheirheadquarters.Administratorsranintotheburningbuildingto rescuebackuptapesbecausenothingwasstoredoffsite.Letmewritethatagain:Ranintoa burningbuilding.Folks,fireisaconstantpossibility,asisthepossibilityofdamagefrom floods(badplumbinganyone?)andotherdisasters.Ifthedataisworthbackingup,its worthkeepingcopiessomewhereelse.Attheveryleast,havesomesortofonsitestorage thatsdisasterproofawaterprooffiresafe,forexample.Doesthatmeanyouhavetouse magnetictape?No,butyouprobablywill,simplybecauseitsrelativelyinexpensive,fairly durable,andeasytoworkwith.Youlllikelyendupusingtapeinconjunctionwith somethingelse,infact,withtapesbeingthesortoflastresortplacetorestorefrom.The pointisthis:Dontassumethatamajordisasterwillnotstrikeyou.Pastperformanceis noguaranteeoffutureresults;justbecauseyouveneverbeenhitbyadisasterbefore doesntmeanyouwontbehitbyoneeventually.Thatswhypeoplebuyinsurancepolicies, andbackupsarebasicallyaformofinsurance. IntermsofBackup2.0,wemightcombineourblockbasedbackupswithsometapebased storage.Ourbackupserver,forexample,mightperiodicallydumpallitsbackedupblocks toatapearray,allowingustocarryasnapshotoffsiteforarchivalpurposesandtoprotect againstatotaldisastertoourdatacenter.
ShouldYouTestBackups?
Thisisatrickquestion.Theansweris,Ofcourse.AsIvealreadyexplained,though,few folksactuallydo.Why?Well,therearereallyafewreasons,manyofwhicharerelatedto ourBackup1.0mindset. First,asImentionedearlier,isthetimecommitment.Spendinghoursdoingatestrestore isntinmostfolksbudgetsthesedays.Ofcourse,ablockbasedrestorecanactuallybe donemorequickly:Yourestreamingtherestorefromdisksoverahighspeednetwork,not readingthemeversoslowlyfromatapedrive. Second,therestheavailabilityofhardware.Now,ifyouretestingsingleitemrecovery, mostbackupsolutionswillallowyoutorestorefilestoanylocationyouwant,soyoujust needasmallspotonanexistingfileserver.Butyoushouldalsobetestingfullondisaster recovery,whereyourestoreaserverthatwascompletelylost(say,tofire)toadifferent pieceofhardware.TheproblemisthatmanyBackup1.0stylesolutionsrequireyouto restoretoidenticalhardware,meaningyouhavetohavealotofspareserversaround.Not gonnahappen.Agoodsolutionwillletyourestoretodissimilarhardware,whichisactually morepracticalfromadisasterrecoveryperspective;anidealsolutionwillletyourestoreto avirtualizedserver,whichisabsolutelyperfect.Soyes:Testyourbackups.Regularly. Performbothsingleitemrestoresandthetypeofbaremetalrestoreyoudassociatewitha totaldisaster,ideallyutilizingmodernvirtualizationtechnologiestoeliminateorreduce theneedforextratestrestorehardware.
13
Don Jones
Virtualization:MorethanJustConsolidation AsIlldiscussinlaterchapters,virtualizationisanintegralpartoftheBackup 2.0mentality.InBackup1.0,afullsitedisastermightmeanretreatingtoa specialpurpose,leasedoffsiterecoveryfacilityandstartingalengthyrestore processusingyourmostrecentoffsitebackuptapes. InBackup2.0,thatfacilitymightliveontheInternetandconsistofoneor morevirtualizationhosts,eachrunningadozenorsovirtualservers.You streamyourlatestbackupsoverthewiretothevirtualservers,performinga barevirtualmetalrecoveryratherthanrecoveringtoactual,physical machines.Thisapproachmakesitmorepracticaltorecoverasetofservers, makesthatrecoveryfasterandcheaper(noleasedfacilities),andmakesit morepracticaltoconductoccasionaltestrunsofacompletedisaster scenario.
AreYouKeepinganEyeonYourBackups?
Doyoumonitoryourbackups?Otherthanjustcheckinganerrorlog,Imean?Youshould. Infact,checkingerrorlogsasidefrombeingincrediblyboringisthekindofoldschool Backup1.0mentalityImtryingtochangeinthisbook.Amodern,Backup2.0stylesolution shouldalertyoutoproblems,andideallymightevenintegratewithanexistingmonitoring solution,suchasMicrosoftsSystemCenterEssentialsorSystemCenterOperations Manager. Whatmightthosealertsactuallyinformyouof?Primarily,problemswiththebackups themselves,suchascorruption.Nothingsworsethansuddenlyfindingyouneedyour backupsandthenrealizingthattheyrenogoodduetocorruption;youshouldbeinformed ofcorruptionassoonasitoccurs.AgoodBackup2.0stylesolutionmightinformyouvia email,mightdropsomethinginaWindowseventlog(whichSystemCenterOperations Managercouldthenpickupandraisetoyourattention),orperformsomesimilarstyleof notification. Figure1.6showshowSystemCenterOperationsManagercanbeusedtoconfigureanalert foragiventypeofeventlogentrysuchasaneventlogentrycreatedbyyourbackup solution,alertingyoutocorruption.NotethatSystemCenterEssentialsworkssimilarlyfor thistypeofalert.
14
Don Jones
Figure1.6:CreatinganeventlogentryalertinSCOM. AreallywithitbackupsolutionmightevencomecompletewithaSystemCenter ManagementPack.AManagementPackisbasicallyapreconfiguredsetofrulesfor monitoringandalertingonaspecificapplication;thepacktellsSystemCenterOperations Managerwhattolookfor(suchasspecificeventlogIDs)andwhattodo(suchassending emailalerts).ButifyourbackupsolutiondoesntcomewithaManagementPack,atleast makesurethatithasitsownbuiltinfeaturesforprovidingthesetypesofnotifications.
15
Don Jones
ApproachestoBackingUp
Havingadvocatedforblocklevelbackups,Iwanttotakeastepbackandbrieflyreviewthe entiregamutofbackuppossibilities.Althoughsomeofthesedontmeetmyrequirements foragoodbackupandrecoverysystem,theynonethelessoffersomebusinessvaluethat youshouldbeawareof.
FileBasedBackups
Filebasedbackupsaretheoldesttypeofbackup,andprobablystillthemostcommon.It involvessimplytakingasnapshotapointintimecopyofoneormorefiles.Thesetypes ofbackupsmayhavedifficultyworkingwithopenfiles,andtheydontcaptureeverychange madetoafiletheyjustgrabacopyofitataspecifictime. Buttheresvaluehere.Forexample,WindowsbuiltinVolumeShadowCopyfeatureis essentiallyanonserverfilebasedbackup,grabbingcopiesoffilesasuserschangethem andstoringtheminalocalcacheontheserver.Userscanaccessthesecachedfieversions ontheirown,usingWindowsExplorersPreviousVersionsfeature,asFigure1.7shows.
Figure1.7:Accessingpreviousversions. 16
Don Jones
Thekeyphrasehereisontheirown.Unlikemanydatacenterclassbackupsolutions, VolumeShadowCopy/PreviousVersionsisdesignedforuserselfservice.Properlyused (meaningyouruserswillneedabitoftraining),thisfeaturecanhelppreventcallstothe Helpdeskandoverheadforyouwhenusersneedtorollbackafiletoasomewhatolder version.Infact,Iveseenthisfeatureagain,withabitofendusertrainingreduce singlefilerecoveryHelpdeskcallsby90%inseveralofmyclients.Thoseorganizationstell methattheaveragecostforcompletingafilerecoveryrequestisabout$75,andtheones thatkeepreallygoodrecordsaverageaboutfourcallsaweek.Thatsasavingsofmorethan $15,000allforfree,sincethefeatureisbuiltintoWindows(well,youdohavetospenda bitextraforthediskspaceneededtostorethecache). InaBackup2.0world,thereshouldberoomforcomplementarysolutions.Previous Versionsmeetsaspecificneed:userselfserviceforindividualfilerollback.Whateverdata centerbackupsolutionyouselectshouldntinterferewithcomplementarysolutions,andin fact,shouldembracethem.WherefilebasedbackupstendtofallshortasIve discussedisinthewholeserverbackupscenario,whichyoudbeperforminginthedata center.
ImageBackups
ImagebackupisanothertermforwhatIvebeencallingblockbasedbackups.Thereare reallytwowaystoachieveanimagestylebackup:byusingsolutionssuchasthetriedand trueimagesnapshotsoftwareandviawhatIllcallstreamingimages. Traditionalimagingsoftware,oftenusedfordeployingoperatingsystemimagestonew computers,commonlymakespointintimesnapshotsofadisk,typicallycompressingthe diskblockssothattheresultingimagefileismuchsmaller.Thissoftwareisntusually positionedasabackupandrecoverysolutionbutratherasadeploymentsolution:You makeatemplatecomputerthemanualway,imageit,andthendeploythatimagerather thaninstallingothercomputersmanually.Itsmostcommonlyusedfordesktop deployment,anditcanserveasakindoflastditchrecoverytoolfordesktops,essentially redeployingtheimagetogetamachinebacktosquareone.Butasapointintime snapshot,itsnotusefulforcapturingdata,changestoapplications,andsoforth. Anotherweakspotisthatsnapshotimagingsoftwareusuallyrequiresthesourcecomputer tobeunavailable.Typically,theimageiscapturedusingaprebootenvironment(likethe oneshowninFigure1.8)asortofstrippeddownOSthatrunsinlieuofWindows.This ensuresthatnoneofthefilesondiskareopenandchangingsothatasingle,consistent snapshotcanbegained.Youcanseewherethiswouldbeabitburdensomeasacontinual backuptechnique.
17
Don Jones
Figure1.8:Imagingsoftwaresprebootenvironment. StreamingimagesarethekindofblockbasedbackupIillustratedinFigure1.5,earlierin thischapter.Thistechniqueisthebasisforalmostrealtime,continuousdataprotectionof multipleservers.Youcouldusethetechniquewithdesktopcomputers,too,althoughI suspectdoingsowouldntbepracticalitwouldinvolvealotofbackupdataflyingaround yournetwork,nottomentionalotofstorage.No,Ithinkthistechniqueisreallygeared bestforthedatacenter,whereyourebackingupserversandwhereyoucaneasilysetup highspeedorevendedicatednetworkconnectionsbetweenthoseservers.
ApplicationSpecificIdeas
Someapplicationsprimarilymissioncritical,alwaysonapplicationssuchasMicrosoft SQLServerandExchangeServerpresenttheirownchallengesforbackupandrecovery. Theseapplicationsexecutablesarealwaysrunning,andtheyalwayshaveseveraldatafiles open,makingitdifficultforfilelevelbackupsoftwaretogetaconsistentsnapshot.
18
Don Jones
Tohelpaddressthis,theapplicationsdesignerstakevaryingapproaches.SQLServer,for example,hasitsowninternalbackupandrecoverycapability,whichistiedtotheproducts ownuniquearchitecture.Traditionally,thebestwaytogetaSQLServerbackupistoask SQLServertodoit.Youmight,forexample,useSQLServersowntoolstoproduceabackup file,thengrabthatfilewithatraditionalfilebasedbackupsolution.Or,youmightcreatean agentthattapsintoSQLServerandgetsthedatathatwaytheapproachusedbymost enterpriselevel,Backup1.0stylebackupsolutions.Figure1.9showsacommondialogbox forabackupsolutionsconfiguration,showingthataSQLServerspecificagentisloaded andabletostreamdatafromSQLServertothebackupsoftware.ExchangeServer functionalitymightworksimilarlyinfact,youcanseethattabinthefigureaswell.
Figure1.9:Applicationspecificbackupagents. ExchangeServersdeveloperstookaslightlydifferentapproach,choosingtointegratewith WindowsVolumeShadowCopyservice.Essentially,theyprovideacopyoftheExchange datafilesthroughVolumeShadowCopy;abackupsolutionsimplyneedstoaccessthe VolumeShadowCopyApplicationProgrammingInterfaces(APIs)andrequestthelatest copyofthedatabase.Again,itsExchangeServerthatsdoingmostofthework,buta dedicatedagentofsomekindisusuallyneededtogettotherightAPIs.AsFigure1.10 shows,evenWindowsServer2008sbuiltinbackupcanbeextendedtoseetheExchange Serverdatabases. 19
Don Jones
Figure1.10:BackingupExchangeServerinWindowsServerBackup. ThedownsideisthattheseapplicationspecificapproachesarestillBackup1.0innature, meaningtheyregrabbingasnapshot.Yourestillatriskforlosingdataandworkthat occursbetweensnapshots;particularlywiththesemissioncriticalapplications,Ithink thatsjustunacceptable. Blocklevelbackupscancertainlysolvetheproblembecausetheyregrabbingchangesat thedisklevelanddontparticularlyneedtounderstandwhatthosediskblocksarefor.A diskblockthatspartofafilelooksthesameasonethatspartofaSQLServerdatabase,so thebackupsolutionjustgrabsemall.Butfromarecoveryviewpoint,yourbackupsolution doesneedsomeadditionalsmarts.Hereswhy:Asimplefilesay,aWorddocument consistsofseveralblocksofdiskspace.Itseasytokeeptrackofwhichblocksmakeupany givenfile,andnodiskblockwilleversharedatafromtwofiles.Ifyouneedtorestore Salaries.xls,youfigureoutwhichblocksthatfileliveson,andrestorejustthose.Easy.
20
Don Jones
WithcomplexdatastoressuchasSQLServerandExchangeServerthingsarentsoeasy. Asinglemailmessagemightoccupymultipleblocksofdiskspace,butthosesameblocks mightalsocontaindatarelatedtoothermessages.Thedatabasealsohasinternalpointers andindexesthatneedtoberestoredinorderforagivenmessagetobeaccessible.Soa blockbasedbackupdoesntneedmuchinthewayofextrasmartstomakeabackup,butit willneedsomeclevernessinordertorestoresingleitemsfromthatbackup.Solution vendorstendtoapproachthisbyusingplugins:Itseasytothinkoftheseasbeingsimilar totheBackup1.0styleagents,buttheyrenot.Thesepluginsdontnecessarilyassistwith thebackupprocess(althoughtheymayrecordspecialinformationtoassistwith recoveries),buttheydocontainthesmartsnecessarytopeerinsidecomplexdatastores torecoversingleitems.
OurBackup2.0WishList
ThefollowinglisthighlightsdesirablecapabilitiesinaBackup2.0stylebackupsolution: Continuousdataprotectionthatsalwayson,alwaysworking,andasclosetoreal timeasispractical Theabilitytomovesnapshotsofmybackupstotape(orotherportablemedia)for offsitestorage Theabilitytorestoreanythingfromasinglefiletoanentireserverquickly,andto differenthardware(orvirtualhardware)ifneeded Blocklevelimagingthatprovidestheabilitytorollbacktoanypointintimeand keepsbackupsphysicallyseparatefromthesource Automaticnotificationsofproblemssuchascorruptionwiththebackups DoesntinterferewithcomplementarysolutionssuchasWindowsownVolume ShadowCopyfeature Theabilitytoeasilytestrestores,fromasinglefiletoacompletedisaster,ideally usingdissimilarhardwareorvirtualization TheabilitytorestoresingleitemsfromcomplexstoreslikeSQLServerorExchange Server
Illaddafewthingstothislistasweprogressthroughtheupcomingchapter,butthisisa goodstart.ItrepresentseverythingthattheBackup2.0philosophyisallabout,andit meetsalltheimpliedrequirementsinourbusinesslevelstatement: Backupsshouldpreventusfromlosinganydataorlosingany work,andensurethatwealwayshaveaccesstoourdatawith aslittledowntimeaspossible.
21
Don Jones
WhatsAhead
Ivegotafullplatecomingupforyou,startingwiththenextchapterinthisbook,whereI gettosharesomeofthehorrorstoriesIverunacrosswithmyconsultingclients,inthe news,andsoforth.Itskindofinterestingtoseetheproblemsothershavehad,butitcanbe instructional,too:Illexamineeachcaseanddrawconclusionsaboutwhatwentwrongand whatyouwouldneedtodotoavoidthatsituation. Chapter3iswhereIlldiveintotheactualtechnologyofwholeserverbackups.Thisisan areawhereIthinkBackup2.0reallyhassomeimmediatebenefit,butIllstartby examiningmoretraditionalwholeserverbackuptechniquesandidentifyingthingsthat dontalwaysworksowell.Ifyoureresponsibleforbackingupdomaincontrollers, infrastructureservers,Webservers,apublickeyinfrastructure,orsimilarservers,then thisisthechapterforyou. InChapter4,IlltacklethetoughtopicofExchangeServerbackupstoughenoughthatthe backupsoftwarethatshippedwithWindowsServer2008couldntevendoit.Again,Illlay outsomeofthemoretraditionalwaysthatExchangebackupshavebeenmade,andthen rethinktheprocessandcomeupwithawishlistofbackupcapabilitiesthatincludeseveral Exchangespecificconcerns.Chapter5willfollowthesameapproachforSQLServer,and Chapter6willexamineSharePointinthesameway. Chapter7willbeabitofadeparture,asIllfocusonvirtualizationserverbackups.Thisis stillarelativelynewfield,andIlllookatwaysinwhichtraditionalbackuptechniquesare beingusedandexaminehowwelltheyreactuallygettingthejobdone.Illcoversomeof theuniqueaspectsofvirtualizationbackupsandexamineinnovativetechniquesthat Backup2.0canbringtothetable. InChapter8,Illpullbackabitforabroadlookatotherbackupconcernsandcapabilities: Baremetalrecovery,dataretentionconcerns,complianceandsecurityissues,mobile infrastructureproblems,andsoon.Illalsolookatinstanceswhereoldschoolbackups mightstillprovidesomevalue,andIllofferadviceforintegratingBackup2.0withmore traditionaltechniques. Allofthebackupsweremakingaregoingtorequiresomeseriousstorage,soIlluse Chapter9tofocusonstoragearchitecture.Illlookathowbackupdataisstructured,and comparetheadvantagesanddisadvantagesofthingslikestorageareanetworks(SANs), tapedrives,localstorage,andsoforth,andexaminepressingissuesofstorage: compression,encryption,security,deduplication,andsoon.Illalsolookatuniqueways thatBackup2.0allowsyoutointeractwithyourbackedupdatamoreeasilyandefficiently.
22
Don Jones
Chapter10willfocusondisasterrecoveryandImeanrealdisasters.Illlookatthingslike baremetalrecovery,andIllcoversomeofthemoreinterestingcapabilitiesthattodays technologiesoffer,suchasusingvirtualizationratherthandedicatedoffsitefacilitiesas partofadisasterrecoveryplan. Chapter11isforallthebusinessmindedreadersoutthere;thischapteriswhereIll discussthecostsinvolvedinrearchitectingbackupandrecoverytouseBackup2.0 techniques.Naturally,Illalsohelpyoudeterminewhetherdoingsoisactuallyworthitto yourorganization,andeventacklesomeofthenontechnicalwhatIliketocall politicalissuesthatyoumayhavetosolveinordertomakeyourbackupsituationmore modernandefficient. Finally,Chapter12willbethechanceformetosharestoriesfrommyownexperiences withBackup2.0asortoftalesfromthetrencheschapter,withcasestudiesthat describesuccesses(andchallenges)IveseenwithBackup2.0. Wehavealongjourneyaheadofus,butyouvemadeagoodstart.Ilookforwardtoseeing youagaininthenextchapter.
23
Don Jones
Chapter2:12HorrorStoriesWeThought WeHadaBackup!
Horrorstories.Talesfromthetrenches.Casestudies.Callthemwhatyouwill,Ilove readingthem.Theyrealookintoourcolleaguesrealworldlivesandtroubles,andan opportunityforustolearnsomethingfrommistakeswithouthavingtomaketheactual mistakesourselves.Inthischapter,Imgoingtosharestoriesaboutbackupstohighlight problemsthatyouyourselfmayhaveencountered.Foreach,Illlookatsomeoftheroot causesforthoseproblems,andsuggestwaysthatamodernizedBackup2.0approach mighthelpsolvetheproblem.Someofthesestoriesareculledfromonlineblogpostings (andIveprovidedtheoriginalURLwhenthatisthecase),whileothersarefrommyown personalcorrespondencewithhundredsofadministratorsovertheyears.Oneortwoare evenfrommyownexperiencesindatacenters.Names,ofcourse,havebeenchangedto protecttheinnocentandthoseguiltyofrelyingonthedecadesoldBackup1.0mentality. Theimportanttakeawayisthateachofthesestoriesoffersavaluablelesson.Canyousee yourselfandyourownexperiencesintheseshorttales?Seeifyoucantakeawaysome valuableadviceforavoidingthesescenariosinthefuture.
Corruption:NotJustforPolitics
Illstartwiththisstory,asitsoneIimagineeveryadministratorhasbeenabletotellat leastonceintheircareers. Thismustbetheoldestbackupstoryintheworld,andyouhaveprobablyheardita milliontimes:Wedutifullytakeourbackupseverynight,eventhoughittakes foreveronthebigmachines.Whenwefinallyneedtouseone,thebackupis corrupted.Eitheritsthetape,whichisactuallyprettyrare,orsomethingwent wrongandthebackedupdataitselfisnogood.Andsothebosswantstoknowwhy weevenhavejobssincewearealluseless.Aaaah! AsIsaid,weveallbeenthere.Twoproblems,bothrelatedtotheoldschoolBackup1.0 mentality: Backupsshouldnttakeforever;theyshouldtakeconstantly.Thatis,apointintime backupisjustasnapshot.Evenifthebackupandrestoreworkperfectly,yourestill losingdata.Withcontinuousbackups,yourenotlosingdataitsreallythatsimple. Whydontbackupprogramstellyouwhentheyseecorrupteddata?Thisshouldbe thesimplestthingintheworldourindustryhashadchecksumsandothermeans ofvalidatingdataprettymuchforever.Itdrivesmecrazythatwehavetodiscover problembackupsbyporingthroughlogfiles.
24
Don Jones
Thisiswhywehavetorethinkbackups:Theyjustdontdowhatweneedthemtodo.Even whenwedoeverythingperfectly,theresahugeriskthatthebackupsjustwontwork.We needtorethinkwhatweredoing,howweredoingit,andhowweremonitoringittomake sureitworks. Ofcourse,Iwontmentionthattestingthosebackupsmighthaverevealedourauthors problembeforeheactuallyneededthosebackups.AnotherproblemwiththeBackup1.0 mentalityisthatnobody(well,hardlyanybodysomeoftheupcomingstoriesdogiveme hope)seemstotesttheirbackups.
CrackberryWithdrawal
Ifeveronedevicehasmanagedtosomehowmakeemailevenmoremissioncriticalthanit alreadywas,ResearchinMotionsBlackberryfamilymustbeit.Whensomethinggoes wrongwiththeBlackberryinfrastructure,itsaracetoseewhatyouspendmoretimeon: fixingtheproblemoransweringthephonecallsfromuserswhoaresuretheyretheonly oneswhovenoticedtheproblem: Iworkinanenvironmentthatlikeeveryone,IguessusesBlackberriesand cannotlivewithoutthemforonemoment.Theinfrastructureisactuallyvery complex:InadditiontoourExchangeServer,theresalsotheBlackberryEnterprise Server(BES)andasupportingSQLServer.Thelasttimewehadafailure,ittookus5 hourstogetbackonline,whichIthoughtwasprettygoodbuttherewasan inquisitionafterwardstofindoutwhywewereofflineforsolong.Evenwithallthe righttapebackups,doyouhaveanyideahowlongittakestorestoreaSQLServer andtheBES?Apparentlytoolongformyboss. ItstheBackup1.0mentality:Relyonthosebackuptapes,eventhoughjuststreamingthe dataoffoftapecantakehoursassumingitsallingoodshape,notcorrupted,andthatthe dataonthetapeisactuallyagoodbackupinthefirstplace.ThisisexactlywhereBackup 2.0methodologiescanmakeadifference:Withadiskblockbasedbackup,streamedinto thebackupstoreinalmostrealtime,youcanrestoreanentireservertoalmostanypointin therecentpastbypushingabutton.Does5minutessoundbetterthan5hours?By restoringintoastandbyvirtualmachine,5minutesisentirelyrealistic.
25
Don Jones
ExchangeFailure
NobodylikesitwhentheExchangeServergoesdown.Honestly,Ithinksomecompanies couldgowithoutafileserverforlongerthantheycouldlivewithoutemailespecially companieswhoseemployeeshaveBlackberries.Soheresastorytomoveyouremotions, drawnfromhttp://appassure.ning.com/profiles/blogs/whatilearnedonmyworst day?mkt_tok=3RkMMJWWfF9wsRow5%2FmYJoDpwmWGd5mht7VzDtPj1OY6hBssJJKtUg %3D%3D: WhenmycompanysMicrosoftExchangeServerfailedattheendofthequarter,it couldnothavehappenedataworsetime.ItbeganwiththeVPofSalesyelling Emailisdown,andcustomerscantsendustheirorders!ThenmyBlackberry startedgoingoff,calls,emails,IMsitwasrelentless.WhenIloggedontothe ExchangeServer,Ifoundthatsomeofmymostcriticalmailstoreswerenolonger mounted.WhenItriedtoremountthem,IreceivedtheambiguousyetominousJET 1601JET_errRecordNotFounderrormessage.Iimmediatelyconnectedtothe replicationserverthatrunsatoneofthecompanysremotesites,onlytofindthatI couldntmountthosemailstoreseither. WhenIcalledMicrosoft,techniciansprescribedthestandardprocedureofrunning Eseutil.Theywarnedme,however,thattheerrormessageprobablyindicateda corruptionproblemdeepwithinthedatabaseandthatrunningEseutilmightresult incleaningthestoresofalluserdata.Itooktheleap,onthechancethatitwouldbe quickerthangettingtherestoreprocessunderway.RunningEseutiltookhours,then failedwiththeevenmoreambiguousJET1003JET_errInvalidParameter.Atthat point,IknewIHADtogotothebackup. MycompanyrunsfullbackupseverySaturdaynightandincrementalbackupsthe restoftheweek.Istartedbyrecoveringthemostrecentfullbackup,thenapplying theincrementalsuntilIhadthebackupfromthenightbeforethefailure.Asyoucan imagine,thecalls,emails,etc.keptcomingallthewhileIwascopyingthemailstores frommydisktodiskbackupalthoughtheydidtaperoffabitafter11:00pm,when ourWestCoastofficeclosed. Onceourdatawasbackontheprimaryserver,itwastimetorollthelogsandmount thedatabase.However,whenthelogswereabout80percentapplied,theyfailed withtheJET501JET_errLogFileCorruptmessage.Atthatpoint,Microsoft supportcouldonlysuggestrunningEseutilthroughmyentirelogchain,notingthe corruptedlog,deletinganythingexceptlogfilesfromthelogdirectory,anddeleting thecorruptedlogandallthelogscreatedthereafter.ThenIcouldfinallyrestartthe logrolloperationfromscratch.Thisproceduretookmorethan6hours.Intheend, mycompanylost2daysofemailmessages,andrecoverytookmorethan30hours. ThecauseturnedouttobeaproblemwiththeRAIDcontrollerdriverthathadtaken monthstomanifestitselfafterapreviousserverupgrade.
26
Don Jones
Executivemanagementfigureditcostthecompanyabout$50K,sotheydefinitely wantedtoknowwhathadhappenedandhowitcouldhavebeenpreventedand howitwouldbepreventedfromhappeningagain.Letsjustsaywantedtoknow meansthatifIdidnthaveagoodanswer,mynamewasgoingonthetopofthenext layofflist.Iwasseriouslycommittedtofindingabetterrecoverysolution. SohereswhatIlearnedonmyworstdayasanetworkadmin:Youcanhave multiplecopiesofyourdataonreplicatedservers,ondisk,andontapebutifyou cantmountthecopies,theyarentanygood. TheBackup1.0mentalitycostthiscompany$50,000.Why?BecausetheBackup1.0 mentalityfocusesonmakingbackups,notrestoringdata.WithBackup1.0,wetendtofocus onbackupwindows,tapedrives,andsoonwedonttendtofocusonwhatwillhappenin theeventofadisaster.Evenwiththeirbackupsand30hoursofeffort,thecompanystill lost2daysofemailsthisisabackupplan? ExchangeServeriscertainlyacomplexanddifficultproductwhenitcomestobackups.The backandforthbetweentheExchangeServerproductteam,theWindowsproductteam, andMicrosoftsownbackupproducts(intheSystemCenterfamily)meansthatExchange andWindowsalonedontofferaneffectivebackupplan.Thirdpartyvendors,however, tendtofocusonExchangespecificagentsthatjustmakecopiesofthedataashandedover byExchange.Asthishorrorstorypointsout,Exchangemightnotalwaysbehandingyou gooddata,meaningyourbackupsareuseless. SohowwouldBackup2.0changethings?WellcoverExchangeServerbackupsindetailin Chapter4ofthisbook,butfornow,letsjustcomparethissituationtomywishlistformthe previouschapter: Thisfellowssituationcouldhavebeenimprovedimmeasurablybyhaving continuousbackupsratherthanpointintimebackupsoflogfilesanddatabases. Blocklevelimaging,whichgrabsthechangesastheyhittheserversharddrives, wouldhaveprovidedanuninterruptedstreamofbackupdataallthewaytothe pointwhereExchangesdatafileswerecorrupted.Theservercouldquicklyhave beenrolledbacktothatpointintime. ExchangeServersnativebackuptechnologiescouldstillhavebeenused,ifdesired. Ourpooradmincouldhavetestedhisrestorecapabilitiesmorefrequently,been morefamiliarwiththeprocess,andlikelyspentfarlessthan30hoursgettinghis serverbackonline.
Remember:Backupsshouldpreventusfromlosinganydataorlosinganywork,andensure thatwealwayshaveaccesstoourdatawithaslittledowntimeaspossible.TheBackup1.0 mentalityoriginallyemployedinthishorrorstorycertainlydidntmeetanyofthose criteria.
27
Don Jones
MigratingtheClusterOrNot
Sometimes,disasterdoesntalwaysmeanafailedserver.Sometimesasolidlackof planningcanprovideallthedisasteryouneed! Wereceivedthenewserversforthenewcluster.Thejob:SwapoutanoldWindows forabrandnewone,configureit,andmoveallthefilesanddataovertothenew cluster.Thishastobedonewithinalatenightmaintenancewindow,whichmeans mywifewillnotbehappyagain,butIllbuyflowers.Starttimeisaround11:00pm, anditmustbedoneby7:00am8hours. BothclusterswouldeventuallyberunningthesameversionofWindows,onceIgot everythinginstalled.TheclusterserverscamewithnoOSinstalledatall,though,so installingWindowswouldbemyfirststep.Fortunately,theserverhardwarewas basicallythesameinbothclusters. Afterdriving300kmahorrordrivingonPolishroadsIsetuptheserver hardware.Theoldclusterishummingalongnexttome,andImreadytoget Windowsinstalled.Wherearethedrivers? Andtheproblemarose:Someonehadlosttheservermanufacturersdriversdisc.I knowwhatyourethinkingjustdownloadthem,right?Well,sufficetosaythatthis isaverysecureorganizationperhapsgovernmentalandnobodyinthebuilding atthathourcouldactuallygettotheInternet.SoIhadtopulloutmymobilephone and,despitethehighcostofdatatransfers,downloadthedriversfortheserverover the3Gcellularnetwork.Thenmyphonesbatterydiedandmewithoutmy charger. Iaskedthemaboutserverbackups,figuringwecouldperhapsjustusethoseto restoretothenewmachine,butalltheybackuparethefiles.Theysaidtheyhad neverbeenabletodoabaremetalrestoreusingtheirbackups,sotheyjuststopped backinguptheoperatingsystem(OS). Internetaccesswasavailableagainat8:00am,andsomeonehadtotakemeintothe secureareawhereInternetaccesswasavailable.Iwastired,theentirenightwas wasted,andIhadtodoitallagainthenextnightwhenIfinallyhadthedriversin hand.
28
Don Jones
Ouch!Weveallhadalatenightlikethatatsomepoint,anditsneverfun.Andthefirst thingIhavetoaskmyselfiswhytheycouldntsimplytakeabackupoftheoldcluster machinesandrestorethemonthenewhardware?BecauseintheBackup1.0world, restoringtodissimilarhardwareisoftenimpossibleoratleastnotrecommended.Asa result,manyorganizationsjustbackuptheirfilesbutabackupisuselesswithout someplacetorestoreit.AmoremodernBackup2.0mentality,however,wouldsaythatthis clustermigrationwasreallynodifferentthanabaremetalrestoreafteracompletecluster failurewhynotrestorethebackuptothenewhardware,andcallitanight?IntheBackup 2.0world,wedbetakingblocklevelbackupsoftheentireserver,sowecouldjustapply thatbackuptothenewhardware(whichweretoldwassubstantiallysimilartotheoriginal hardware)andcallitanight.Lesstimeonthejob,alowercellphonebill,andaless frustratedwife.
VirtuallyExchange
Itsbecomingmoreandmorecommontorunserversoftwareinsidevirtualmachines, oftenonavirtualizationhostrunningVMwareorMicrosoftsHyperV.Butyoudbe surprisedhowthisscenariomightimpactyourdisasterrecoverysituation: Afterjust2weeksonmynewjob,ourExchangeServerdecidedtooperatein comatosemodeandstopsendingandreceivingemail.Ifoundthatitwasrunning onavirtualserverandwasbeingbackedupdailybyathirdpartysoftwarepackage. IfiguredIwasfine. Thebackupadministrator,itturnsout,reallyhadnoideaabouthowExchange workedorwhat,ifanything,wasneededtocompletethebackupsproperly.The backupshadneverbeentested,soIwasstartingtothinkfondlyofmyoldjob. Iwasabletogettheserverpartiallyoperationalbutwounduphavingtocallthe softwarevendoronlytogetarefundformysupportcallandatersestatement thattheydidnt(atthetime)supportExchangerunninginavirtualenvironment. Oops. Iwounduphavingtoportmyhalffunctioningsystemtoaphysicalserverandwas eventuallyabletorecovereverythingexceptfor2daysworthofemail,whichdoes notendearyoutoyournewbosses,letmeassureyou. Notsupportedinavirtualenvironmentiskindofacheaptrickforasupportorganization topull,butithappens.Thatswhyitscriticaltoworkwithsoftwareandbackupsolutions thatexplicitlysupportvirtualenvironments.InaBackup2.0world,virtualizationis expected;anyonewhothinksthatvirtualizationisstillnovelorunusualisadinosaur.
29
Don Jones
Ofcourse,itsalsocriticaltotestthosebackups,andfranklythebackupsolutiononce installedandconfiguredshouldjustwork.Iftheressomethingspecialneededtomake avalidExchangebackup,thesolutionshouldknowaboutitandtakethosesteps automaticallythatswhyitscalledasolution,notanadditionalproblem.Therightvendor willhavealltheknowledgetheyneedontheirownstaff,andshouldmakeabackuptool thatdoesthejobyoupaidfor.Itshouldmaketestingthosebackupseasyespeciallyina virtualizedenvironmentwhereyoudontevenhavetocomeupwithanotherphysical machinetodoatestrestoreon!
ForWantofaCheckBox
Windowsnativebackupcapabilitiesare,andalwayshavebeen,prettyprimitive.Theyre alsofirmlyentrenchedintheBackup1.0mentalityofjustmakeacopyofthenecessary data.Sometimes,thatmentalitycanreallybiteyouwhereithurts: Itwaslate2006,andIwasworkingforabranchoftheUSgovernment.One afternoon,webeganreceivinglargenumbersofcallsattheHelpdeskthatpeople werehavingtroubleloggingintoaparticularActiveDirectory(AD)domain.Now, thisparticulardomainwasusedforcertainspecialprojects,andonlyhadtwo domaincontrollers,buttheydidsupportabout1000usersallofwhomwere prettyupsetthattheycouldntlogonandaccesstheresourcestheyneeded. Bothdomaincontrollers,itturnedout,hadsomehowbeencorrupted.Inever figuredoutwhathappened,butIwasevenhavingproblemsloggingontothe consoledirectlyoneithercomputer.Soalongwithacoupleotheradmins,Idecided tojustdoafullrestoreoftheentiredomain.Afterall,thatswhatbackupsarefor, right? Tomyhorror,Idiscoveredthatwhiletheserverwasindeedbeingbackedupevery singlenight,nobodyhadevercheckedtheSystemStatecheckboxtoensurethat ADwasbackeduptoo.Sothebackupswereessentiallyuseless. Ittookfouradministrators96hours,workingaroundtheclockinshifts,torebuild thatdomainbyhand.Wesleptanhourortwoatatimeatourdesks,thengotback upandstartedworkingagain.WestartedonaTuesdayanddidntleaveuntilearly SaturdaymorningallwearingthesameclotheswedworkinonTuesday. Somebodyatsomepointhadthepresenceofmindtogopickupsometoiletriesfor us,butitwasthelongestTuesdayIcanrememberallbecauseonestupidcheck boxwasntchecked.
30
Don Jones
ThisdrivesrightbacktotheBackup1.0mentality:Onlybackupwhatweabsolutelyneed. ThatattitudecomesfromafewfactsoflifeinaBackup1.0world: Backupstakealongtime.Weseektominimizethetimeittakestopullabackup, sowecutoutunnecessaryfiles,folders,andsystems.Inthiscase,the unnecessaryfileswereunfortunatelytheverythingthatwasactuallyneeded. Backupwindowsarelimited.Theveryconceptthatwecanonlybackupourdata duringcertaindeadorslowtimesiscrazy,whenyouthinkofit.Whywouldntwebe backingdataupwhileitschangingsothatwecancaptureeverychange?What aboutuserswhomighthavejustbeencreatedonthedayofthefailuretheywont beinthepreviousnightsbackup,soweloseallthatwork? Thebackupsareonaschedule.Youhope.Toooftenyoufindoutwhenitstoo latethatthebackupwasntworkinglikeyouthought.Theproblemisthatthe scheduleisusuallylateatnightwhennobodysaroundtonoticeaproblemand nobodythinkstocheckeverymorning. Wedontneedtotestourbackups.Thistranslatesas,itstoomuchofapaininthe necktotestourbackups,andwedonthavethetime,sowerenotgoingtodoit. Testing,ofcourse,ishowyoufigureoutthatyourbackupsarentworking,andfind outbeforeyouactuallyneedtorelyonthem.
AsolidBackup2.0mentalitychangesthingsaround: Backupsarecontinuous.Theydonttakealongtimebecausetheyrealways running,grabbingeverylittlechangethatcomesdowntheline. Backupwindowsareunlimited.Theyre247365,infact,becausethebackup windowisalwaysopen,usingasolutionthatgrabseverychangeasithappens.This point,combinedwiththepreviousbulletpoint,meansyoujustgoaheadandbackup everything.Whenyourenotpickingandchoosingdatatobackup,youwindup alwayshavingwhateveryouneedtodoarestore. Thebackupsarentscheduled.Theyrealwaysrunning,andyoucanalwayssee thathappeninginrealtime. Itseasytotestbackups.Whetherrestoringtoavirtualserveroraphysicalone, testingbackupsshouldbeeasyandfastsothatthosetestscanbecomeroutine.If yourenotgoingtotestyourbackups,whybothermakingtheminthefirstplace?
31
Don Jones
WeightLossPlan
ThisoneisjustfunnyIhadtoshareit.Takenfrom http://appassure.ning.com/profiles/blogs/backupswhatbackups: Iwasleadtechforabigserverupgradeforacompanythathadtokeeprecordsfor7 years.Wewereimagingtheoldserverdatauponthenetwork,thenreimagingback down[tothenewserver]. Well,onedayIthoughtIwouldmultitaskfourmachinesatthesametimeandgoto lunch.WhenIcamebackfromlunch,Ididawipediskoftheoldharddrivesand noticedthatoneofthetechskeptleaningovertheworkbench;hisbellywassobig thatitwouldhitthespacebar,whichwouldcancelthedatatransfer. Well,tosaytheleast,IfoundthathehaddonethisonthedayIwenttolunch.Well, noonehadanybackups,so7yearsofdatawasgone. Iknowitsnotfunny,butitkindofis.Thelessontolearnhere,though,isthatpointin timeimagesarereallynobetterthananoldstyletapebackup.Anythingthatisjustmaking acopyofthedataataparticularpointintimeisBackup1.0mentality,whereweremore concernedwithgettingacopyofthedataand,asweveseeninthisstory,alotofstrange thingscangowrongtointerruptthatcopyandmakeituseless. SohowwouldBackup2.0solvethisproblem?Byhavingacontinuousbackupoftheold serverinthefirstplace.Theredbenoneedtotakeanimageduringaserverupgrade,and infact,theupgradecouldbedonefasterandmoresmoothlybyjustrelyingonthatblock byblock,Backup2.0stylebackup.Anupgradeisreallynodifferentthanabaremetal disasterrecoveryjustperhapslessurgent.Soagoodbackupsolutionshouldbeableto assistwithaserverupgradeormigrationallthemorereasontohaveagoodbackup solutioninplace.
So,640KBIsntReallyEnoughforEveryone?
Theamountofstorageusedbymodernorganizationsistrulystaggering.Infact,getting enoughtimeandnetworkbandwidthtobackupallthatdatacanoftenbeimpossible. Impossible,thatis,whenyourelivinginaBackup1.0worldthatsfocusedonpointintime copiesofdata: IamtheonlyStorageAdministratorforamediumsizedenterprise.Wehavemore than280terabytesofstorageusedonvariousvendorsequipment.WhenIfirst started,wewererunningallofourbackupsthroughasilowithLTO1drivesinit. Wecouldnteverbackupallweneededtointhebackupwindowweweregiven.We finallyupgradedtoLTO4drivesafterseveralfailedattemptstorecovercriticaldata andservers,alongwithcomplaintsaboutnetworkslowdowns,which,ofcourse, camerighttome.
32
Don Jones
Abitofdiggingrevealedthatthecompanywasactuallypullingweeklyfullbackupsof30to 45terabytes,whicheachrequiredabout14hourstocomplete.14hours!Andobviously someoneishavingtopickandchoosethedatathatgetsbackedupbecausetheyreonly backingupabout16%oftheirdata. Onceagain,itsthemostinsidiouspointsabouttheBackup1.0mentality:Youhavetograb backupsduringsomefixedpointintime,youonlyhaveacertainamountoftimetowork with,andyouhavetograbwhatyoucanduringthatwindow.Backup2.0,bycontinually grabbingchangesastheyoccurcanbackupmuchlargerdatastores,keepthebackups entirelyuptodateatalltimes,consumelessnetworkbandwidthindoingso,andgraball yourchangesnotjustthedatayoucangrabina14hourwindow. ThisstorybringsoutsomeotherweaknessesofBackup1.0.Noticethattheauthorstarted withLT01drivesbuteventuallyhadtoupgradetoLT04drivesfortheimprovedspeedand reliability.Backup1.0byforcingustolivewithinbackupwindowsoftenforcesusto spendalotmoreonourbackupinfrastructurethanisnecessary.Youcouldeasilydrop $15,000onanLT04tapelibraryequippedwithafastFibreChannel4Gbpsnetwork interfaceandeverypennyofthatexpenseissimplytoallowyoutocrammoredatainto yourbackupwindow.Butifyouexpandedthatwindowto724365throughcontinuous backup,youdbeabletocaptureeverythingonsignificantlylessexpensivehardware.
SQLServerSyndrome
IveworkedwithSQLServerformanyyears(sincev6.5,Ithink),andorganizingbackups hasalwaysbeenachallenge.Mystoryisalotlikethisone: IhavebackupsinthreeversionsofSQLServer,indevelopment,production,and testingenvironments,onmultipleservers.Lotsofbackups,inotherwords probably45to50instancestotal,withasmanyas45databasesperinstance. Wegrabfullbackupseverynight,andtransactionlogbackupsevery15minutes.So ourmaximumdataloss,ifeverythinggoesright,is15minutes.Wetestourbackups byrestoringthefullbackupstoourtestenvironment.Thathelpskeepthetest environmentcloselymatchedtotheproductionenvironment,whichisvaluablefor thesoftwaredevelopers,andhelpsusmakesurethosebackupsareworking.Weroll lastweeksbackupsintoadifferentfolderandgrabthosetotape. WegetfailurenotificationsthroughemailandSystemCenterOperationsManager, andwerelyonmanualinspectionofbackupjobsandfolderstomakesure everythingworks. Rightnowweretryingtoplaywithdifferentbackupschedulesandlookatusing differentialbackups,alltolessenthenetworktrafficandthedrivespaceoccupied bythemanyfullbackups.However,weneedtokeepourrecoverytimesshort,so weretestingtoseehowmuchoverheadthedifferentialsaddtorecoverytimes.
33
Don Jones
Haveyouevergoneonvacationforacoupleofweeksandforgottentoputaholdonyour mail?Youcomebacktoanenormouspileofmailandspendhoursgoingthroughitall.But dealingwiththatsamemailspreadoutoverthecourseofaweekismucheasier,right? Backupsarethesameway.TheBackup1.0mentalitytellsustowaituntiltheeveningor theweekendtograballofourdata.Thatmeanswehavetohammerthenetworkhardto pullallthatdataovertothebackupserverquickly(afterall,thetimewindowisonlyso big).Wouldntitbeeasierifwecouldjuststreamthechangesoverconstantly,asthey happen?Readingonepostcardadayisntabigdeal;comingbacktoastackofthematthe endofthevacationiswhatspainful.StreamingthechangesiswhatBackup2.0isall about. Remember,too,thegoalofbackups:Backupsshouldpreventusfromlosinganydataor losinganywork,andensurethatwealwayshaveaccesstoourdatawithaslittledowntime aspossible. Losing15minutesworthofwork,justbecausethatswhenyoumadeyourlasttransaction logbackup,isnuts.Whyshouldanydatabeatrisk?Pointintimebackupsalwaysplacedata atrisk;continuousbackupsdont.IwanttorevisitmyBackup2.0wishlist,fromthe previouschapter,onemoretimeandseehowitappliestothiscasestudy: Continuousbackupseliminatebackupwindowsnomoreatriskdata,nomore poundingthenetworkduringabackupwindow. Backedupdatacanbemovedtotapeonadailybasisforsafekeepingwithoutallthe cumbersomerollingoffiles.WhobecameaDBAsothattheycouldspendtime managingfilesondisk? Beingabletorestoreeitherasingledatabaseoranentireserverwouldprovidethe testenvironmentwithalotmoreflexibility,inadditiontobeinganexcellenttestof restorecapabilities. Blocklevelimagingwouldallowanydatabasetoberolledbacktoapointintime withouthavingtofigureoutwhatcombinationofdifferentials,fullbackups,andlog backupsisneeded.Whatsmore,blocklevelimagingpermitsrestorationtoany pointintime,whereasourauthorstransactionlogsonlyproviderollbackin15 minuteincrements. Stillneedthoseoldschoolbackupfilesforotherpurposes?FineatrueBackup2.0 solutionwontinterferewiththem.
ItjustseemsasifBackup2.0,asasetofpracticesandcapabilities,ismuchmorewellsuited toSQLServerthantheoldstyleBackup1.0mentalityourauthorisrelyingon.Welllookat SQLServerinmoredetailinChapter5.
34
Don Jones
PatchProblem
ThisstoryreallyillustrateswhyIdislikepointintimebackupssointensely.Sure,youcan achievealotofbusinessgoalsusingBackup1.0techniquesandtools,butyouhavetobeso verycarefulinordertogetexactlywhatyouwant.Whoneedsthatextramentaloverhead? Iworkinoneofmycompanyslargerdatacenters,andsupportabout100servers. Mostofthesearefileservers,butthereareacoupleofdomaincontrollers,afew SQLServermachines,andthreeExchangeServerboxes. Weareverygoodaboutpatchingourcomputers.Wetypicallywillnotapplyasetof patchesuntilwehaveafullbackupofthecomputersOSandapplicationfiles,and weusuallymakeabackuprightafterapplyingthepatch,too.Wetendtoapply patchesduringmaintenancewindowswhentheserversarentotherwiseavailable. Onsomeofourlargerservers(theExchangemachinescometomind),itgets difficulttograbonefullbackupandapplypatchesduringour6hourmaintenance windows(youtrytakingemailawayfrompeopleforlonger),sosometimeswetake abackuponenightandapplypatchesthenext,thentakeasecondbackupthe followingnight. Thesystemworksbutnotalwayswell. IcanrecallacoupleinstanceswhereExchangepatcheshavecausedproblemswith someofourthirdpartysoftware,andweneededtorollbacktotheprepatch backup.Unfortunately,awholedayofworkhadpassedsincethatbackupwasmade, sowelostallthatwork.Peoplebecomeincrediblyunhappywhenemailgoes missing. Inatleastonceinstance,wedidntrealizeageneralWindowshotfixwascausing problemsforaboutaweek.Atthatpoint,theprehotfixbackupwasprettyaged. Thiswasonadomaincontroller,sowedecidedtoapplytheoldbackupanyway, knowingthatthedomainwouldbringitselfuptodatethroughreplication. Unfortunately,thebackupalsowefoundouthadsomedeletedobjectsinthe domain,whichwereneartheendoftheirtombstonelife.Thepracticaleffectwas thataboutadozenformerlydeletedobjectssuddenlyreappearedinthedomain. Oursecurityauditorsfreakedout,peoplewereyelledout,anditactuallytookusa whiletoworkoutwhathadhappened,sincethatsnotascenarioyouseeeveryday. Wevesincedecidedtorelylessonbackupsforundoingpatches.Wevestarted spendingmoretimetestingpatches,whichisofcourseagoodideabutitsvery boringandittakesalotoftimewedidntreallyhavetospare.Italsomeansour patchesonlygetrolledoutabouteveryothermonth,ratherthaneveryotherweek, andIworryaboutwhathappenswhenoneofthosepatchesfixessomemajor securityholeandwehavetoleavetheholeopenfor2monthsjustbecauseofour processes.
35
Don Jones
Again,theBackup1.0mentalityhasdeeperreachingeffectsthanjustdisasterrecovery problems.Inthisinstance,thecompanyhasactuallydecidedtorunoutofdatesoftware forlongersimplybecauseofthewaytheirbackupprocesseswork.Unbelievable.Ifeverthere wasacaseofthetechnologydrivingthebusinessratherthantheotherwayaroundlikeit shouldbethismustbethatcase. Thereareeasytorecognizeproblemshere,whichshouldbefamiliartoyouatthispoint: Backup1.0spointintimesnapshotsdontprovidemuchgranularitywhenitcomes timetorollbacksomething. Backup1.0srelianceonbackupormaintenancewindowstookawaysomeofthe authorsflexibilitywithregardtohisExchangeinfrastructure. Exceptinafewspecialsituations,Backup1.0tendstobeanallornothing proposition:eitheryourollbacktheentireserveroryoulivewithwhatyouvegot. Therearentmanygoodwaystorestoreasingleapplication;Backup2.0,by contrast,canmoreeasilypulloutjustthebitsrelatedtoaspecificapplicationand restoreitwithasingleclick.
VirtualHotSpares
VirtualizationisbeingusedinmoreandmorecreativewaysanditssadwhenBackup1.0 cantkeepup.Heresanexcellentstoryaboutusingvirtualizationtoprovidehotsparesfor criticalcomputersIcanseethistechniqueeliminatingtheoldschoolrentalfacilities whereyoudhaveabunchofphysicalserversreadytoactasstandbysintheeventofa disaster.ButseeifyoucanspotwhereBackup1.0methodologieswrecktheeleganceofthe solution: Myorganizationusedtorelyonanoutsourcedhotsitefordisasterrecovery.The theoryisthat,intheeventourdatacenterwashitbyameteororsomething,we wouldrelocatetothisoffsitefacility.Theyhavelotsofservershandy,andwedjust restoreourlatestbackupstothoseservers.Sure,wedlosesomedatabutwe wouldntloseitall.Wecouldthenoperateoutofthatsiteuntilourowndatacenter wasbroughtbackonline. Thecostforthesefacilitiescanbestaggering,soweverecentlyconstructedourown recoverycenterinoneofourlargerremoteoffices.Wecantaffordtobuyallthe serverswedneed,sowearerelyingonvirtualization.Weveidentifiedourtwo dozenorsomostimportantservers,andwehaveenoughserversinthespare facilitytovirtualizeallthecriticalservers.Performancewontbetops,butinthe eventofadisasterthatserious,wereokaywiththetradeoff.
36
Don Jones
Tokeepthesehotsparesworking,weregularlytakeofflineourcriticalserversfor maintenanceanddoaphysicaltovirtual(P2V)conversion,convertingeach physicalserverintoavirtualmachineinthesparesite.WeactuallydotheP2V conversionlocallyduringthemaintenancewindow,thencopythenewvirtual machineimageslaterbecausethefilesareofcoursehugeandtheWANcant supportgiantcopieslikethatveryquickly. Intheory,thatmeansourcriticalserversarealwaysreadytogoandarenomore thanaweekorsooutofdate.Ourplanwouldbetorestoreourmorerecentbackups toeachvirtualmachinetobringitevenmoreuptodate. Greatplanalmost.HavingtopullserversofflinetodoaP2Vmigrationisnuts.Whytake serversofflineatall?Thebonesofthismethodareagoodidea,butthewholeBackup1.0 snapshotmentalitywhichmostP2Vmigrationsplayintoismessingthingsup. Considerthisinstead:UseaBackup2.0stylesolution,whichmakescontinuous,blocklevel backupsofyoursourceserverswithouttakingthemoffline.Restorethosebackupstoan emptyvirtualmachineonewithoutevenanOSinstalled.Thisisessentiallythesameasa baremetalrestore,justthatthemetalinquestionisvirtual.Yourbackupswillalwaysbe uptodatetothelatestchanges,andyoucandoaweeklyrestoretoyourhotsparevirtual serverssothattheyrereadytogoatamomentsnotice.Or,ifyourbackupsarebeing storedsafelysothatacompletedisasterinyourdatacenterwontalsotakeoutyour backupsyoucouldjustdoyetanotherbaremetalrestorewhendisasterstrikes,andyour hotspareswillbevirtuallyindistinguishablefromtherealservers. Bytheway,wheretokeepyourbackupssothattheyresafeisobviouslyakeypartofthe strategyanditssomethingwellexamineinmoredetailinChapter9.
IsitaServeroraPhotocopier?
Thisscenariooutlinesaproblemthatalotofushave,butthatweoftendontthinkabout. Itsnotuncommonforcompaniestoexercisestorageresourcemanagementontheirfile serversprohibitingcertainfiletypes,forexample,orrestrictinguserstoacertain maximumamountofdiskusage.Oneoftquotedreasonforthiskindofstorageresource managementisthatstorageisexpensivetomanageandmaintainespeciallybackups.But itsstrangethatevencompanieswiththestrictestdiskquotasrarelyexerciseanykindof storageresourcemanagementonthebackupsthemselves: Irecentlystartedworkingforanewcompanyandwaspleasedtoseethattheyhada verywellimplementedDistributedFileSystem(DFS)infrastructure.Idontlike mappingnetworkdrives,andtheusersinthiscompanyhadlearnedtoaccessUNC pathslike\\Company\Sales\Files\NovemberratherthanrelyingonthegoodoldS: drive.
37
Don Jones
ThecompanywasalsousingDFSreplicastohelpusersinremoteofficeswhich oftenhaveslowerWANlinksgettocriticalfiles.Ifyourenotfamiliarwithit,DFS usesWindowsFileReplicationServicetocopyagivenDFSleaftooneormoreother fileservers.Soaccessing\\Company\General\Policiesmightactuallygetyoutoany oneofadozenserversthatallhostthatsamecontent.Inthisorganizationscase, eachlocalofficefileserverhasacopyofthisandothercommonlyaccessedfile shares,like\\Company\General\Sales\Forms.Thefilesinthesesharesarent updatedreallyfrequently(maybeacouplefilesadaychange),sothereisntreally muchreplicationtraffic,andhavinglocalcopiesletseveryonegettothefiles withoutrelyingontheWAN. Likeanygoodcompany,webackupallourservers.Smallerremoteofficesactually havedirectattachedtapedrivesforthispurpose.Theyvebeenusingthismodelfor years,butonlyrecentlyhavewestartedrealizingthatthebackupswerent completingbecausethetapedrivesdidnthaveenoughcapacity. Istartedlookingintoit,andrealizedthatthecompanyhasbeendoingalotofDFS replicasabout50to60GBworthrightnow,andtheyreaddingmoreallthetime. Thisisthesamedatabeingbackedupoverandoveragainindifferentlocations.All thatduplicateddataiswhatscausingtheproblem.Istartedtryingtofigureout exactlyhowmuchduplicateddatawehavesothatIcouldmakeabusinesscaseto management.Indoingso,Ifoundthatmanyofourfileservershavemultiplecopies ofthesamefiles,allonthesameserver.Alotofthetimeitcomesfromusershome directories:thetwoserversthathostthe\\Company\UsersDFSnodehavetensof thousandsofduplicatedfilesbecauseusersdragcommonlyusedfilesintotheirown homefolders. Wewerebackingupatotalof13.2TBofdata,andcloseto30%ofitwasduplicated data.Onethird!Werecurrentlytryingtofigureouthowtoexcludetheduplicates, butitsactuallyverytrickywecantjustnotbackupusershomefolders! Dataduplicationthebaneofstorageresourcemanagerseverywhere.Infact,datade duplicationisanincrediblyhotnewtechnology;somuchsothatindustrygiantslikeEMC, Dell,andHPbattleitoutoveracquisitionstheybelievewillputthemattheforefrontofthis importantnewtechnology.Soimportant,infact,thatImgoingtoaddabulletpointtomy Backup2.0wishlist: Backupsshouldnotcontainduplicatedata.Backupsomethingonce,notmorethan once. Thisisatallorder,anditmightonlybepossibletoimplementittoadegree.Forexample,a solutionmightdeduplicatedataacrossanentireserverbutallowduplicatesofthatsame datatobebackedupfromotherservers.Othersolutionsmighttakeabroaderviewandde duplicatedataacrosstheentireorganizationthatwouldcertainlybepowerful,butit seemslikeatechnologicallydifficulttask.
38
Don Jones
HorrorStoriesandtheLessonsTheyTeachUs
Thesetwelvestorieshaveobviouslyhadcommonthemes.BeforeIwrapupthischapter,I thinkitsworthspendingjustafewmomentsreviewingthosethemes,focusingonthe takeaway(whatweshouldlearnfromthem),andreiteratingwhatweshouldbedoing insteadtoprovidebetterbackup,recovery,andothercapabilitiesforourorganizations: Pointintimebackupsarehorrible.Yourealwaysgoingtolosesomedataifyou havetorelyonpointintimebackups,andwhowantstolosedata?Continuous backups,orcontinuousdataprotection,ifyouprefer,offermuchmoreflexibility, lessriskofloss,andgenerallylessmanualeffortandoverhead.Pointintime backupsalsolackgranularityeventhosequarterhourSQLServertransactionlog backupsleavetoomuchdataatriskanddontletyourollbacktheservertoa precisepointinthepast,ifneeded. Backupwindowsarehorrible.Takingserversofflineisnevergoingtoaddmuch valuetotheorganization.Sometimes,fortruemaintenance,youhavenochoice;you dohaveachoicewhenitcomestobackups.Backupwindowsforceyoutochoose whatdatatobackupyoucanonlygrabwhatwillfitwithinthewindowplace unnaturalstrainsonthenetwork,anddriveanumberofotherconstraining decisionsthatsimplyaddnovaluetothebusiness.Continuousbackupsdont requireawindow,soyouarefreetomakebetterdecisionswithoutallthetime constraints. Duplicateddataishorrible.Nobodylikesthatwehavetodobackups,sowhyback upanythingmorethanonce?Duplicateddatabloatsyourbackups,requiresmore time,andmoreimportantlyrequiresmorespaceforbackupstoragespacethat mightbeexpensive(especiallyforoffsitestorage),slow(liketapes),andsoon.Look forsolutionsthatcanautomaticallydeduplicatedatawhenmakingbackups. Anythinglessthantheentireserverishorrible.Sure,yourworstnightmare mightbelosingasingleemailfromtheCEO,butthatsfarfromtheonlynightmare youneedtopreparefor.Dontmakedecisionsaboutwhatsimportantbackitall up.Withtherightsolution,thatoneallinclusivebackupwillenableeverythingfrom singlefilerestorestobaremetalserverrecoveryandeverythinginbetween, includingrestoringanentireapplicationtoanearlierpointintime. Backupsgetcorruptedwhichishorrible.Backupsolutionsshouldtellyou whensomethinggoeswrongnotwaitforyoutofindoutforyourself. Nottestingyourbackupsishorrible.True,oldschoolbackupsolutionsmakeit incrediblydifficulttoactuallytestbackups,butthatsnoexcuse.Orrather,itsan excuseforfindingasolutionthatmakestestingbackupseasier,likeonethat supportsrestoringtoavirtualmachineordissimilarhardware,sothatyoucan practicallytestyourrestores.
39
Don Jones
Theresplentytolearnfromthesestories,andplentytolookforinaBackup2.0solution. Thekeytothewholethingseemstobethisideaofcontinuousdataprotection,grabbing individualblocksoffdiskassoonastheychangeandstoringthoseblocksinawaythat makesitpossibletorestoreanentireserverorjustasinglefile.Yes,solutionslike SharePoint,SQLServer,andExchangeServeraddsomecomplexitytothepicture,butby continuallystreamingchangeddiskblocksintothebackupsystem,youcangrabanytypeof backupyouneedcontinuously,withoutthepointintimetroublesofBackup1.0.
ComingUpNext
ItstimetodigintothisBackup2.0philosophyabitdeeper.Inthenextchapter,Illstart lookingatbackupsbeginningwiththemostcommontypeofbackuporwhatshouldbe themostcommontypeofbackup:wholeserverbackups.Whetheryouusethemtoprotect dataortoprepareforacomplete,wholeserverdisaster,thesebackupsshouldbethe stapleofyourdisasterrecoveryplan. Illdiveintothetechnicaldetailsandhurdlesthatwholeserverbackupspresent,andcover someofthenativesolutionsthatWindowsprovides.Illoutlinespecificproblemsand challengesthatbothnativeandthirdpartysolutionshavetodealwith,andlookatsomeof theBackup1.0methodologiesyouredoubtlessfamiliarwith.Thatwillallhelpsetthe contextforadiscussiononrethinkingserverbackups:Illmakeawishlistofcapabilities andfeatures,outlinebettertechniquesthatmorecloselyaligntobusinessrequirements, andpointoutwaysthatBackup2.0canmakebackupmanagementeasier,too.Illfinishby examiningspecificscenarioslikedomaincontrollers,publickeyinfrastructure(PKI),and Webserverswherethesetechniquesofferanadvantage.
40
Don Jones
Chapter3:WholeServerBackups
Recoveredfromthehorrorstoriesofthepreviouschapter?Readytostartensuringsolid backupsinyourenvironment,theBackup2.0way?Thatswhatthischapterisallabout, andwhatIcallwholeserverbackupsisdefinitelytherightplacetobegin.Thisiswhere Illaddressthemostcommonkindsofservers:fileservers,printservers,directoryservers, andevenWebserverstheworkhorsesoftheenterprise.Illshowyouwhatsomeofthe nativesolutionslooklike,discusssomeoftherelatedBackup1.0styletechniquesand scenarios,anddetailwhytheyjustdontcutitfortodaysbusinesses.ThenIllassemblea sortofBackup2.0wishlist:Allthethingsyouwantinyourenvironmentforbackupand recovery.Illoutlinewhichofthosethingsareavailabletoday,andwrapupbyapplying thosethingstosomerealworldserverrolestoshowhowthosenewtechniquesand technologiesimpactrealworldscenarios.
TheTechnicalDetails
Whatssotechnicalaboutwholeserverbackup?Windowsstorescriticaldatainanumber ofplaces,andsomeofthemarefilesanddatabasesthatarecontinuallyopenandunder modificationbytheoperatingsystem(OS):theWindowsregistry,ActiveDirectorys(ADs) database,andcertaincriticalOSfilesarejustafewexamplesofthese.Thesefilesare difficulttobackupandrestoresimplybecausetheOSitselfcantlockthefilestoprovide acleanimageofthefile.Inotherwords,becausethefilesareconstantlyopenand constantlychanging,traditionalbackupsolutionscanteasilyseethecompletefiletoback itup. Wholeserverbackupalsoincludesuserfiles,suchasthosestoredonafileserver,along withnumerousconfigurationdatabasesassociatedwithWindowsitself.Althoughallof thesemightnotbeopenatalltimes,theycanstillbetrickytobackupbecausetheymaybe openduringthebriefwindowwhenthebackupsoftwareisrunning. Sothegoalwithwholeserverbackupistogetagood,usablebackupoftheentireserver. Andbyusable,Imeanthatthebackupcanserveourmainbusinessgoalsrelatedto backupandrecovery,whichIstatedinChapter1: Backupsshouldpreventusfromlosinganydataorlosinganywork,andensurethat wealwayshaveaccesstoourdatawithaslittledowntimeaspossible.
41
Don Jones
Aslittledowntimeaspossiblesuggeststhatweneedtodomorethanjustbackuptheentire serverinawaythatfacilitatesrestoringtheentireserver;wemayalsoneedtorecovera singlefile,orasingleconfigurationdatabase,orasingleADobject.Beingabletorecover justthedatawewantfromabackupwillhelpreducedowntime,asrecoveringasinglefile isobviouslyorshouldbemuchfasterthanrecoveringanentireserver. Wealsohavetorecognizethatdowntimedoesntjustapplytotheserverthatwere recovering;italsoappliestothepeoplewhoarewaitingfortherecoverytobecomplete. Wecangetauserbacktoworkfasterbyrecoveringtheonefilethattheuserneedsrather thanhavingtorecovertheentireserver.Thatsaid,wellcertainlywanttheabilityto recovertheentireserver,intheeventthatacompletedisasteroccursandwelosethe entireserver.Whenthathappens,theidealistoloseaslittleworkaspossible,meaning whateverwereusingforbackupsshouldbecontinuous.
TraditionalTechniques
Thetechnicaldetailsherepresentonesignificanttechnicalchallenge:openfileaccess.That is,theabilityofabackupsolutiontogetacomplete,consistentbackupofafilethats currentlyopenandinuse.Overtheyears,anumberoftechniqueshavebeendevelopedto addresstheseissues. Onetechniqueiscalledalockedfilebackup(LFB),anditsafairlygenerictechniquethatsin useonmanyOSs.Basically,whenanapplicationrequestsabackupofafile,theLFBlogic whichmaybeembeddedintheOSorprovidedbyanagentofsomekindcheckstosee whethertheOShasthefileopenforanyotherapplications.Ifitdoes,theLFBlogicwaits forapauseinwriteactivitytothefileapauselastingforsomepredeterminedamountof time.Whenapauseoccurs,theLFBmakesacopyofthefileasitexistsatthetime,and offersthatcopytothebackupapplication.Insomeinstances,LFBwilloperateonanentire diskvolumeratherthanonaperfilebasis.Workingacrossanentirevolumehelpsensure thatthefilesareconsistentwithoneanotherbutalsomakesitmoredifficulttofindapause duringwhichalltheopenfilescanbecopiedandcached. WindowsintroducedanewtechniquewithitsVolumeShadowCopy(VSC)service.Ill discussVSCsoperationinmoredetaillaterinthischapter;fornow,sufficetosaythatthe VSCkeepsalocal,onvolumebackupofchangedfiles.Itcanthenofferthosefilestothe backupapplicationwhenafilescurrentversionisopenatthetimeofthebackup.That meansthebackupistypicallygettingthepreviousversionofafileratherthanthecurrent version;youmightsaythatbecausethefileisopen,thecurrentversionhasntyetbeen created. Note MicrosoftreferstothisfeatureasVSS,orVolumeShadowcopyService. Bothofthesetechniquesare,asyoucansee,prettycomplicated,andtheydontworkevery time.FilesthatareconstantlyopenwilldefeattechniquessuchasVSC,forexample.LFBcan bedefeatedbyfilesthatneverhavealongenoughpauseinwriteactivity,orinthecaseof wholevolumeLFBlogicbylarge,busyvolumesthatneverhavealongenoughvolume widepauseinwriteactivity. 42
Don Jones
NativeSolutions
IwanttotakealookatthenativebackupsolutionsthatarebuiltintotheWindowsOS. Therearereallytwo:thebundledbackupapplicationandtheaforementionedVSC.
WindowsBackup/WindowsServerBackup
PriortoWindowsServer2008,WindowsServercamewithafairlyprimitivebackupand restoreapplicationthatwasessentiallyunchangedsinceitsintroductioninWindowsNT 3.1intheearly1990s.Figure3.1showsNtbackup.exe,whichofferedbasicbackupand recoveryoffilesondiskand,asshown,ofthecriticalSystemState.
Figure3.1:WindowsBackup,orNtbackup. TheSystemStateconsistsofWindowsbootfiles,theWindowsregistry,COMclass registration(primarilythenamesandlocationsofDLLfiles),andotherconfigurationdata relatedtotheOSitself.Theapplicationcouldbackuptolocallyattachedtapedrivesortoa file,anditcouldrecovertheentireserverorindividualfiles.Ashortcominginthewhole serverrecoverymethodisthatyoufirsthadtoinstallWindows,thenusetheapplicationto recovertheserver.Inotherwords,theapplicationdidntsupportanykindofbaremetal recovery,wherethebackupcouldbeappliedtoabrandnewserverthathadnoOS installed.TheneedtomanuallyinstallWindowspriortobeginningtherecoveryaddeda fewhours,atbest,totherecoveryprocess,andmadethetoolunsuitableforallbutthevery smallestandrisktolerantenvironments.
43
Don Jones
Theapplicationalsolackedinternalschedulingofbackups.Instead,itusedtheWindows ScheduledTasksfunctionalitytoschedulethecommandlineversionofthetool.This providedthebasicabilityforschedulingbackups,butfromabusinessperspective,itwas suitableonlytotheverysmallestenvironments. Althoughvaluingthemanythirdpartyapplicationsthatspranguptofillthebackupand recoverygapleftbythenativeapplication,Microsofteventuallyrecognizedtheneedto providesomewhatmoremodernandsophisticatedbackupcapabilitiesintheOSsnative toolset.So,inWindowsServer2008aftermorethanadecadeofNtbackupMicrosoft introducedWindowsServerBackup.AsFigure3.2shows,thisfeaturemustbeexplicitly addedtotheserverbyanadministrator.
Figure3.2:AddingWindowsServerBackuptoaserver. Onceadded,theapplication,seeFigure3.3,offersanumberofimprovementsoverits predecessor,includingtheabilitytonativelyschedulebackupsandtheabilitytoconnectto remoteserversandmanagetheirbackups.Theapplicationcancreatearecoverydisc,which makesiteasierandfastertodobaremetalrecovery.
44
Don Jones
Figure3.3:WindowsServerBackupinWindowsServer2008. Asidefromimprovementsinitsuserinterface(UI)andtheadditionofnativescheduling, WindowsServerBackupisntsignificantlydifferentfromitspredecessor.Itstillworksona filebyfilebasis(althoughitbacksupentirevolumes),stillhastheabilitytobackupthe WindowsSystemState,andsoon.SomeoftheUIimprovementsaregreatsuchasthe abilitytorestorefromabackupfilethatislocatedonthenetworkbutinmanyways, WindowsServerBackupprovidesfeaturesthatthirdpartieswereprovidingadecade earlier. Note MicrosoftdoesntoversellWindowsServerBackup;itspecificallypositions thetoolasusefulforsmallerorganizationsorindividualswhoarenotIT professionals. YoucanreadmoreaboutWindowsServerBackupat http://technet.microsoft.com/enus/library/cc754572(WS.10).aspx. Bothofthesenativeapplicationshaveamajorshortcominginthattheyaresnapshotbased, meaningtheygrabbackupsonlyduringadesignatedbackupwindow.Anyfilesthat changedafterthebackupwasmade,andbeforethenextonewascompleted,wouldbelost intheeventofadisasterorotherproblem.
45
Don Jones
VolumeShadowCopy/PreviousVersions
IntroducedinWindowsServer2003,VSCisanative,OSlevelfeaturethatcreatesshadow copiesoffilesonanyvolumeforwhichthefeatureisenabled.Administratorsspecifya maximumamountofstoragetobeusedfortheshadowcache,andtheserviceautomatically makescopiesofsharesfilesthatarechangedaswellasfilesthatareopenwhenabackupis requested. AsFigure3.4shows,VSCisconfiguredonapervolumebasisontheserver.Onceenabled, itwillautomaticallybegincreatingshadowcopiesoffilesthatarecontainedwithinshared foldersonereasontheconfigurationUIdisplaysthenumberofactiveshares.VSCwill maintainseveralshadowcopiesforagivenfile,providingmultipleoldversionsofafile.It willcontinuemakingshadowcopiesuntilitsadministratorallottedspaceisfull,andwill thendiscardoldercopiestomakewayfornewones.
Figure3.4:ConfiguringVSC. Note WindowsServerBackupandlaterversionsofNtbackupincludesupportfor VSC,meaningtheycannativelymakeuseofVSCstorestobackup applicationsthatexposetheirdatathroughVSC.Suchapplicationsinclude MicrosoftSQLServer.
46
Don Jones
VSChastwodistinctfunctions: Itwillmakeshadowcopiesofopenfilesautomaticallywhenthosefilesareaccessed forbackup.ThisrequiresabackupapplicationthatisVSCaware,meaningthe backupapplicationhastoaskforashadowcopy.Someapplications,suchas MicrosoftSQLServer,aredesignedtoexposedatathroughVSC,providinga commoninterfacethatbackupapplicationscanusetoaccessapplicationspecific data. Itwillmakeshadowcopiesofsharedfiles,andmakethoseshadowcopiesaccessible toendusersthroughthePreviousVersionsfeatureinWindowsclientOSs,suchas WindowsVista.Thisprovidesenduserswithaselfservicemethodforrecovering individualfilesandfoldersthathavepreviousversionsavailableasshadowcopies. Figure3.5showsanexample.
Figure3.5:ThePreviousVersionsfeatureinWindowsVista.
47
Don Jones
VSChasafewshortcomings:First,itisntexactlycontinuous.Thefeaturedoesntmake backupsofafileeachtimethefileissaved,althoughitwillperiodicallymakeanewshadow copyofchangedfiles.VSCdoesntprovidefinegrainedcontrol,either.Inotherwords,an administratorcantdecidewhichfilesaremoreimportantandshouldberetainedlonger; VSCdiscardsshadowcopiesinafirstin,firstoutcycle;administratorscanonlydetermine thetotalamountofspaceavailableforallshadowcopies.VSCisntusefulinfullserver recovery,onlyinsinglefilebackupandrecovery.VSCdoesntcreateshadowcopiesoffiles thatarentshared,anditdoesntprotecttheentireOSitignoresSystemState,for example.Generallyspeaking,mostadministratorsregardVSCasagoodsupplementtoa properbackupapplication;VSCcanhelpreduceHelpdeskcallsforsinglefilerecoveryby makingthatfunctionalitymoreselfservice,butVSCitselfisnotabackupsolution.
TraditionalNonNativeSolutions
SincetheintroductionofWindowsserverOSs,theirfairlyweaknativebackupandrestore capabilitieshavespawnedanenormousecosystemofthirdpartysolutions.Foryears, however,thesesolutionsessentiallyreplicatedthebasicfeaturesofthenativebackup application.Tobesure,theyaddedagreatdealofadministrativeconvenienceand flexibility,buttheydidbasicallythesamejob.Figure3.6showsonesuchapplication.
Figure3.6:Anexampletraditionalthirdpartybackupapplication.
48
Don Jones
WithWindowsBackup,administratorscouldselectthefilestheywantedtobackup;with thirdpartysolutionsasshowntheydidbasicallythesamething.Thethirdparty solutioncouldcompressthebackedupdatafortransmissionacrossthenetworktoa centralbackupserverwithanattachedtapelibrary,ofcourse,whichisamajor improvement.Manythirdpartysolutionsprovidepowerfulschedulingcapabilities, designedtomaximizetheamountofdatathatcouldbecopiedinalimitedbackupwindow. Mostprovidedbackupmediamanagement,andmostprovidedapplicationspecificagents toincludeapplicationssuchasExchangeServer,SQLServer,andsoonintheirbackups. Mostprovidedsingleitemrecoveryaswellaswholeserverrecovery,andmostprovided themeanstoperformbaremetalrecoveryintheeventofacompletedisaster.Ultimately, though,theirrelianceonthesamebasicprinciplesandtechniquesasNtbackupalwaysled tothesamebasicproblemsandchallenges.Itswhatmakesthisentirecategoryofbackups feellikeBackup1.0.
ProblemsandChallenges
BoththenativeWindowsbackupcapabilitiesandthesimilarlydesignedtraditionalthird partysolutionshavethesamechallengesandproblems: Theyreschedulebased,meaningtheyresnapshotbased.Withoutcontinuousdata protection,yourealwaysatriskforlosingdata.Inmostcases,backupwindowsare intheevening,soyourealwaysatriskforlosinganentiredaysworthofwork. Mediamanagementisasignificantchallenge,andmanyadministratorsusingthese solutionsspendafewhourseachweekjustshufflingbackuptapes. Restorationisatimeconsumingprocess,asfileindexesanddatamustbeloaded fromtape.Evenrestoringasinglefilecantakealongtime:therighttapemustbe locatedandloadedandread,thefilemustbeselectedfromalist,thetapemustbe woundtothecorrectlocation,andfinallythefilecanberead.
Whatisitwewantfromourbackups,again? Backupsshouldpreventusfromlosinganydataorlosinganywork,andensurethat wealwayshaveaccesstoourdatawithaslittledowntimeaspossible. TheseBackup1.0techniquesmissoutonpreventingthelossofanydataoranywork; theresalwaysdataatriskbecauseofthesesolutionssnapshotbasis.Theydontensurewe alwayshaveaccesstoourdata,andthedowntimetheyoffercertainlyisntminimal.Even solutionsthateschewtapeinfavorofmoreexpensivediskbasedbackupstillhaveto performanincredibleamountoffileoperationstolocatetherightdataandbringitback intoproduction.
49
Don Jones
Theresadeeperproblemthattheseoldschooltechniquesalsotendtomiss:datade duplication.Todaysenterpriseshavealotofduplicateddataduplicatedfilesinusers homefolders,forexample.Traditionalbackupsolutionstendtoblindlycopyeverything theysee,meaningyourewastingnotonlyspacestoringthatduplicatedatabutalsotime andcapacitybackingitup.Datadeduplicationisstartingtobecomeawatchwordin storagemanagement,andaBackup2.0stylesolutionwouldcertainlyincludede duplicationcapabilities.
IntheOldDays
IwanttotakeabriefsectiontosummarizesomeofthekeyBackup1.0principlesand elementssothatwecanpullouttheirspecificproblemsandthinkofwaystosolvethemor improveuponthem.
BackupTechniques
ThesolutionsIvediscussedsofarrelyonasingleprimarytechniquewithafewsupporting ones.Mainly,thesesolutionsaresnapshotbased,meaningtheyseektobackupafileasit existsatthemoment.Theytypicallybackupgroupsoffilesduringasinglebackupwindow, whichisoftenintheeveningorduringotherperiodsoflowornoutilization.Supporting techniquesprimarilyrevolvearoundbackingupopenfiles,andmayincludeopenfile managementutilities,LFBagentsorfeatures,orspecialfeaturessuchasVSC. Recognizingthatsomeserversmaycontaintoomuchdatatobebackedupinasingle backupwindow,someorganizationsmaychoosetouseapartitioningscheme.For example,halftheserversdatamightbebackedupinfulloneevening,whiletheotherhalf receivesanincrementalordifferentialbackupthatevening.Thesemanagementtechniques helpmakethemostoflimitedbackupwindowsbutalsocomplicaterestoreprocedures. Themainproblemisthatbackupissnapshotbased,meaningthereisalwayssomedataat riskofloss.Asecondaryproblemisthatofduplicateddata,whichwastesbackupstorage space.
RestoreScenarios
Restorationtypicallyinvolvesrestoringasinglefileorotherresource,oftenattherequest ofauserwhoaccidentallychangedordeletedafile.Selfservicesolutionssuchas WindowsPreviousVersionsfeature(inconjunctionwithVSC)workwellwhenthedesired dataisavailableforrestoration;whensuchfeaturesarentavailableortheneededdata isntavailable,administratorsmustturntotheirbackupsolution.Thistypicallyinvolves identifyingthecorrectversionoftheresourcetoberestored,locatingthestoragemedia containingthatbackup,andthenreadingthedatafromthatmediawhich,inthecaseof tape,canrequiresometime,asthetapedoesntsupportrandomaccessandmustbewound tothecorrectreadlocation.
50
Don Jones
Themainproblemisinidentifyingtheneededdataandlocatingthemediaonwhichitis stored.Withcomplicatedbackupschemes,thismayinvolveidentifyingafullbackup,one ormoreincrementalbackups,and/oradifferentialbackup.Anotherproblemisinthetime ittakestoperformtherestore,particularlywithtapebasedbackups.Afinalproblem relatestothesnapshotbasednatureofthebackup,meaningtherewillbetimeswhenthe desireddatasimplyisntavailableonabackup.
DisasterRecovery
Wholeserverdisasterrecoveryusuallycomesinoneoftwoforms: ThebaseOSmustbeinstalledfirst,possiblyalongwithanyrecentservicepacks, usingstandardinstallationprocedures.Then,oneormoresoftwareapplications mustbeinstalledtosupporttherecovery.Finally,therecoverycanbegininearnest, readingdatafromthebackupmediaandrestoringittotheserver. Abaremetalrecoveryusuallyinvolvesaspecialbootdiscthatincludesastripped downOS,suchasWinPEorWindowsRecoveryEnvironment,thatcontainsenough smartstoaccessbackedupdata,formattheserversdisks,andcopythebackup datatothosedisks.Thisformusuallyinvolvesfewerstepsandismuchfasterthan thepreviousform.
Bothtechniquescanbetimeconsumingandsufferfromtheinherentsnapshotbased natureofthebackup,meaningtherewillalwaysbesomedatamissingeveninaperfectly executedrecovery.
BackupManagement
BackupmanagementcanbecomplicatedusingtheseBackup1.0styletechniques.Backup schedulesandtypes(full,incremental,ordifferential)canbecomplex,andrequirecareful managementnotonlyofschedulesbutalsoofstorageresources,networkcapacity,andso forth.Physicallymanagingtapesrotatingthemoffsite,verifyingtheiraccuracy,andso forthcanbetimeconsuminganderrorprone. Thesedays,backupmanagementiscomplicatedbymanycompaniesdataretention requirements.Dothebackupsincluderegulatedinformation?Inthatcase,theymayneed toberetainedforaspecificperiodoftimeordiscardedafteraspecificperiodoftime.They mayneedspecialsecurityprecautions,aswell,toprotectthedatacontainedinthebackup.
51
Don Jones
RethinkingServerBackups:AWishList
LetsstartthinkingBackup2.0:Whatareouroldwholeserverbackuptechniquesmissing thatwedliketoadd?Theskysthelimit;atthispoint,wedontneedtoworryaboutwhats possibleoravailablejustwhat,inaperfectworld,wedbeabletoimprove.
NewandBetterTechniques
IthinkthemostimportantimprovementthatBackup2.0canofferisachangefrompoint intimesnapshotbackupstocontinuousdataprotection.Hereshowitmightwork: Everythingstartswhenanapplicationofanykindmakesachangetodisk.Applications dosobypassingdatatotheWindowsOSsfilesystemapplicationprogramminginterfaces (APIs),basicallyaskingWindowstosavethisdatatothisfile.Whenthathappens, Windowsfilesystemtakesthedataandbeginsbreakingitintoblocks. Blocksarethebasicunitofmanagementfordataonadisk.Whenyouformatadisk,you selecttheallocationunitorblocksize.Thatdetermineshowmuchdataiscontained withinagivenblockofdiskspace.Figure3.7showsthatWindowsusuallypicksitsown defaultvalue,whichiswhatmostpeoplegowith.
Figure3.7:Formattingadiskandselectingblocksize. Asingleblockcanholdthecontentsonlyforasinglefile.Ifyourblocksizeis2kilobytes,for example,andyousavea512bytefile,thenthreequartersoftheblockwillbeemptyand wasted.Largerfilesaresplitacrossmultipleblocks,andNTFSkeepstrackofwhichblocks gowhere:fileXYZ.txtconsistsofblock324,thenblock325,thenblock789,andsoforth.
52
Don Jones
Microsoftrecognizesthatthirdpartyutilitiesmightneedtobenotifiedoffileoperations, andmightinfactneedtomodifyorcancelthoseoperations.Thatshowmostthirdparty diskquotasystemsandfilefilteringsolutionswork:theygetthefilesystemtotellthem whenfilesarechangedondisk,andtheyupdatetheirquotadatabaseorblockfilesfrom beingsaved,orwhatever.ThetechniqueMicrosoftprovidesforaccomplishingthisiscalled afilesystemfilterorshim.Essentially,theshimregistersitselfwiththeOS,isnotifiedoffile operations,andisgiventhechancetoblockoralloweachoperation. InthecaseofaBackup2.0stylesolution,theshimmightjustpayattentiontowhichdisk blockswerebeingmodified.Asblocksaremodifiedondisk,theshimcouldcopytheblocks data,compressit,andtransmititacrossthenetworktoabackupserver.Figure3.8 illustratestheprocessImproposing:
Figure3.8:Capturingchangeddiskblocks. 53
Don Jones
InStep1,somethingontheservermodifiesablockofdiskspace.InStep2,thischangeis passedtothebackupsoftwaresfilesystemshim,andthemodifiedblockiscopiedand transmittedtoabackupserver. Thistechniquewouldallowforcontinuousdataprotection.Ofcourse,inadditiontojust copyingblocks,thesoftwarewouldalsomakeanoteofwhichfiletheblockwentwith.On thebackupserver,softwarewouldkeeptrackoffilesandtheirassociatedblocks.Thisisa powerfultechnique:Ratherthanmessingaroundwithcumbersomeandcomplicatedopen filetechniques,asBackup1.0solutionswoulddo,thissystemisgrabbingthelowleveldata changesastheyarephysicallyinscribedontheharddiskbytheOS.Thedataisbeing grabbedbelowthelevelofanentirefile,soevenpartialchangestoafilesuchasa MicrosoftAccessdatabasecanbegrabbedimmediately,almostinrealtime,andthe backupsolutiondoesntcareifthefilehappenstobeopenornotatthetime.When someoneneedstorestoreafile,thebackupserverssoftwaresimplylooksupthemost recentlysavedblocksforthefile,andusesthemtoreconstructthefileinitsmostrecent condition.Bestyet,thecentralbackupservercouldalsosavepastcopiesofafilesblocks meaningitcouldreconstructthefileasitexistedinanyparticularpointintime. Ifthesavedblockswerestoredonahighspeedstoragesystem,suchasaRAIDarray,files couldbereconstructedalmostinstantlyandsavedtoanylocationonthenetwork. Consideringourprimedirectiveforbackups: Backupsshouldpreventusfromlosinganydataorlosinganywork,andensurethat wealwayshaveaccesstoourdatawithaslittledowntimeaspossible. ThisBackup2.0techniquewoulddothetrick.Wereallywouldntloseanydataorwork, becauselowlevelchangeswouldbecapturedcontinuously.Wewouldntneedtoworry aboutbackupwindowsbecausetheentiredaywouldbeourbackupwindow.Wedalways beabletoreconstructdataasneeded,withminimaldowntime. Note Infact,solutionsexistthatdojustthis:Althoughtheymayusedifferent underthehoodtechniquesthanthefilesystemshimIvedescribed,the Backup2.0ideaofcopyingblocksnearlyinstantlyisverymucharealthing, notjustawish. Thistechniquealsohastheadvantageofbeingabletocaptureeverythingthatgoestodisk, includingfilepermissions,alternateNTFSstreams,registrychanges,ADchanges,and more,allwithoutnecessarilyneedinganyspecialknowledgeoffiletypesorstructures. Infact,thistechniqueislogically(althoughnotphysically)similartoRAID1diskmirroring, whichalsooperatesatablocklevelalbeitnormallyviaaRAIDcontrollercardandnotvia software,althoughsoftwareRAID1isalsopossible.Ratherthanmirroringblocksinreal timetoaseparatedisk,thisbackuptechniquemirrorstheblocksacrossthenetworktoa backupserver.And,ratherthanonlykeepingthecurrentblockdata,asinadiskmirror,the backupsolutioncankeeppastcopies,too,enablingrecoverytoanyearlierpointintime.
54
Don Jones
BetterRestoreScenarios
Thetricktoallofthis,ofcourse,ishavinggreatmanagementsoftwareonthatcentral backupserver.Itneedstobeabletotrackwhichchangedblocksgowithwhichfiles,for example,andforconveniencemaywanttokeeptrackofspecialfilesliketheWindows registryorADdatabases. ThebackupserversoftwarecouldeasilyexposeaselfserviceUI,ifdesired,toprovide functionalitysimilartobutmorecontrollablethanWindowsPreviousVersion/VSC feature.ThesoftwarecouldmoreeasilyrecoverdatainapplicationssuchasADbecause thesoftwarecouldreconstructaportionofthedatabasefileoreventheentiredatabase file,downtoaspecificpointintime(inallprobability,youwouldwantthesoftwaretobe abletoattachpointsintimetospecificADoperations,suchasthedeletionofauserorthe creationofagrouporwhateverjustsoyoucouldmoreeasilyidentifythepointintime youwanttorollbackto).
BetterDisasterRecovery
Withacopyofeveryblockofdiskspace,restoringanentireserverwouldbe straightforward:Youdneedsomesortofrecoverydisk,suchasabootableDVD,togetthe serverupandrunningandtotalktothebackupserversoftware.Youdthensimplystream themostrecentversionofeverydiskblockbacktotheserver,writethoseblockstodisk, andthenrestarttheservernormally.Withgoodcompressiontospeedupthetransferof dataacrossthenetwork,baremetalrecoverycouldbedonequicklyandyoudlosevery little,ifany,data. Infact,theopportunityexiststorecovertheservertoavirtualserver,ifneedbean excellentdisasterrecoveryscenarioaswellasapowerfulphysicaltovirtualmigration technique.Youmightwellbeabletorecovertodissimilarhardware,too,sinceinmany casesWindowscanreadjustitselfwhenitfindsitselfsuddenlyrunningondifferent hardware. Tip Asyoubeginlookingatblockbasedrecoverysolutions,askfordetailson howtheydealwithdissimilarbaremetalrecoveryscenarios.Would Windowsrequirereactivationwhenitfindsitselfrunningondifferent hardware?Howdissimilarcanthehardwarebe?Themoreflexibilitythe solutionoffers,thegreaterthenumberofscenarioswhenitwillbeableto savethedaywhenitsneeded. Heresakillerscenario:Imaginegrabbingrealtimediskblocksfromaproductionserver, andthenimmediatelyapplying(restoring)thoseblockstoavirtualmachine.Instant virtualstandby!Ifthemainproductionserverfails,thevirtualservercanstepinandtake overwithlittledowntimeandlittleornodataloss.Dependingonhowyouarchitectthe virtualinfrastructure,asinglevirtualhostmightsupportseveralvirtualstandbys.Those standbysmightoperatewithlessperformancethanthenonvirtualproductionmachines (again,dependingonhowyousetthingsup),butyoudhaveahotspareanytimeyou neededon.Figure3.9illustratesthisidea.
55
Don Jones
Figure3.9:Blockbasedvirtualhotspares.
EasierManagement
Managingblocklevelbackupscanbemucheasierbecausetheylltypicallybestoredfirst ondisk.Youcanthenmaketapebasedbackupsfromthediskbasedbackupsmeaning yourproductionserversdontparticipateinthetapebasedbackup,andyourbackup windowcanbeaslongasyoulike.Tapebackupswouldtrulyrepresenttimespans,and couldbecompleteandinternallyconsistentunliketodaysmixoffull,incremental,and differentialbackups,whichmustbetreatedasasetinordertoretaintheirusefulness. Blockbasedbackupswouldalsobeaneffectivewaytoimplementdatadeduplication,as setsofblockscouldeasilybeindexedandcomparedtocheckforduplication.Thatwould helpcutdownbothondiskstorageandtapestorageaswellasallthemanagement overheadthatcomeswithanykindofstorageresource.
56
Don Jones
Note Somedatadeduplicationvendorsclaimthatyoucanreducethesizeofyour backupsbyupto70%.Evenifthatsanoptimisticnumber,youcould conservativelyexpecttosaveasignificantamountofspace! Thebackupsoftwarecouldalso,intheory,allowyoutomountbackedupdataasarealdisk volume.ItwouldsimplyneedtoprovidesomekindofdiskdriverforWindowsthatcould talktothebackupsoftwareandreassemble,inrealtime,themostrecentbackedupblocks intoasinglediskvolume.Youcouldthenmountandexplorebackedupdataaseasilyas livedata,allowingyoutocomparefiles,draganddropfilestodifferentlocations,andso forth.Ifthebackupsolutionwasstoringpastversionsofblockdata,youcouldmountadisk thatresembledanygivenpointintime,makingiteasytocomparefilesordatafrom differentpointsintime.
Greatfor
Again,mostofthecapabilitiesIvewishedforareavailabletodayfromavarietyofthird partyvendors.Backup2.0forentireserversisntsomethingyouhavetowishfor;its somethingyoucanchoosetodonow,helpingyouachievethecapabilitiesthatbackups havealwaysallegedtoprovide: Backupsshouldpreventusfromlosinganydataorlosinganywork,andensurethat wealwayshaveaccesstoourdatawithaslittledowntimeaspossible. ThefollowingsectionshighlightafewspecificscenarioswheretheseBackup2.0 techniquescouldsavetheday.
DomainControllers
Today,companiesspendthousandsonADbackupsolutionsthatcreatepointintime backupsofADandallowforsingleobjectrecoveryaswellasmorecompletewhole directorydisasterrecovery.ABackup2.0stylesolution,however,wouldbeabletonatively backupAD,becauseintheend,ADscomplexdatabaseisjustblocksondisk.Certainly,a backupsolutionwouldneedtoofferADspecificfunctionalityforrestores,ormighteven offeranADmanagementconsoleextensionthatmadeADrecoverycapabilitiesavailable rightfromthatconsole.Butwhenyoustartusingblocklevelbackups,ADsuddenlyisntso difficulttobackup.Youcangetrealtimebackupsofeverychange,automatically,withno extraeffort.
57
Don Jones
InfrastructureServers
InfrastructureserversareperfectforBackup2.0:YoullcaptureeveryDNSchange,every DHCPchange,andmoreonserversthatyoumightonlybackuponceaweekinthe Backup1.0world.Whybotherwithrealtimebackupsonaninfrastructureserver? Ifyouhavetodoabaremetalrecovery,thingslikeDHCPleaseswillstillbeintact yournetworkwonthavetogothroughaperiodofrecoveryandreadjustment. BothstaticanddynamicDNSrecordswillbemaintainedagaineliminatingthe periodofconfusionthatnormallyoccurswhenaDNSservercrashesandisbrought backonline. StillusingWINS?Eventhatdatabasecanbebackedupinrealtimeandrecoveredto aspecificpointintimehelpingeliminatetheneedforyournetworktostart massivebroadcastsandreregistrations,aswouldhappenwhenanemptyoroutof dateWINSdatabasewasrecovered. Infrastructureserverscanbeeasilyrestoredtoavirtualmachineintheeventofa completedisasterevenanoffsitevirtualmachine,makingiteasiertorecreate yourexactproductioninfrastructure,virtually,inadisasterrecoverymode.
WebServers
Webserversarejustfilesandfolders,right?Whynotcaptureeverychange,andmakeit easiertorollbackanentireWebsitetoaknowngoodpointintime,includingWebserver configurationfilesandmetadata?ItdoesntmatterifyoureusingIISorApacheor somethingelseasyourWebserver,itsallblocksondisk,andaBackup2.0solutioncan grabitall. Backup2.0canevenhelpmakeWebfarmmanagementeasier.DesignateoneWebserver asyourmasterperhapsitsevenaprotectedserverthatdoesntacceptlivetraffic.Back thatupinrealtimeusingaBackup2.0stylesolution,capturingnewlyuploadedfilesfrom yourWebdevelopersaswellasconfigurationchangesfromWebmasters.Restorethose changestomultiplehotsparesinyourWebfarmwithnoeffortonyourpart,every memberofyourWebfarmnowhasidenticalWebcontentandserverconfigurationfiles! Yourmastercontentwouldremainprotected,soevenifoneofyourWebserverswas compromised,youcaneasilyandquicklyrestoreittothemostrecent,correctversion, bringingitbackintoproductionquicklyandeffortlessly. WanttorebuildyourWebfarmasvirtualizedservers?Noproblem:JustuseyourBackup 2.0solutiontorestoreyourmasterWebservertooneormorevirtualmachines. Reconfigureafewsettingssuchascomputernames,reconfigureyourloadbalancerto pointtothenewservers,andyourWebfarmismoved.Withtherightmanagementtoolson topofthebasicdiskblockbasedbackupsystem,itsallpossibleanditbringsvalueto yourbackupsolutionthatextendsbeyondmerebackupandrecovery.
58
Don Jones
PublicKeyInfrastructure
IfyouoperateaPublicKeyInfrastructure(PKI),youknowhowcriticalthoseCertification Authority(CA)serversare.Withdiskblockbasedbackups,youllalwayshaveareliable backupofeveryCAincludingeverycertificate,everypublickey,everyrevocation,and everyoutstandingcertificaterequest.IntheworstcasescenarioofacompleteCAfailure, youcanstillbringthePKIbackupquickly,eitherbyrestoringthemostrecentdiskblocks totheoriginalhardware,todissimilarhardware,oreventoavirtualserver.
ComingUpNext
Inthenextchapter,Illbetakingthesameapproachasthisonebutfocusingexclusivelyon MicrosoftExchangeServer.TheresnoquestionthatExchangeformsabigpartofmost organizationsmissioncriticallist,andhavingsolid,reliableExchangebackupsisequally critical.ButExchangecertainlymakesbackupandrecoveryabitmorechallenging,asyou needtosupportahighlevelofgranularityfromsinglemessagerecoverytobaremetal serverrestores.YouvegottodoallthatdespitethefactthatExchangesdatabaseis essentiallyabigblackbox,notabunchofmoreeasilymanipulatedlittlefiles.Itsarealtest oftheBackup2.0methodology.
59
Don Jones
Chapter4:ExchangeServerBackups
Askanyoneintheorganizationwhattheirmostmissioncriticalpieceofinfrastructureis, andyoullprobablyhearemailasacommonanswer.Oryoumightnot:Manyfolkstake emailforgranted,althoughtheyexpectittobeasavailableandreliableasatelephonedial tone.Userswhohaveneversufferedanemailoutagealmostcantimaginedoingso;once theydoexperienceanoutage,theymakesureeveryoneknowshowmuchtheyresuffering. Asoneofthemostpopularsolutionsforcorporateemail,ExchangeServeroccupiesa specialplaceinyourinfrastructure.Itsexpectedtobealwayson,alwaysavailable,and alwaysreliable.Disasterssimplycantbetolerated.Whatsmore,usersownmistakesand negligencebecomeverymuchyourproblem,meaningyouhavetoofferrecoveryservices thatarequickandeffective,evenwhenyourerecoveringsomethingthatausermistakenly deletedontheirown.
NativeSolutions
ExchangeServersnativebackupandrestorecapabilitiesaretiedinparttotheunderlying Windowsoperatingsystems(OSs)capabilitieswhichisntalwaysagoodthing.Partof Exchangesrecoverycapabilitiescomefromthefactthatdeletedmessagesarentactually deletedfromthesystem.EmailclientssuchasMicrosoftOutlookautomaticallymove deletedmessagesintoaRecycleBin,wheretheystayforaconfigurableperiodoftimeor untiltheusermanuallyemptiestheRecycleBin.Whenusingotheremailclients,suchasa genericIMAPclient,deletedmessagesareretainedontheExchangeServercomputereven iftheyrenotactuallymovedtotheRecycleBin;deletedmessagesaresimplyleftintheir originalfolderandhiddenfromtheusersviewuntilaconfigurableamountoftimehas passed,oruntiltheuserspecificallypurgesdeletedmessagesaspartofacleanup operation.AsFigure4.1shows,Outlookactuallydisplaysthesedeletedinplacemessages inaspecialfontratherthanhidingthemcompletely,illustratinghowIMAPmessagesare leftinplacewhendeleted.
Figure4.1:DeletedmessagesinMicrosoftOutlook.
60
Don Jones
Allofthisfunctionalityisdesignedtoprovideuserswithaselfservicerecoveryoption:If theyaccidentallydeleteamessage,theycaneitherundeleteitorretrieveitfromthe RecycleBin. OnceamessagehaspassedbeyondtherealmoftheRecycleBinorotherundeleteoptions, recoveringamessagebecomesyourproblem.Unfortunately,ExchangeServerdoesnt includeanybuiltinbackupandrecoverymechanisms;itdoesincludeanapplication programminginterface(API)thatallowsotherapplicationstointeractwithExchange Servertomakebackupsandperformrestores.ExchangeprovidessupportforWindows VolumeShadowService(whichIdiscussedinthepreviouschapter)forbackups,toothe WindowsnativeBackuputility,includedwithWindowsserver2003,canusetheVolume ShadowServiceinterfacetomakebackupsofExchange. Note Inwhatmusthavebeenamiscommunicationbetweenproductteams,the newWindowsServerBackuputilityincludedinWindowsServer2008does notoffersupportforExchangeServerbackups,meaningyouhavenonative optionforproducingbackupsofyourExchangedata.Microsoftdoesofferan additionalcostbackupsolutionthatsupportsExchangebackups,and obviouslynumerousthirdpartiesprovidevaryinglevelsofsupportfor Exchangebackups.AsofExchangeServerServicePack2,Exchangeincludes anewpluginthatdoesenableWindowsServerBackuptonativelybackup andrestoreExchangedatabases. AsofExchangeServer2007,ExchangeincludesafeaturecalledClusteredContinuous Replication.CCRisdesignedtoreplicateExchangedatabasetransactionstheindividual changesthataremadetothedatabasetoaseparateExchangeServercomputer.There arespecifichardware,software,andenvironmentalrequirementstomakeCCRwork,andit doesrequirethatyouhaveadditionalExchangeServercomputersintheenvironment.CCR isMicrosoftspreferredsolutionforwholeserverrecoverybecauseitessentiallykeepsa sparecopyoftheExchangedatabaseonaseparatemachine.ThecostsinvolvedinCCRcan bequitehigh,however,becauseyourebasicallymaintainingacomplete,spareExchange Servermachinehardwareandall,unlessyourspareisvirtualizedjustsittingaround waitingforthefirstservertofail. Figure4.2showswhatCCRlookslike;noticethatathirdwitnessexistshere.Thewitness jobistomakesurethattheprimaryactivenodeisworking;ifitisnt,thewitnessiskeyin makinganautomaticfailovertothepassivenodehappen.
61
Don Jones
Figure4.2:CCR. AvariationofCCRisLocalContinuousReplication;LCRdiffersinthatitusestransaction replicationtocreateacopyoftheExchangedatabaseonthelocalserver,onaseparateset ofdisks.Thisgivesyouacopyofthedatabasewithouttheneedforaseparateserver, althoughyourExchangeServerhardwareobviouslyremainsasinglepointoffailureinthat scenario.LCRislessexpensivethanCCRbutdoesrequireextralocallyattachedstorageon theExchangeServercomputer. CCRisprimarilyusefulforwholeserverfailuresafullondisaster,inotherwords(LCRis onlyusefulatprotectingagainstadatabasefailureiftheentireserverfails,youlosethe LCRreplica,too).Neitherofthesereplicationtechniquesisespeciallyusefulforrecovering singlemessages.Infact,neitherWindowsnorExchangeoffersaparticularlyeffective solutionforsinglemessagerecovery.Instead,theyrelyentirelyontheRecycleBin functionalityimplementedinOutlook,themostcommonlyusedExchangeclient application.
62
Don Jones
Exchangeoffersuniqueproblemsandchallengesinthebackuparena.Illdiscussmanyof theseinmoredetaillaterinthischapter,butforrightnow,Iwanttobrieflyintroducethem fromabusiness,ratherthanatechnology,perspective.
ABitAboutHowExchangeServerWorks
MostofthesederivefromExchangesarchitecture,soitsworthtalkingabitabouthowthat architectureworks.Exchangeisbuiltaroundatransactionaldatabase.Inthisregard, ExchangeissimilartoSQLServer,althoughtheunderlyingstructureofExchangedatabases isverydifferentfromtherelationalonesfoundinSQLServer.Transactionalmeanssome importantthings: Whenchangesaremadetothedatabase,anoteofthosechangesisfirstmadeinthe transactionlog,aspecialfilemanagedbyExchangeServer.Changesmightinclude newmessages,incomingmessages,orevenchangestomessagessuchasmarking amessageashavingbeenreadorrepliedto.Thetransactionlogisbasicallyatodo listmarkmessage#164783asreadorinsertnewmessage#7847829withthe followingcontents. Afternotingthechangeinthetransactionlog,Exchangesearchesitsdatabasefor thebitsofdatathatactuallyneedchanged.Thosebitsareloadedintotheservers memory,andthechangesareappliedtothedatainmemory. Afteracertainamountoftime,thechangeddataiswrittenbacktothedatabaseon disk.Thishappensquickly,butoftennotimmediately;Exchangemaychooseto cachedatainmemorysothatmultiplechangescanbeappliedinsuccession. Oncechangeddataissuccessfullywrittenbacktothedatabaseondisk,Exchange goestothetransactionlogandchecksoffthechangeashavingbeencompleted andcommitted.
IftheExchangeServercomputercrashesorlosespower,noworkislostprovidedthe transactionlogisintact.Eventhoughchangesexistingonlyinmemorymaynothavebeen safelywrittentodisk,Exchangecangobacktothetransactionlogandsimplyredo,or replay,anytransactionsnotcheckedoffascommitted.Thistransactionlogalsoprovides thebasisforLCRandCCR:IndividualtransactionsarereplicatedfromtheExchangeServer logandreplayed,creatinganexactduplicatedatabase.
63
Don Jones
WhyExchangeServerBackupsCanbeTricky
SotheExchangedatabasefilesareactuallytheresultofmillionsoftransactionsbeing appliedinmemoryandthensavedtodisk.Thedatabaseisobviouslyindexedsothat individualmessagescanbefoundeasily,andagreatdealofstructureddatasuchaswhich messagesbelongtowhichusersisstoredinthedatabasealongwithactualmessagedata. Normally,whileExchangeServerisrunning,onlyitsprocesseshaveaccesstothedatabase. Allofthisconspirestomakecertainbackupandrecoverytasksabitmorecomplicated: IndividualItemRecovery.Becauseindividualmessagesarestoredinamonolithic databaseratherthanasindividualfilesondisk(asisthecasewithmostUnixbased mailapplications),recoveringanindividualmessagecanbetough.Youmight,for example,havetoretrievetherightwholedatabasebackup,thendiveintoitas Exchangewouldtofindthemessageormessagesyoureafter. DataCorruption.Theslightestdatacorruptiononabackupcanrendertheentire databaseuselessmeaningwhateveryoureusingtomakebackupshastobe100% reliableandcapableofdetectingandrepairingdatacorruption. DataDeDuplication.Exchangeactuallyhasbuiltindatadeduplicationofasort. Individualmessages(and,ifconfigured,attachments)addressedtomultipleusers arestoredonlyonceinthedatabase,providedalltheusersmailboxesareinthe samemessagestore.Thus,backupsoftwaregetstheadvantageofthisde duplication,creatingsmallerbackupsthanifeachmessagewasextracted individually. However,inordertofacilitatesinglemessagerecovery,manybackupsolutionsnot onlybackuptheentiredatabasebutalsoextractindividualmessagesandbackup thoseindependently.Thisletsthemindex,search,andrecoverindividual messagesbutwithsomesolutionslosesthebuiltindeduplicationandincreases thesizeofthebackupdata. SearchandeDiscovery.Exchangedoesnthavegoodbuiltincapabilitiesfor searchingacrosstheentiredatabase,whichisoftenneededwhenyouneedto recoveramessageorperformeDiscoveryforlegalpurposes.Becausemanybackup solutionsgrabtheentireExchangedatabase,theyreoftenincapableofsearching throughthatdatabaseeither.
Obviously,someofthesespecialconcernsareoneswellhavetoaddressinaBackup2.0 solution.
64
Don Jones
IntheOldDays
IwanttolookathowoldstyleBackup1.0solutionsaddressboththebasicandspecial needsofExchangeServerrecovery.Itsimportanttorecognizewhatworksandwhat doesntinthesetraditionalsolutionssothatwecanidentifyareasforimprovementaswell asareasthatshouldberetainedinanewstyle,Backup2.0solution.
BackupTechniques
ExchangeServerbackupscanbecomplicatedfromaprocessviewpoint.ConsiderFigure 4.3,whichisexcerptedfromablogentryathttp://blog.thejpages.com/2008/03/18/why arewestillbackingupexchange.aspx.TheauthorproposesthatExchangeServernotbe backedup.Instead,hesuggestsenablingcircularloggingmeaningExchangestransaction logwillautomaticallyoverwriteolderentriesasneededtowritenewones.UsingCCR,the Exchangedatabaseisreplicatedinthisproposaltotwopassivenodes,makingtwo completecopiesofthedatabase.Onepassivenodesitsinthedatacenter,readytotake overintheeventofafailureintheproductionExchangecomputer.Theotherpassivenode isinanoffsitelocationanddelaysreplayingincomingtransactionsby7daysmeaningits copyofthedatabaseisalways7daysold.
Figure4.3:ProposaltonotbackupExchange.
65
Don Jones
Therearesomedownsidestothisapproach.Althoughitprovidesgreatalmostinstant recoverycapabilitieswithverylittlelostdata,itwouldbeveryexpensivetoimplement. YourelookingattwoadditionalExchangeServerlicenses,eveniftheyreonvirtual machines.Iftheyrenotvirtualized,thatstwoadditionalExchangeServermachines,too, andagreatdealofnetworkbandwidth.Youdstillhavetohavesomekindofbackup runningtosupportuserswhodeletemessagesandwantthembackintheproposal, Exchangeretainsdeleteditemsforjust30daysandmanyorganizationsmustduetolegal orindustryregulationsretainmessagesforafarlongerperiodoftime. TraditionalExchangebackupstypicallyseektograbtheentiredatabase,usuallyby connectingtoExchangeServersVolumeShadowServiceAPI.Asmentionedearlier,these solutionsmayalsoextractindividualmessagesthroughotherAPIs,givingthemnotonlya completecopyofthedatabaseusedfordisasterrecoverybutalsoaccesstoindividual messages.
RestoreScenarios
ThemostcommonrestorescenariosinExchangearesinglemessagerecoveryorsingle mailbox(includingallofitsmessages)recovery.Anarticleat http://www.msexchange.org/tutorials/ExMergeRecoverMailbox.htmldetailsafairly commonwayofdoingthisinExchangeServer2003:StartbyinstallingthefreeExMerge utility(availablefromhttp://microsoft.com/download).Restoreyourdatabasebackup fromtapeorwhateveryoumaywinduprestoringittoadifferentExchangeServerso thatyourenotaffectingyourproductionserver.AsFigure4.4shows,youllbeabletouse ExMergetoexportthedesiredmailboxtoaPSTfile,whichcanbeopenedwithMicrosoft Outlook.Ifyouwanttorecoverasinglemessage,youattachthatPSTtoanOutlookclient andgohuntingforthemessageyouwant.MessagescanbedraggedoutofthePSTfile,via Outlook,anddroppedintoanactiveExchangemailboxtogetthemessagebackontothe server.
66
Don Jones
Figure4.4:RecoveringasinglemailboxinExMerge. NewerversionsofExchangeofferaMailboxRecoveryCenterthatperformsessentiallythe sametaskwithouttheneedforExMerge.Youstillhavetoaddastoragegrouptothe RecoveryCenter,restoreanExchangedatabase,mountthedatabaseintotheRecovery Centerstoragegroup,thengobrowsingforthemailboxyouwanttorecover.Figure4.5 showswhattheRecoveryCenterlookslike.Itsstilltimeconsuming,butperhapsprettier thanExMerge.
67
Don Jones
Figure4.5:MailboxRecoveryCenter. Frankly,thiswholeprocessisnuts.Itsslow,cumbersome,andincrediblylaborintensive foradministrators.ThisiswhyIveneverbeeninasingleenvironmentthatdoesnthave somekindofthirdpartybackupsolutionoftenjusttoprovidemoreefficientsingle messageandsinglemailboxrecovery. Thirdpartybackupsolutionsworkmorequicklybutinvolvesubstantiallythesame process.Yougogetyourbackupdataoffoftapewhichwilltakesometimebecauseyou mighthavetorestoreafullbackupandmultipleincrementalordifferentialbackupstore createthepointintimeyouwanttorestorefrom.Thebackupsoftwareusuallymaintains itsownindexesofavailablemessagesandmailboxes,soyoubrowseorsearchthroughthat untilyoufindwhatyouwant.Whatthesolutiontypicallyautomatesisthepainof extractingthemailboxormessagefromthedatabaseandputtingitbackintoExchange Serversotheprocessislesslaborintensivefortheadministratorbutstillpretty awkwardandnotnecessarilyveryfast.
68
Don Jones
DisasterRecovery
DisasterrecoveryinExchangeisstraightforwardandusuallytimeconsuming.You restorethemostrecentfullbackup.Yourestoreanyincrementalordifferentialbackups madesincethen.Yourestoreanytransactionlogbackupsmadesincethen.Finally,you standbackandletExchangesortitalloutandbepreparedtowaitbecausetheprocess cantakehours.ThedevelopmentofCCRandLCRtechnologieswasdriveninlargepartby thetimeconsumingnatureofmoretraditionalbackups;withCCRorLCR,youvegota sparedatabasesittingrightthere,readytobeusedprovidedwhatyoureafteristhemost recentdatabase.Intruedisasterrecoverysituationsatotalhardwarefailureorevena datacenterdisasteryouusuallydowanttogetthelatestversionofthedatabasebackup andrunningquickly. WhereCCRandLCRfailisifsomethinggoeswrongthatgetsreplicatedinthosecases,the copyofthedatabasewillalsocontainwhateverwentwrong,andyoullbebacktotime consumingtaperestorestorebuildyourdatabases.
BackupManagement
Dependingonthebackupsolutionyoureusing,managingtraditionalbackupscanbequite ascience.BecauseExchangedatabasescangrowquitelarge,someorganizationsdonthave thetimetograbafulldatabasebackupasoftenastheydlike.Thatmeansyourestuck managingfullbackups,incrementalbackups,differentialbackups,andinmanycases,log backupsbackupsoftheExchangetransactionlog. Infact,managingtransactionlogbackupsiscrucialtominimizingdatalossintheeventofa failure.ThetransactionlogliterallycontainseverysinglepieceofworkthatExchange needstodo;Exchangesabilitytoreplaythatlogtorecreateworkisaneffectiverecovery technique.Someorganizationswillgrabtransactionlogbackupsthroughouttheday;many thirdpartyExchangebackupsolutionswillbundletransactionsfromtheloginto15 minuterecoverypoints.Ofcourse,losing15minutesofemailtrafficisstillaprettybig disaster. Thedownsidetoallthisissimplymanagementoverheadandstoragespace.Exchange backupscanoccupyalotofdiskandtapespace,andkeepingtrackofallthefilescanbe complex.Infact,mostthirdpartyExchangebackupsolutionsareprimarilysolutionsfor managingbackupfilesafterall,theactualbackupfunctionalitycomesfromExchanges APIs.
69
Don Jones
LetsrevisitourBackup2.0missionstatementfromChapter1: Backupsshouldpreventusfromlosinganydataorlosinganywork,andensurethat wealwayshaveaccesstoourdatawithaslittledowntimeaspossible. ThisisatrickystatementtoevaluatewhenitcomestoExchange.Certainly,withCCRor LCR,wecanachievebackupsthatofferverylittledowntime;inthecaseofCCR,downtime mightamounttoafewseconds.Wewouldcertainlyloseverylittledata,althoughsomedata lossispossiblebecausebothCCRandLCRutilizeasynchronousreplication,meaningits possibleforafewminutesworthoftransactionstooccuronthesource,yetnotreplicateto themirrorwhenafailureoccurs. ButCCRandLCRhavetwodistinctproblems:First,theyonlymaintainacurrent,working copyofthedatabase;theydontprovidealongtermarchiveandtheydontallowyouto restoretoaparticularpointintime.Ifyouaccidentallydeleteamailbox,thatdeletionis replicated;afterthedeletedmailboxretentionperiodelapses,themailboxisgoneforever nomatterhowmanyCCRorLCRreplicasyouhave.Second,bothCCRandLCRare expensiveintermsofstorageresourcesforboth,andintermsofadditionalhardwareand softwarelicensesforCCR.NeitherLCRnorCCRaredesignedtoprovidesinglemessage recoverybeyondthedeleteditemretentionperiodinExchange. Note Itsbeensuggestedtomebysomeadministratorsthattheycouldsimplyset thedeleteditemretentionperiodveryhighashighas5yearsinonecaseI saw.Idontrecommendit;theExchangedatabasewillgethuge,anditsimply isntintendedtobeapermanentarchive.Performancewillsuffer,anditll becomeharderandhardertogetrealbackupsasthedatabasebloats. Traditionalthirdpartybackupsolutions,whichrelyeitherontheExchangetransactionlog ortheVolumeShadowServiceAPI,haveallthefailingsofanyBackup1.0solutionwhichI explainedatlengthinthepreviouschapter.Backupfilemanagement,backupwindows, timeconsumingrestores,andotherdisadvantageshavebeenfrustratingExchange administratorsformorethanadecade. Letmeclueyouintoalittlesecret:ThereasontraditionalExchangebackupscanbe frustratingorincompleteeveninthecaseofCCRandLCRisthattheyrelyoncatching transactionsastheyoccurorreadinginformationfromExchangesownAPIs.Inother words,everythingdependsonhowExchangeworks.Foranuptodatebackup,likewhat CCRandLCRoffer,youhavetoreplicatetransactions.Forafullpointintimebackup,you havetotalktoExchangeanddealwiththedatathewayExchangegivesittoyou.Whenit comestoExchangeBackups1.0,thatstheultimateproblem.Thesolution?Stopdealing withExchangeforyourbackups.
70
Don Jones
Inthepreviouschapter,Ipositedanewtypeofbackupsolutionthatfocusedondiskblocks. Figure4.6illustratesmyproposal:ForgetabouttalkingtoAPIsandjustgrabthe informationasitiswrittentodisk.
Figure4.6:AproposalforBackup2.0. Thinkaboutit:SoftwaredevelopersliketheoneswhowroteExchangeServerknow thatservermemoryisntreliable.Alossofpower,asoftwarecrash,whatever,andmemory islost.Disk,however,ismuchmorereliableandispersistent.Exchangestransactionlogis designedtohelpprovideacoverforunreliablememory:Intheeventofamemoryfailure, thetransactionlogallowsworktobereplayed.Forthatreason,thelogitselfsitson persistentdiskstorage. ThatmeansaBackup2.0solutioncansimplygrabblocksofdiskspaceastheyarechanged ondisk.Thatwillgrabanychangestothetransactionlogaswellaschangestothemain Exchangedatabasefiles(thereareafewfilesforeverymailstore).Byimmediately shippingcopiesofthosediskblockstoaseparateserver,youhaveacontinuousbackupthat doesntrelyontalkingtoExchangeAPIs,replicatingtransactions,oranyothercomplexity.
71
Don Jones
Now,thatsallwellandgoodforsimplefilesondisk:Ifyouwanttorestoresomething,just trackdownthediskblocksthatcomprisethefileandputthoseblocksbackontheserver. Poof,filerestored.ForExchangedisasterrecovery,asIllexplaininamoment,itsafast waytogetanentireserverbackonline.However,howdoesithelpwiththemorecommon restorescenarioslikesinglemailboxrecoveryorsinglemessagerecovery?
ThisiswhereweputBackup2.0tothetest:Doesitmeetthemissionstatementwhenit comestoExchange? Backupsshouldpreventusfromlosinganydataorlosinganywork,andensurethat wealwayshaveaccesstoourdatawithaslittledowntimeaspossible. Ithinkitdoes.Wevegotanearlyrealtimebackupofeverythingthatgetswrittento Exchangesdisksincludingdatabasesandtransactionlogs.Thatmeanswedontloseany data,andrestoringdatadoesntneedtoincludetakingExchangeoffline(unlesswere talkingaboutacompletedisasterrecoveryscenariowhichIllcovernext). Withtherighttoolsandinterfaces,Backup2.0enablessingleitemrecoverysomething Illoutlineabitlaterinthischapter.Thatsreallythekeyrestorescenario,althoughyou mightalso,fromtimetotime,needtorecoveranentireExchangedatabaseandmountit elsewherefortestingpurposes. Forexample,IhaveoneclientwhoroutinelyrestoresExchangedatabasestoa disconnectedExchangeServer,wheretheyperformvulnerabilityscanningandantispam testing.Backup2.0excelsatthis,asitcanquicklybringbackanindividualfileevenone aslargeasanExchangedatabaseveryquickly.ButthatskindofaBackup1.0mentality: WithBackup2.0,youcouldmakethattestingoperationevenfasterbysimplyexportingthe entireserversdiskimageintoabootablevirtualmachine.Thatvirtualmachinecanbe easilysegregatedfortestingsothatitwouldntinterferewiththeproductionnetwork(that kindoftestingiswherevirtualizationsawitsfirstwidespreaduses,infact). Note Backup2.0isallbasedondiskblocksraw,diskleveldata.Wherethatdata sitsdoesntmatter,meaningBackup2.0isalsoagreatwaytodophysicalto virtual,physicaltophysical,virtualtovirtual,andvirtualtophysicalmoves andmigrations.IveusedBackup2.0stylesolutionsinmanycasestomove physicalExchangeServercomputersintovirtualmachinesaspartofalarger enterprisevirtualizationproject.
72
Don Jones
DisasterrecoveryiswhatBackup2.0isallabout,andExchangeServerisnoexception. Withadiskblockbasedbackupimage,youcanquicklyrestoreyourentireExchange Servertonotjustthemostrecentbackupbutalsotoanygivenpointintime.Youcaneven restoreyourExchangeServertoavirtualmachine,whichisgreatforhugedisaster recoveryscenarioswhereyoumightbehostingthosevirtualmachinesatarecoveryfacility oreveninsomeonlinehostingprovider. SohowdoesBackup2.0complementorinterferewithCCR,Exchangesbuiltinrecovery solution?KeepinmindthatCCRrequiresapassive,standbyserver.Thatmeansthe expenseofadditionalWindowsandExchangelicenses,andpossiblytheexpenseof dedicatedhardware,allasastandbytoafailure.Thatsanexpensesomeorganizationsare happytobear,butitsnotforeveryone.Insomecases,yourCCRpassivenodemightbea lesscapablemachine(thatmightbethecaseifitwasrunninginavirtualmachine,for example)designedtogetyouthroughatoughtimewithlessthannormalperformance.In thoseinstances,Backup2.0canhelpbygettingExchangeupandrunningmorequicklyon youroriginal,fullpoweredhardware.FororganizationsthatcantaffordCCR,Backup2.0 provideswhatisperhapsthenextbestthing:AfastwaytoquicklybringExchangeback onlineinabaremetalrecoverysituation. Backup2.0complementsCCRinthatBackup2.0providestheabilitytorollbacktoa previouspointintime;CCRdoesnot.CCRsgoal,remember,istocreateanexactreplicaof yourExchangedatabaseswithaslittlelatencyaspossible.Thatmeansifyoudosomething wrong,thatsomethingisgoingtoreplicateviaCCRveryquickly,meaningyourbackupis alsomessedup.CCRcantundeleteamailboxoramessage,anditcanthelprecoverfrom accidentalormaliciousactions.Backup2.0,however,candosoanditcanprotectyour passiveCCRnodesatthesametimeitprotectsyouractiveExchangeServercomputers. WhatIlikebestaboutBackup2.0isthatthebackupscanberestoredalmostanywhere. LoseanExchangeServercomputeranddonthaveasparehandy?Noproblem:Restoretoa virtualmachine(Itellclientstoalwayshaveatleastonevirtualizationhosthangingaround thathassomesparecapacityforjusttheseemergencies).NoreinstallingWindows, reinstallingExchange,reconfiguringExchange,andwaitingontapebackupstounspool yourbackedupdatabasesjustdumptheentireserverdiskimageintoavirtualmachine. Itsafastprocessandtheresultisacompletelyconfiguredcomputerthatisthecomputer youlost.Clientsjustreconnectandstartworking.
EasierManagement
MyBackup2.0ideadoesneedtohaveafewmoreExchangespecificcapabilities.For example,Exchangeisdesignedsothattransactionsremaininitsloguntilabackupofthe databaseismade;thatway,youreassuredofthelogservingasabackupfortransactions untiltherelateddataissafelyontape.InaBackup2.0world,traditionalbackupsdont occursothebackupagentrunningontheExchangeServercomputerneedstobesmart enoughtotruncatetheExchangetransactionlogaftertransmittingtherelateddiskblocks offtothebackupserver.
73
Don Jones
Fromthere,youreleftwithoutmuchtomanage.Nologbackups,nofullbackups,no differentialbackupsjusttheabilitytorestore,fromanimage,anybitofExchangeyou needto,atanytime.Asyoullseeinthenextfewsections,though,Backup2.0canenable someprettyimpressivenewmanagementscenarios. WhatAboutPerformance? DoesntallthisBackup2.0magicplacesomeseriousburdenonyour ExchangeServercomputers?Inmyexperience,no.Themajorityofthe overheadkicksinwhenyoustartdeduplicating,compressing,indexing,and savingdatathatisusuallyoffloadedtoacentralizedbackupserver.Allthe ExchangeServerbasedagenthastodoistransmitdiskblockdeltasoverthe networkyoumightseelowsingledigitincreasesinthingslikeprocessor utilization,butthatsit.Backup1.0solutionstendtohitExchangeServer harderbecausetheyrenotgrabbingdatainsmallchunksallday;theyre tryingtocramalltheirbackupactivityintoasmalleveningwindow.
ExchangeSpecificConcerns
SohowwillBackup2.0workwithExchange?Ivealreadypointedouthowspecialized Exchangeisinthewayitworks;willBackup2.0beabletoworkwithitandstillprovidea betterbackupsolutionthanBackup1.0does?MuchofExchangesfunctionalityand architecturearespecificallydesignedtoaccommodateandworkwithinaBackup1.0 worldwillturningthatworldintoBackup2.0breakeverything?
CCR
ABackup2.0solutionneedstobeCCRaware.Afterall,CCRisstillavaluablehigh availabilitytactic,givingyounearinstantaneousfailoverintheeventofacompleteserver failure.CCRevensupportsgeographicallydispersedclustering.Soabackupsolutionneeds tounderstandCCRandworktotruncatetheactiveCCRnodeslogbasedonthepassive CCRnodesreplicationoftransactions.
IndividualItemRecovery
RecoveringmailboxesorindividualmessagestoaPSTfilemaybeusefulbutyou shouldntbestuckwiththatasyouronlyoption.Honestly,givingauseraPSTfileand tellingthemtodraganddropmessagesinOutlookisinsanelyprimitive.ABackup2.0 solutionshouldeliminatethatoverheadandletyourestoredirectlytoaliveExchange Servercomputer. Further,aBackup2.0solutionshouldbeabletorecoverindividualmessagesfromthesame backupthatwouldbeusedfordisasterrecovery.Inotherwords,youshouldnthaveto extractindividualmessagesoutofExchangeyoushouldbeabletorecoverfromthe backupimageoftheactualExchangedatabase.How?
74
Don Jones
Practicallyspeaking,itwouldprobablybeatwostepprocess.Assumeyouhaveanimage levelbackupofanExchangeServercomputer.Thatmeansyouvegoteveryblockofdata fromdisk,soyoucanreconstructtheentireserver.Withtherighttoolset,youwouldbe abletomountthatbackupasafilesystemineffectbrowsingthebackedupfilesystem fromaspecificpointintimeasifitwereanetworkdrive.Butyourejustlookingatthe backedupfilesExchangeisntrunninganditsdatabasefilesarentopenedandbeing used;yourelookingatastatic,pointintimecopyofthosefiles.Fromthere,therighttool wouldletyoumountandbrowsetheExchangedatabasegivingyouaccesstoindividual mailboxes,messages,andotherdata.YouwouldntneedtodotheusualExchange escapadesofrestoringthedatabasefilefrombackup,mountingthedatabasetoaRecovery StorageGroup,andrunningExMergeorotherutilitiesagainstthemounteddatabase. Instead,youdjustdiveintothedatabaseusingautilitythatunderstandsthedatabase structure,andgetwhatyouneed.ItmightlooksomethinglikeFigure4.7.
Figure4.7:Restoringmessagesfromamountedbackupimage.
DeDuplication
Exchangessingleinstancestorageallowseachmessagetobestoredonlyonceinthe messagestoreitsaformofdatadeduplication.Butitdoesnthelpwhenamessageis senttouserswhosemailboxesareondifferentservers,orevenuserswhosemailboxesare indifferentstoresonthesameserver.Inthosecases,themessagewillbeduplicatedat leastonceperstore. ButBackup2.0candoabetterjobofdeduplicatingdataatleastdatafromasingle serverbecauseitsexaminingdiskblocks.Thesamemessagestoredondisklooksthe same,eventhoughthatmessagemighthavetobeduplicatedacrossmultiplemail databases.SothemessagewillbeduplicatedinExchange,butonceitsbackedupbythe Backup2.0solution,thatsolutioncandetecttheduplicatediskblocksandstoreonlya singleinstancemeaningthebackupcanbesmallerthantheoriginaldatabase.Addin compression,somethingevenBackup1.0solutionsoffer,andthebackupcanbemany timessmallerthantheoriginaldata.
75
Don Jones
DataCorruption
Becausedataisbeingbackeduponablockbyblockbasis,itseasierforaBackup2.0 solutiontodetectandcorrecterrors.Asingleblockofdiskdatamightbeassmallas4 kilobytesdetectingatransmissionerrorinthatsmallapieceofdataiseasy.
SearchandeDiscovery
SearchandeDiscoveryarerapidlybecomingkeycomponentsformanyorganizations.The USFederalCourtSystem,forexample,hasimposedstrictrulesthatrequireprettyrapid responsestoeDiscoveryrequestsduringcourtproceedings;failuretomeetthese requirementscanleadtofinesandevensummaryjudgments.KnowingthattheExchange databasedoesntprovidesolidsearchcapabilitiesnatively,manycompaniesrelyon dedicatedmessagearchivalandretrievaltoolsanadditionalexpenseandyetanother massofstorageresourcestomanage.ABackup2.0solution,however,canprovidesolid searchandeDiscoverycapabilitiesbuiltrightin. Considertheabilitytomountapointintimebackupimageasabrowseablefilesystem,as Ivedescribedearlier.AlsoconsidertheabilitytobrowseanofflineExchangedatabase fromthatfilesystem.GiventhosecapabilitieswhichaBackup2.0solutionmightwell offerasameansofperformingsinglemessagerecoveryyoucouldeasilyimplementa powerfulmessagesearchfunctionthatmakesmessagesearchingandeDiscoverypossible. Figure4.8showswhatitmightlooklike.
Figure4.8:Searchingformessagesinabackupimage.
76
Don Jones
Importantly,thistoolwouldneedtobeabletoattachtomultipleExchangedatabasesto search;youcannevertellaheadoftimewhichdatabasewillcontainthemessagesyoure after,andyoudontwanttohavetosearcheachoneindividually. InaneDiscoveryscenario,youlltypicallywanttorestoremessagesnottoaliveExchange ServercomputerbutrathertoaPSTwhichwilloftenbedeliveredtolegalcounsel.Figure 4.9showshowaBackup2.0toolsetmightimplementthatexport.
Figure4.9:ExportingsearchresultstoaPSTfile. ItsimportantthatyounotoverbuildyourexpectationsforaBackup2.0solution,though. MessagesearchandrecoveryisonlyoneaspectofeDiscovery;manyorganizationsthat routinelydealwithlegalsummonsprefertotagandcategorizemessagesastheyresent; thismakesiteasiertolocatemessagesondemand(andhelpscategorizemessagesfor securityandmonitoringpurposes,too),butobviouslygoeswellbeyondwhatyoudexpect fromabackupsolution.SotherewillstillbeamarketforspecificeDiscoverysolutions, especiallyforverylargecompanieswhoroutinelyhavetoperformeDiscoverytasks.
ComingUpNext
IfyouthoughtExchangeServerhadsomespecializeneeds,waituntilIgetintoSQLServer inthenextchapter.Microsoftsrelationaldatabasemanagementsystemisoneofthefew Microsoftproductsthathashadawellunderstoodbackupandrestoresystemformany yearsbutonceagain,IlltryandturnBackup1.0onitsheadandshowyouwhereour longusedroutinesandtechniquesjustdontmeetmodernbusinessneeds.
77
Don Jones
Chapter5:SQLServerBackups
MoreandmorecompaniesareusingMicrosoftSQLServerthesedaysandinmanycases, theydontevenrealizeit.WhileplentyoforganizationsdeliberatelyinstallSQLServer, manybusinessesfindthemselvesusingSQLServerasasideeffect,becauseSQLServeris thedatastoreforsomelineofbusinessapplication,technologysolution,andsoon.Infact, SQLsprawlmakesSQLServeroneofthemostchallengingserverproductsfromabackup perspective:NotonlyisSQLServerchallenginginandofitself,butyouwindupwithtons ofinstances! HereswhatIseehappeninginmanyorganizations:Thecompanyhasoneormoreofficial SQLServerinstallations,andtheITteamisawareoftheneedtobackuptheseinstanceson aregularbasis.ButtherearealsonumerousstealthinstallationsofSQLServer,often runningontheExpresseditionofSQLServer,thattheITteamisunawareof.Thedata storedinthesestealthinstallationsisnolessmissioncriticalthanthedataintheofficial installations,butinmanycases,thatdataisntbeingprotectedproperly.Dealingwiththis sprawlisjustoneoftheuniquechallengesthatBackup2.0facesinSQLServer.
NativeSolutions
SQLServerhasalwaysofferedanativeapplicationprogramminginterface(API)for backingupdatabases.Infact,SQLServerhaslongbeenoneofthefewMicrosoftserver applicationsthatnativelysupportstapebackup,withoutusingWindowsownbackup utility.Thenativebackuptoolsetisactuallyquiterobust,supportingfeatureslike compression(highlightedinFigure5.1),encryption,andsoforth.
78
Don Jones
Figure5.1:SQLServersnativebackupinterface. TounderstandSQLServersnativebackuptechnology,youneedtofirstknowabitabout howSQLServerworksunderthehood.
HowSQLServerWorks
SQLServerstoresthingsondiskin8KBchunkscalledpages.Italsomanipulatesthose same8KBchunksinmemory,meaningthesmallestunitofdataSQLServerworkswithis 8KB. Whendataiswrittentodisk,anentirerowofdatamustfitwithinthat8KBpage.Its possibleformultiplerowstoshareapage,butarowcannotspanmultiplepages.So,ifa CustomerstablehascolumnsforName,Address,City,State,andPhone,thenallthatdata combinedmustbelessthan8KB.Anexceptionismadeforcertaindatatypessuchas binarydatalikephotos,orlargegobsoftextwheretheactualpageonlycontainsa pointertotherealdata.Therealdatacanthenbespreadacrossmultiplepages,oreven storedinafile.SQLServergathersallthese8KBpagesintoasimplefileondisk,which usuallyhaseitheran.MDForan.NDFfilenameextension.
79
Don Jones
WhenSQLServeristoldtodosomething,itsbymeansofaquery,writtenintheStructured QueryLanguage(SQL)syntax.Inthecaseofamodificationquery,SQLServermodifies thepagesofdatainmemory.Butitdoesntwritethosemodificationsbackouttodiskyet, astheremightbeadditionalchangescomingalongforthosepagesandthesystemload mightnotofferagooddiskwritingopportunityrightthen.WhatSQLServerdoesdo, however,ismakeacopyofthemodificationqueryinaspeciallogfilecalledthetransaction log.Thisfile,whichhasan.LDFfilenameextension,keepsarecordofeverytransactionSQL Serverhasexecuted. EventuallymaybeafewsecondslaterSQLServerwilldecidetowritethemodified pagesouttodisk.Whenitdoesso,itgoesbacktothetransactionlogandchecksoffthe transactionthatmadethemodificationsessentiallysaying,Okay,Imadethatchangeand itsbeenwrittentodisk.ThatwaySQLServerknowsthatthechangeissafeondisk. IntheeventthatSQLServercrashes,ithasanautomatedrecoverymodethatkicksinwhen itstartsbackup.Itgoesstraighttothetransactionlogandlooksforuncommitted transactionsthosethathavenotyetbeencheckedoff.Itknowsthatthecheckedoff transactionsaresafeondisk;anythingelsehadnotbeenwrittentodiskandwasstill floatingaroundinmemorywhentheservercrashed.SoSQLServerreadsthose transactionsoutofthelog,reexecutesthem,andimmediatelywritestheaffectedpagesto disk.ThisprocessallowsSQLServertocatchupwithanyinprogresswork,andensures thatyouneverloseanydataprovidedyourdiskfilesareokay,ofcourse. Thinkaboutthisimportantfact:EVERYTHINGthathappensinSQLServerhappensonly throughthetransactionlog,andSQLServercanrereadthelogtorepeatwhateverhas happened.ThisprocessmakesnearlyeverythingthatSQLServerdoespossible.
HowSQLServerNativeBackupWorks
SQLServersnativebackupsystemworksinconjunctionwiththetransactionlog. Essentially,therearetwotypesofbackupSQLServercanmake:databackupsandlog backups.Databackupsare,asyoumightsuspect,ofthedatabaseitself.Thesearedoneina Backup1.0stylemanner,grabbingasnapshotofthedataasitsitsduringthebackup.Log backupsgrabthecontentsofthetransactionlog. SQLServersnativebackupcapabilitiesincludetheabilitytobackupadatabasewhileits inuse,althoughdatabaseperformancecanslowslightlywhileabackupoperationis underway.TheabilitytobackupaninusedatabasemeansthatSQLServerislessimpacted bybackupwindowsthanmanyotherserverproducts,anditmeansthatyoureabitless tiedtotheBackup1.0modelofonlygrabbingbackupswhilethedataisntbeingused. ButthatdoesntmeanSQLServerisentirelyfreeofbackupproblemsandchallenges.
80
Don Jones
ThereareafewdistinctchallengespresentedbytraditionalSQLServerbackuptechniques: Sprawl.AsImentionedearlier,mostorganizationshavealotmoreSQLServer installationsthantheyoftenrealize,andbackingupthemallcanbepainful.Insome cases,particularlywiththeExpresseditionsoftenembeddedintolineofbusiness applicationsandITtools,SQLServerisrunningonaclientcomputerthatisntbeing treatedlikeaserverintermsofbackupandrecovery. Snapshots.JustlikeanyBackup1.0scenario,SQLServerbackupsarebuiltaround theideaofpointintimesnapshots.AsIlldescribeinabit,SQLServerdoesoffer someuniqueabilitiesthatletyoutakemoresnapshotsmorefrequently,butyoull alwayshaveacertainamountofdataatrisk. Recoverytimes.AlthoughSQLServercanbeprettyflexibleinhowitmakesyoudo backups,restoringisstillatimeconsumingoperation.Sotimeconsuming,infact, thatsomecompanieshavecreatedtoolsthatcanattachadatabasebackuptoSQL Server,allowingthebackupdatatobequeriedwithoutactuallyhavingtorestore thedatabase.Thistrickisusefulforthingslikechangecontrol,butitdoesnthelp fromabackupandrecoveryperspectivesimplybecausetheattachedbackupis readonly. Transactionlogs.InSQLServer,backupsareintimatelytiedtothetransactionlog, andbackupsarerequiredinordertokeepthetransactionlogfromgrowinglarger andlarger.AnybackupplanthatdoesntusethenativeAPIsneedstodealwiththis fact.
AnyproposedbackupsolutionthatdoesnotuseSQLServersnativeAPIswillbe challenging.Infact,mostthirdpartybackupsolutionsaresimplyagentsthatsitontopof SQLServersnativeAPIs!ThissetupensuresthatSQLServersinternalneedslikethe transactionlogaretakencareof,butitalsohashistoricallylimitedthirdpartysolutions tothesamebasicfeaturesetasSQLServersnativecapabilities.MostthirdpartySQL ServerbackupsolutionsarereallylittlemorethananagentthattakesdatafromSQL ServersnativeAPIs,andtransmitsthatdataacrossthenetwork.
IntheOldDays
SohowhasSQLServertraditionallybeenincludedinabackupandrecoveryplan?Lets considersomeofthetechniques,scenarios,andtoolsthatarecommonintheBackup1.0 world.
81
Don Jones
BackupTechniques
SQLServernativelyoffersthreetypesofbackup.IknowIsaidtwoearlier,buthearmeout: FullBackup.Thisisacompletebackupoftheentiredatabase.Oncemade, committedtransactionsinthetransactionlogarecleared,aprocesscalledlog truncation;thisiswhatkeepstransactionlogsfromgrowingforever. DifferentialBackup.Thisisalsoabackupofthedatabase,butonlythedatathat changessincethelastfullbackupisincluded.Again,thetransactionlogistruncated. TransactionLogBackup.Thisdoesntgrabanyoftheactualdata;itsimplygrabs thecurrentstateofthetransactionlogandthentruncatesthatlog.
Sotwokindsofdatabackupandalogbackup.AlthoughSQLServercanbackupanactive database,itsnotsomethingyouddoduringpeakdatabaseusageduetoperformance concerns,sofullandevendifferentialbackupsarestillusuallydoneduringoffpeak periodsorduringaneveningorweekendmaintenancewindow.Becauseitcanbedifficult togetanightlyfullbackupoflargedatabasesinthatwindow,administratorstypically resorttoatieredbackupplangrabbingfullbackupsontheweekends,forexample,and differentialseachevening.Tohelpreducetheamountofatriskdata,transactionlog backupscanbemadeperiodicallythroughouttheday.Thesebackupsareveryfastand offerlittleperformanceimpact,soapracticalbackupplanmightlooksomethinglikethe oneinFigure5.2.
Figure5.2:TypicalSQLServerbackupplan.
82
Don Jones
Withthisplan,themaximumamountofatriskdataisaboutanhour,asthatstheinterval betweentransactionlogbackups.Ofcourse,inabusydatabase,anhourcanbealotofdata! ReviewingourmanifestoforBackup2.0: Backupsshouldpreventusfromlosinganydataorlosinganywork,andensurethat wealwayshaveaccesstoourdatawithaslittledowntimeaspossible. Anhourofatriskdatacertainlydoesntpreventusfromlosinganydataorlosingany work.Inaddition,therestorescenarioassociatedwiththiskindofbackupplanis,asyou shallsee,hardlyconducivetoaslittledowntimeaspossible.
RestoreScenarios
SQLServerrecoverycanbeatimeconsumingthing.Essentially,youhavetostartwith yourmostrecentfulldatabasebackup,thenaddonthemostrecentdifferentialandevery transactionlogbackupmadesincethen. Infact,youhavetobeveryspecificaboutwhatyouredoingwhenyouconductarestore anaspectofSQLServerthatIvefranklyseenalotofadministratorsmessupprettybadly. Ifyouconductanormal,fulldatabaserestore,SQLServerwillbydefaultputtherecovered databaseonlineassoonasitsdonewiththerestoreoperation.Ifyoustillhavea differentialorsomelogbackupstoapply,youreoutofluck;youhavetostarttherestore over.ThetrickistotellSQLServer,asyourerestoringthefullbackup,thatyouhavemore filestorestore.Youcontinuetellingitthatuntilyourestorethelasttransactionlogbackup, atwhichtimeyoutellSQLServerthatitssafetostartrecovery.ThenSQLServerwillstart applyingthedifferential,thenthetransactionlogbackups,andthenyourdatabasewillbe readytouse.Aslittledowntimeaspossibleisntverylittle,inmostcases,andyoullstill bemissinganychangesthatoccurredafterthemostrecenttransactionlogbackup. SQLServerRecovery Foralargedatabase,SQLServersrecoverytimecanbequitelengthy.Lets sayyouusethebackupplanshowninFigure5.2,andsomethinggoeswrong at4pmonFridayafternoon.Youllhaveafullbackupfromtheprior weekend,Thursdaynightsdifferentialwhichmaybequitelarge,sinceit containsallthechangesfromthefullbackupuptoThursdaynightand hourlytransactionlogbackups. Notonlydoyouhavetowaitforallthosefilestostreamofftapeorwherever youstorethem,youhavetowaitforSQLServertoworkthroughthem.Ithas toapplythedifferentialbackuptothefullbackup,thenithastoreplayeach individualtransactionfromeverysingletransactionloginessence,ithasto reperformalltheworkthatwasdonealldayFriday.Foralarge,busy database,itmaybealongtimebeforethedatabaseisreadytouse. SQLServerdoesntnativelysupportsingleobjectrestores.Whatyoucandoisrestorea backuptoadifferentdatabase,thenmanuallycopyanyobjectsyouwantrestoredfromthat backup.Thisletsyourecoversinglestoredprocedures,tables,orevenrowsofdata providedyouknowhowtodosomanually.
83
Don Jones
SQLServerdoessupportpointintimerecovery,withtheobviouscaveatthatitcant restoretoapointintimelaterthanyourmostrecentbackup.Pointintimerecoveryonly workswithtransactionlogbackupsbecausetransactionsinthelogaretimestamped.If youdiscardThursdaystransactionlogbackupsaftermakingadifferentialbackupon Thursdaynight,thenthefirstpointintimeyoucanrecovertoisthetimeofthatThursday nightdifferential.Thisactuallymakesbackupmanagementtrickybecausetoenable maximumpointintimerecovery,youhavetokeepalotoffileshangingaround:full backups,everynightsdifferential,everyhourstransactionlog,andsoforth. Considerthisscenario:YoureusingtheexamplebackupplanfromFigure5.2,which entailsaweeklyfull,nightlydifferential,andhourlytransactionlogbackups.Letssayyou keep3weeksworthofbackups,andWeek3isthemostrecentset.ItsFridayafternoon, andyourealizesomeonedeletedacriticalstoredprocedure.Youneedtorecoverthe databasetothepreviousWednesday(Week2)afternoon;Figure5.3illustratesthefilesthat youhaveonhandandwhichonesyoullhavetorestore.
Figure5.3:Samplerecoveryplan.
84
Don Jones
So,thats: ThebeginningofWeek2fullbackup TheTuesdaynightdifferentialfromWeek2 TheWednesdaytransactionlogbackupsuptothepointWednesdayafternoonyou wanttorecoverto
Thatssixorsofilestorecover,andthenyouwaitforSQLServertosortitallout.Intotal, youllbekeepingsomethinglike140fileslyingaround,assumingyoutakeatransactionlog backupeighttimesaday(onceeveryhourduringthenormalworkingday).
DisasterRecovery
SQLServerdoesntofferanykindofnativedisasterrecoverycapabilities.Essentially,ifyou loseanentireserver,youllhavetorecoverWindows,installSQLServer,andthenstart restoringSQLServerbackupstobringyourdatabasesasuptodateaspossible.Traditional thirdpartyimagingsoftwareisnteffectivebecauseitsdifficulttoimageanactiveSQL Serverinstallation,andbecauseimagingdoesntalwaysworkwellwithSQLServersnative backupcapabilitiesmeaningitcanbetrickytorestoreanimageandthenalsorestore normalSQLServerbackupstobringyourdatabasesmoreuptodate. Inshort,letshopeyoudontloseanentireSQLServer. Infact,wholeserverdisasterrecoveryforSQLServerissounsatisfyingthatMicrosofthas madeaconsiderableinvestmentinSQLServerhighavailabilityfeaturesthattrytoreduce theneedtoeverdoawholeserverrecovery.Someoptionsinclude: TransactionLogShipping.Theideahereistostartoffwithtwoserversthathave anidenticalcopyofadatabase,thenshipthetransactionlogsfromtheactive servertothehotspareserver.Thesparereplaysthetransactionstobringitscopy ofthedatabaseuptodate;thetheoryisthatifthemainserverdies,thehotspare canbebroughtittoreplaceit. DatabaseMirroring.Essentiallythesameideaastransactionlogshipping,onlythe hotspareiskeptmoreuptodateandcantakeoverautomaticallyifthemain serverdies. Clustering.UtilizingWindowsnativeclusteringcapabilities,thisprovidesa completelyredundantserverwithdirectaccesstothesamedatabasefilesasthe mainserver.
AlloftheseoptionsrequireadditionalSQLServerinstallationsandhardware(orvirtual servers),andtheyrealldesignedtohandleacompletefailurescenario;noneofthese actuallyprovidesforpointintimerecoverycapabilities,sotheyretobeusedinadditionto normalbackuptechniques.Itcangetprettyexpensive,especiallyforsmallerandmidsize companieswhomaynotbeabletoaffordthislevelofrecoverabilityatleastinaBackup 1.0world.
85
Don Jones
BackupManagement
Itouchedonthisearlier,buttheshortmessageisthatSQLServerbackupmanagementcan beprettypainful,unlessyoureonlyworriedaboutrestoringthedatabasetoitsmost recentstate.Inthatcase,youkeepthemostrecentfullbackup,mostrecentdifferential,and alltransactionlogssincethedifferential;thatsstillalotoffilestomaintainbutitsalot lessthantryingtokeepafewweeksworthoffiles. Ioncehadajobwhereweneededtobeabletorestorethedatabasetoanypointintimefor 3months.Youcanimaginethenumberoffileswehadtomaintain;Ithinkitwascloseto 600backupfiles,allfloatingaroundondifferenttapes,someofwhichhadtoberotatedoff siteitwasanightmareandjustdescribingitisgivingmeunpleasantflashbacks.Infact,it wasatthatexactpointintimethatIstartedtorealizethattheBackup1.0wayofdoing thingswasnotveryefficientespeciallybecausemanagingthatmanyfilesstillleftusat riskforanhourormoreofdataandwork.
SohowcanBackup1.0beimprovedfromaSQLServerperspective?Therescertainly plentyofroomforimprovementbasedonthetraditionaltechniquesandapproachesIjust discussed.
Thewholeideaofbeingabletomaketransactionlogbackupstohavelessdataatriskis wonderful,butitisultimatelyakludge.Itsaworkaroundtothesnapshotoriented approachofBackup1.0;IvesaiditbeforeandIllsayitagainhere:Backupsshouldbe continuous.Itsnotpracticaltocontinuallymaketransactionlogbackups,andthatsthe bestSQLServercanoffer;thatmeanswehavetomoveoutsideofthenativeAPIs.Thats scary,Iknow,becausesomuchofSQLServerdependsonfolksusingthosenativeAPIs.But stickwithme. IfweacknowledgethatSQLServersnativeAPIsarentgoingtogiveusfrequentenough backups,weneedtolookatotherwaysofgettingtothedata.GoingthroughSQLServeris nottheanswerbecauseSQLServerdoesnthavethebandwidthtofeedusanykindof continuousdatastream.Instead,weneedtograbthatdatadirectlyfromtheoperating system(OS),asthedatahitsthedisk.Keepinmind:AscomplicatedasSQLServeris, ultimatelyitsalljustbitsondisk.TheresnoreasonwecouldnthaveaBackup2.0style agentsittingontheSQLServercomputer,grabbingdiskblocksasSQLServerwrites changestothedisk.
86
Don Jones
Cleverreaderswillhavespottedaproblemwiththistheory,frommyexplanationonhow SQLServerworks: WhenSQLServeristoldtodosomething,itsbymeansofaquery,writteninthe StructuredQueryLanguage(SQL)syntax.Inthecaseofamodificationquery,SQL Servermodifiesthepagesofdatainmemory.Butitdoesntwritethose modificationsbackouttodiskyet,astheremightbeadditionalchangescoming alongforthosepagesandthesystemloadmightnotofferagooddiskwriting opportunityrightthen. Oops.IfSQLServerdoesntwritethedatatodiskquickly,thenthatdataisatriskbecause allweregrabbingarethechangesthatactuallymakeitontothedisk.Buttheanswerto thispotentialproblemalsoliesintheverywaythatSQLServerworks: WhatSQLServerdoesdo,however,ismakeacopyofthemodificationqueryina speciallogfilecalledthetransactionlog.Thisfile,whichhasan.LDFfilename extension,keepsarecordofeverytransactionSQLServerhasexecuted. Thetransactionlogitselfisjustafileondisk;Microsoftknowsperfectlywellthatanydata livingentirelyinmemoryisalwaysatrisk,andsothetransactionlogsentriesarewritten todiskimmediately.Allouragentwouldneedtodoisalsograbthechangestothe transactionlog.Then,inafailure,wedsimplyrestorethedatabase,restorethetransaction log,andletSQLServersnaturetakeitscourse.
TherestorescenariosinaBackup2.0worldwouldbevastlyimproved.Forone,the conceptofBackup2.0involvescontinuouslystreamingchangeddiskblockstosomecentral repository;thatbeingthecase,youdsimplyselecttheexactpointintimeyouwantedto restoreto,thenstreamthosediskblocksrightbacktowheretheycamefrom.Youmight havetoshutdownSQLServerwhileyoudidthat,butyoumightnot;programmerscanget prettycleveratmanipulatingSQLServer,andSQLServeritselfisprettyopento manipulationinthisregard. Suddenly,nomoreworryingaboutfullbackups,differentials,andtransactionlogs.You dontcareaboutthefilesperse;youonlycareaboutthediskblocksfromtheactive databaseandlogfiles.YourenotmakingbackupsintheSQLServersenseoftheterm; youreactuallyjustputtingthecomputersdiskbacktotheconditionitwasatacertain pointintime.SQLServerneveractuallyentersitsrecoverymode,becauseyouvenot restoredanyfilesintheSQLServerfashion.SQLServersimplyresumesworkingwiththe databaseandlogfilesjustasitnormallyworkswiththem. Thisentireidea,whichisattheheartofBackup2.0,tookmeawhiletoreallysortoutinmy mind.Intheend,everythingweknowaboutbackupsiswrong,whichiswhyIchosetouse thetermBackup2.0.Thisisanentirelydifferentwayoflookingatthings.
87
Don Jones
Whataboutsingleobjectrecovery?Thatwouldstillbetricky.Backup2.0willcertainlylet usrestoreasingledatabase,ratherthananentireserver,ifdesired.Butkeepingtrackof whichdiskblockswithinadatabasefilegowithaparticularstoredprocedure,for examplethatwouldprobablybeimpossible.Itcertainlysoundsdifficult.ButaBackup2.0 solutionshouldallowustoquicklyrestoreadatabasetoadifferentlocationafterall,a databaseisjustabunchofdiskblocks,andtheyshouldntcarewheretheywindupand wecoulduseSQLServersnativetoolstoscriptoutastoredprocedureandthenrunthat scriptonourproductiondatabase,oruseSQLServertoolstojustcopydatabaseobjects likeusersorwhateverfromonedatabasetoanother. SingleSQLObjects:Tricky,Tricky PartofthereasonitssotrickytorecoverasingleSQLServerdatabaseobject isthatSQLServerstoresobjectslikestoredproceduresastextdefinitions withinasetofsystemtablesinsidethedatabase.Inotherwords,objectsare externallyindistinguishablefromdata;inmostregards,objectsaredata. Ausefultooltohavehandy,then,issomekindofSQLServercomparison tool;useyourfavoritesearchenginetolookforsqlserverdiffandyou shouldfindseveral.Thesetoolscomparetwodatabaseslikearestored databaseandaninproductionversionandshowyouthedifferences.In mostcases,differencescanbeforwardedintotheotherdatabase,makingit easiertospottheexactdifferencesbetweenarestoreddatabaseanditslive counterpart,andtorestorespecificobjectsfromtherestoreddatabaseinto thelivedatabase. SomeBackup2.0toolsetsmightevenincludesuchcomparisonutilities,and mightevenincorporatethemintotherecoveryprocessforamoreseamless experiencewhenyourejustlookingforasingleobjectoutofabackup.
TheresnodoubtthatBackup2.0canofferabetterdisasterrecoveryoptionthanmore traditionaltechniques.JustconsiderFigure5.4,whichcomparesthetwophilosophiesina practicaldisasterrecoverytimeline.
88
Don Jones
Figure5.4:Comparingdisasterrecoverytechniques. Backup1.0isontheleft,whereyourespendingatonoftimemanuallyrecoveringsoftware andlettingSQLServerdealwithitsbackupfiles.Backup2.0isontheright,whereyoure simplypushingalltheserversdiskblocksbacktotheserversdiskrecoveringtheOS, SQLServer,datafiles,andeverythingallatonce,andtoaspecificpointintime. Now,IdorealizethatmanythirdpartybackupsolutionsoftheBackup1.0varietydomake backupsoftheentireserver,andthatmanyofthemofferbootableCDsorDVDsthatcan kickstartawholeserverrecovery.Thatsgreatexceptthatitstillleavesyourecovered toanoldsnapshot.Makingabackupofanentireserverisevenmoretimeconsumingthan backingupasinglelargedatabase;yourelesslikelytohaveanuptotheminutebackup, meaningmoredataisatriskandmoretimewillbespentduringarestore.
89
Don Jones
AnotheradvantageoftheBackup2.0techniqueasIvepointedoutinpreviouschapters isthatdiskblocksreallydontcarewheretheylive.Diskblockscouldberestoredtoa differentserver,iftheoriginaloneshardwarewasirrecoverable.Diskblockscouldbe writtentoavirtualserver,givingyouafantasticoptionforoffsiterecoveryintheeventof adatacenterdisaster,likeafloodorlossofutilitypower.
EasierManagement
Management,forme,iswhereBackup2.0reallyhasachancetoshine.Havingmanagedthe 600ishfilesinvolvedinapreviouscompanysSQLServerbackupplan,Ilovethewaythat Backup2.0doesntfocusonspecificpointintimesnapshots.Thatmeansnomanaging backupfiles.Instead,youmanageasinglebackuprepository,whereallthebackedupdisk blockslive.Ratherthanjugglingfilesandtapes,youuseacentralizedmanagement consoleliketheoneshowninFigure5.5,forexampletomanagetheentirerepository, whichmightwellhandlebackupsformany,manyservers.Youselecttheserveryouneed torestore,selectthediskvolumethatcontainsthefilesyouwanttorestore,andindicate thepointintimeyouwanttorestoreto.Therepositoryfiguresoutwhichdiskblocksare involvedinthatrecoveryoperation,andstreamsthemtotheserveryoudesignate. Anythingfromasinglefiletoanentireservercanberecoveredinthesamefashion.
Figure5.5:ExaminingtheBackup2.0repository. Easiermanagementlettingthesoftwarejugglethebackupdataisoneofthereal advantagesthatcanberealizedwhenwestartrethinkingwhatbackupsareallabout.
90
Don Jones
SQLServerSpecificConcerns
SohowwillBackup2.0helpaddresssomeoftheconcernsthatareuniquetoSQLServer? Obviously,itdependsontheexactBackup2.0stylesolutionyouretalkingabout,butthere arecertainlywaysinwhichsolutionvendorscouldhandleSQLServerissues.
Sprawl
SprawlisntaproblemwithExchangeServerorSharePoint;thoseapplicationsliveinthe datacenter.SQLServer,however,spreadsthroughouttheorganizationindesktoplevel installationswhereusersmightnotevenrealizethattheirdataiscontainedinSQLServer. Evenwithclientlevelbackupagents,thisSQLServerdataoftengoesunprotected;client levelagentsareusuallydesignedforsimplefileandfolderbackup,anddonttypically includeaSQLServerspecificagent.ThehiddenSQLServerinstancesruncontinuously, justlikeanyinstallationofSQLServer,thwartingsimplefileandfolderbackupschemesat theclientlevel. Backup2.0canhelp.BecausethewholeideaofBackup2.0isbasedoncapturingdisk blocks,itcanwedgeitselfintothefilesystemataverylowlevel,usingbuiltinWindows hooksdesignedforexactlythissortofactivity.Figure5.6illustrateswhereaBackup2.0 agentcanworkitswayintothesystemwhileonlyadding1to2%ofoverheadtotheclient system.Thislowlevelofoverheadmakesthistypeofagentperfectlysuitablefor workstationsrunningExpressordesktopinstancesofSQLServer.
Figure5.6:Howbackup2.0fitsin.
91
Don Jones
HereshowIenvisionitworking:ABackup2.0agent,writtenasafilesystemshim, registersitselfwiththeOS.WhenSQLServersavesdatatodiskwhethertoadatabasefile ortoatransactionlogtheshimisnotifiedbythefilesystem.Asthefilesystemwritesthe datatophysicalstorage,theshimcanreadthenewlywrittenblocks,compressthem,and sendthemacrossthenetworktoacentralrepository. Itsfarmoreefficient,especiallyforclientcomputers,thansnapshotbackups.Workstations runninganExpresseditionofSQLServertypicallyhavefairlylowSQLServerworkloads, sinceSQLServerisreallyservingonlyasingleapplicationinusebyoneorafewusers. Ratherthanlaboriouslybackinguptheentiresystemoreverydatabasefileeverysooften, Backup2.0juststreamsthefewdiskblocksthathavechanged.Inasprawlenvironment, itstheperfectwaytocreateconsolidatedSQLServerbackupsforallthatimportantdata thatslivinginallyourstealthSQLServerinstallations.
LogTruncation
IfBackup2.0isjustcapturingdiskblocks,whendoestheSQLServertransactionlogget truncated?Well,insomeinstancesyoumightthinkyoucouldjuststopusingthe transactionlog.AsFigure5.7shows,aSQLServerdatabasecanbeconfiguredtousea Simplerecoverymodel.UnliketheFullmodel,whichoperatesasIdescribedearlierin thischapter,theSimplemodelbasicallytruncatesthelogassoonasatransactions changeshavebeenwrittentodisk.Inotherwords,thetransactionlogstillexists,butits notarecoveryoptionbecauseitwontevercontainverymanytransactionsitllonly containthosetransactionswhosedatapagesarestillinmemory.
Figure5.7:Configuringadatabaseforsimplerecoverymodel.
92
Don Jones
Atfirstglance,itmightseemlikethiswouldbeperfectinaBackup2.0world.Afterall,you stillgettransactionlogrecoveryforanunexpectedserverpoweroutage,butthe transactionlogisselfmaintaininganddoesntneedtobetruncated.YourBackup2.0 solutionisgrabbingdiskblocksalmostinrealtime,andshippingthemofftoabackup repositorysowhatgoodisthetransactionlog? Insomescenarios,IdsaygowithSimplerecovery!Butinothers,Istillliketohavethe pieceofmindthatthetransactionlogoffersandsoIdlookforaBackup2.0solutionthat hadtheabilitytotruncatethelogjustasSQLServerdoeswhenitmakesasuccessful backup.Inotherwords,ifthebackupsolutionisntusingnativeSQLServerAPIswhichdo truncatethelogafterasuccessfulbackupthenthebackupsolutionshouldfullyreplace thoseAPIs,includinglogtruncationcapability.
SingleObjectRecovery,Corruption,andOffLocationRestores
Singleobjectrecovery,corruption,andofflocationrestoresmightseemlikethreepretty randomtopicstothrowtogether,buttheyreallpotentialissuesthataresolvedbythe samething.AsIvedescribedearlierinthischapter,singleobjectrecoveryinSQLServer prettymuchalwaysinvolvesrestoringthedatabasetoadifferentlocation,thencopying objectsfromtherestoreddatabasetotheproductiondatabase.Thatsjustinherentinthe waySQLServerworks.Theproblemisthatrestoringabackup,aswevediscussed,cantake alotoftime.Spendinghoursrestoringfilesjusttograbasingleaccidentallydeletedstored procedureorviewispainfulandunrewardingintheextreme. Therearemanyotherreasonstorestoreadatabasetoadifferentlocation,whichisalso calledanofflocationrestoreoralternatelocationrestore.Onereasonistocomparea backedupdatabasetothecurrent,livedatabase,usingsomekindofcomparisontool. Anotherreasonmightbetoaccessdatathatwasdeliberatelypurgedfromthelive database.Typically,thedatabaseisrestored,frombackup,toadifferentserver,ortothe sameserverunderadifferentdatabasenamebothofwhichrequirescratchspace,or temporaryspacetoholdtheentirerestoreddatabaseforhoweverlongitisneeded.Then, ofcourse,theresalsothetimetorestoreallthedatabasefilesandletSQLServerprocess them. Mythirdissueisoneofbackupdatacorruption,whichisaninsidiousthingweveallrun into:Thebackuptapeiscorrupted!AlthoughBackup2.0reliesless(ornotatall)on tapes,datacorruptionisstillarealconcern,andanydecentbackupsolutionwillofferways todetectcorruption. Allthreeoftheseissuesofflocationrestores,singleobjectrecovery,anddata corruptioncanbehandledbyasinglefeaturewecanaddtoourBackup2.0spec:I proposecallingitbackupmounting.Thinkofitlikethis:Ourbackuprepositorycontainsa bunchofdiskblocks,eachonetimestampedtoletusknowwhenitwascaptured.Theres reallynoreasonwecouldntselectabunchofdiskblocksfromagivenpointintime,feed themtoaspecializedfilesystemdriver,andmountthemlikeanormaldiskvolume. Figure5.8showswhatImean.
93
Don Jones
Figure5.8:Mountingbackedupdiskblocksasadiskvolume. Afilesystemdrivesjobistotakesomeformofstorageandmakeitlooklikeadiskvolume, files,andfolders.MicrosoftbasicallydoesthissamethinginWindows7,wheretheOS allowsyoutomountavirtualmachineimageasadiskvolume.Imsimplyproposingthat Backup2.0includeaspecialfilesystemdriverthatletsWindowsseeselectedportionsof thebackuprepositoryasiftheywereareal,live,readonlydiskdrive. Oncethatsaccomplished,thebackupsolutioncanmakeSQLServerdatabaseandlogfiles availablewithoutperformingarestore.Soinzerotime,yourdatabasefilescouldappear inreadonlyform,ofcourseandbeattachedtoaliveinstanceofSQLServer.Thiswould enablethebackupsolutiontoconductattachabilitytests,todeterminewhetherthe backedupimagecouldbeoperatedasafulldatabase.Ifitcouldnt,thencorruptionwould besuspect.Youcouldalsoattachbackedupdatabasesondemandforcomparison purposesorforsingleobjectrecoverywithouteverhavingtorestoreanything.Youstay moreproductive,youdontneedscratchspace,andyoucanattachaversionofthe databasefromanypointintime.Trulyaremarkablesetofcapabilitiesfromafairly simplisticnotionwhichisreallywhatBackup2.0isallabout:Simple,newnotionsthat radicallychangethewaywework,forthebetter.
ComingUpNext
ThelastmajorserverproductIhavetocoverisSharePoint,whichoffersitsownunique challenges.Infact,SharePointwhichhasrapidlygrowninpopularityinthepastfew yearsmaybethebiggestchallengethatBackup2.0hastoface.AsIhaveinthisandthe previouschapters,Illlookatnativesolutions,coverproblemsandchallenges,andcompare the1.0wayofdoingthingswithamoreenlightened2.0approach.
94
Don Jones
Chapter6:SharePointServerBackups
MicrosoftsSharePointServerhasprobablyhadthemostvarietyinitsbackupandrestore solutions.ThefirstversionoftheproductwasessentiallyamodifiedversionofExchange Server,andusedthesamedatabaseenginethatExchangedidatthetime.Today, SharePointServerusesmultipledatabasestostoreitscontent,configuration,search catalogs,andmoreandevenstoressomecriticalfilesassimplediskfiles.Allthatdata storedindifferentplaceshelpsmakeSharePointServeroneofthemostdifficultMicrosoft serverproductstoworkwithintermsofbusinesscontinuityanddisasterrecovery.It becomesevenmorecomplexwhenyoustartdealingwithSharePointServerfarms collectionsofserversdesignedtoserveupthesamecontentforloadbalancingpurposes.Is itevenpossibletomovebeyondtheBackup1.0mindsetandstartusingBackup2.0whenit comestoSharePoint?
NativeSolutions
MicrosoftdefinesthreelevelsofdatarecoveryforSharePointServer: ContentrecoveryiswhenyourecoveroneormoreitemsusingaRecycleBinor retrieveapreviousversionoftheitemsfromthecontentdatabase.Thisrelieson functionalitywithinSharePointitselfandisaccessibletoendusers. SiterecoveryiswhenyourecoveranentireSharePointServersite,orWebsite. Thisisthetypeofrecoverymostadministratorsareconcernedwith. Disasterrecoverytypicallyinvolvessiterecoverytonewhardware.
ContentrecoverydoesntaffectanythingbuttheSharePointcontentstore;usersutilizethe SharePointuserinterface(UI)toaccesspriorversionsofafileortorecoverdeletedfiles fromtheinproductRecycleBin.Otherformsofrecovery,however,dealwiththesearch catalogs,siteconfiguration,andotherdatastores.Natively,SharePointServersupportsa dizzyingarrayofbackupandrecoveryoptions,allofwhichessentiallyoverlaptoprovide fullcoverageoftheproductsdata.Illcoversomeofthemajornativeoptionsinthenext fewsections.
95
Don Jones
Versioning
Versioningisdesignedtoprovidesingleitemrecoveryforpastversionsofanitem.Whena useroverwritesanexistingdocument,SharePointretainstheoldversion.Userscanalways retrievethoseoldversions,goingasfarbackasSharePointhasretained.Versioningmust beenabledbyanadministrator,andtheadministratorcanselecttwooptions: Majorversions.Eachnewversionofthedocumentisgivenasequentialnumber, suchas1,2,and3.Allusershaveaccesstoallversions,andyoucanspecifyhow manypreviousversionsareretained. Majorandminorversions.Thisallowsforminorversions(1.1,1.2,andsoon)of documents,althoughonlymajorversions(thoseendingin0)canactuallybe published;everythingelseisconsideredadraftrevision.Youcanchoosehowmany majorandminorversionsyouwillkeep.
Versioningisaneffectiverecoverytoolforenduserstoapoint.Itsnotpracticaltostore aninfinitenumberofversions,soatsomepoint,youmaystillhavetogotoatapebased backupinordertoretrieveapastversionthatisolderthanwhatsretainedinthe SharePointdatabase.VersioningalsoassumesthatSharePointisupandrunning,meaning itisnteffectivefordisasterrecoveryorotherscenarioswheretheentireserverislost.
RecycleBin
SharePointincludesatwostageRecycleBin.Thefirststageexistsattheenduserlevel, providingasimpleundeletefeaturethatletsusersrecoveraccidentallydeletedfilesand otheritemsfromwithinthesite.Thesecondstageisaccessibletoadministrators,who alsohaveaccesstoallthesitesfirststageRecycleBinitems.Whenauseremptiestheir firststageRecycleBin,thedataisretainedinthesecondstageRecycleBinforrecoveryby anadministrator. Itemsarealsoremovedcompletelyafter30days(bydefault)intheRecycleBinsystem. Afterthattimeperiodelapses,itsbacktobackuptapestorecoverolderitems.Itemsare alsoremovedfromthesecondstageRecycleBinwhenitreachesitsstoragequota;the oldestitemsarepermanentlydeletedtomakeroomfornewitems.Aswithversioning,the RecycleBindependsonSharePointbeingupandrunning,soitisntintendedasadisaster recoveryorwholeserverrecoverytool.
SharePointDesignerandSTSADMImport/Export
OfficeSharePointDesignerhastheabilitytobackupandrestoreindividualSharePoint sitesandsitecollections,downtotheindividualfilelevel.ThetoolutilizestheSTSADMtool toexportorimportthesite;priortoSharePointServer2007ServicePack2(SP2),backups werelimitedto25MBwhichisntmuch.AfterSP2,backupsarelimitedto2GB,whichisa lotofspacebutstillmightnotbeenoughtocompletelybackupaverylarge,busy SharePointsite.
96
Don Jones
Thebackupscreatedinthisfashionareactuallycontentmigrationpackages;theonlyway toutilizethesebackupsistorestoretheentiresiteoftentoanofflineserverthatisused specificallyforthatpurpose.BackupsdonotincludeRecycleBinfiles,soanythingthatwas deletedwhenthebackupwasmadewillnotbeincludedinthebackup.Thebackupsalsodo notincludeworkflowdefinitions,alerts,orsitecollectionpropertiesmeaningthese backupsdonotcontaineverythingthatdefinesyourSharePointsite. YoucanalsousetheSTSADMtoolbyitselftoexportasite(createabackup)orimporta site(restorefrombackup).Thetoolhasthesamelimitationswhenusedbyitselforfrom theSharePointDesigner. TherearesomeotherweaknessesofSTSADM,includingthefactthat,althoughitcanback uptheSharePointconfigurationdatabase,itcanonlyrestoretheconfigurationtoaserver ofthesamenamemakingituselessfordisasterrecovery.Microsoftsguidancedocument forSharePointbackupandrecovery(http://technet.microsoft.com/en us/library/cc262129.aspx)notessomeotherdisadvantages: CannotbackupdirectlytotapebackuplocationmustbeaUNCpath Doesnotprovideautomaticdeletionofoldbackupfiles Aspartofafarmbackup,canbackuptheconfigurationdatabaseandtheCentral Administrationcontentdatabasebutwillnotrestorethem DoesnotbackupanycustomsolutionfilesintheInetpuborOffice12hive(Program Files\CommonFiles\MicrosoftShared\Webserverextensions\12) Doesnotbackupalternateaccessmappings(AAM) DoesnotbackupInternetInformationServices(IIS)settingsincludinghostheaders, dedicatedIPaddresses,andSecureSocketsLayer(SSL)certificates Sitecollectionbackupsaffectperformanceandcancauseaccesserrorsandshould onlybeusedwhenthesitecollectionislocked;sitecollectionbackupscanbeslow whenworkingwithcollectionslargerthan12to15GB,soMicrosoftrecommends thatyouusefarmbackupsifyouareworkingwithcollectionslargerthan15GB
Contentdatabaseslargerthan100GBwhichisntreallythatmuchintodaysworld arentsuitableforSTSADM,andMicrosoftactuallyrecommendsyougobuysomething fromsomeoneelsetohandleyourbackups.
SQLServerBackup
BecauseSharePointServerusesSQLServerasitsmaindatastore,SQLServersownbackup toolscanbeusedtocreatepartialbackupsofaSharePointsite.Isaypartialbecausenotall oftheSharePointsitedataisinsideSQLServer.
97
Don Jones
YoucanalsocreateaSQLServerdatabasesnapshotoftheSharePointsite.Thisisaread onlyversionofthedatabaseasitexistsatthetimeofthesnapshot.Thesesnapshots shouldnt,however,beconsideredrealbackups,althoughtheydoprovideameansto retrievedatafromthedatabaseasitexistsatthetimeofthesnapshotsuchasrecovering individualitems.DoingsorequiresaSharePointsitetobecreatedthatusesthedatabase snapshot,andthatsitewillbereadonly;obviously,SharePointServerneedstobe functionalinordertodothat.Thereareotherdisadvantagesagain,fromMicrosofts guidancedocument: DoesnotincludefrontendWebservercustomsolutions CanbackuptheconfigurationdatabaseandCentralAdministrationcontent databasebutrestoringisnotsupported DoesnotbackupIISsettingssetoutsideofOfficeSharePointServer,includinghost headers,dedicatedIPaddressesandSSLcertificates IfusingSearch,youmustrecrawlafterrestoringcontentbecauseindexesarenot backedupinSQLServer ShouldnotbeusedtobackuptheSearchdatabasebecauseitcannotbe synchronizedwiththeSearchindex YoumustmanuallyreattachyourdatabasestotheWebapplicationsafterarecovery
Whydothesenativetoolssupportbackinguptheconfigurationdatabasebutnotrestoring it?Whatwouldbethepoint,exactly?
CentralAdministration
ThecentralSharePointadministrativeinterfaceprovidesbasicbackupandrecovery capabilities.Itseasytouse,andbraceyourselfcanautomaticallyhandlebackupsthat runmorethan17hours.Iknow,theverythoughtofa17hourbackup,letalonesomething thatrunslonger,issomiredintheBackup1.0mindsetthatIdontevenwanttothink aboutit. CentralAdministrationdoes,however,havealengthylistofdisadvantages(from Microsoftsguidancedocument): Doesnotprovideschedulingfunctionality Doesnotprovideautomaticdeletionofoldbackupfiles Cannotbackupdirectlytotape;backuplocationmustbeaUNCpath Aspartofafarmbackup,canbackuptheconfigurationdatabaseandtheCentral Administrationcontentdatabasebutwillnotrestorethem DoesnotbackupanyconfigurationchangesorcustomsolutionfilesintheInetpub orOffice12hive(ProgramFiles\CommonFiles\MicrosoftShared\Webserver extensions\12) DoesnotbackupanycustomizationsmadetotheWeb.configfile
98
Don Jones
DoesnotbackupAAM DoesnotbackupIISsettings,includinghostheaders,dedicatedIPaddresses,and SSLcertificates Ifabackuporrecoveryjobisnotsuccessful,theunsuccessfuljobmustbemanually deletedfromtheTimerjoblistontheBackupandRestoreStatuspage;inCentral Administration,clickOperations,thenclickBackupandRestoreJobStatusif thefailedjobisnotdeletedmanually,subsequentbackuporrecoveryjobsfail
So,ithastobeusedmanually,doesntbackupeverything,cantbackuptotape,andhasto bemanuallymonitoredandmaintained.Wonderful.ThatsnotevenBackup1.0;itsmore likeBackup0.5!
Thebiggestproblemwiththenativesolutionsisthatwell,frankly,theyrealloverthe place.Noneofthemarereal,triedandtruebackupsolutionsinthetraditionalsenseofthe word.NoneofthenativesolutionsIvediscussedprovidefordisasterrecovery,for example,whichisimportantevenintheBackup1.0mindset. Disasterrecovery,infact,canbeincrediblycomplicatedwithSharePoint.Youhavethe contentdatabase,customizations,SharePointlevelconfigurationsettings,binaryfiles (SharePointServersownfiles),IISlevelconfigurationsettings,andfinallythebinaryfiles oftheWindowsoperatingsystem(OS)itself.Nothinginthenativetoolsetcanbackupall thatinformation. Microsoftsrecommendationfordisasterrecoveryisto: UseWindowsServerBackuptobackupthebinaryfilesalthoughinafulldisaster recoveryscenario,MicrosoftrecommendsreinstallingtheOS,SharePoint,andSQL Server,whichwilltakeseveralhourstocomplete. Documentwritedown,thatis,notactuallybackuptheIISconfigurationsettings. Youresupposedtomanuallyrecreatethecorrectconfigurationinadisaster recoveryscenario. Documentagain,writedown,notactuallybackuptheSharePointconfiguration settings.Althoughitspossibletobackuptheseandrestorethem,doingsois dangerousbecausetheyneedtobeperfectlysynchronizedwiththeotherdata. Failuretomaintainthatsynchronization,whichisdifficulttodomanually,will resultinrandomsiteerrorsandforceyoutorecreatetheSharePointsitefrom scratch. Packageanycustomizationsassolutionssothattheydonthavetobebackedupbut caninsteadbereinstalledifyouneedtorebuildthesite. BackuptheSQLServerdatabasestoprotectthesitecontent.YoucanuseSQL Servertoolsforthis,exceptinthecaseoftheSearchdatabase,whichmaynotbe backedupusingSQLServertools.YouhavetouseSharePointtoolstobackupthat. 99
Don Jones
Honestly,youvegottobekidding.SharePointappearstobeaproductthatsimplydoesnot liketobebackedup.Sure,itsnicetohavegreatfunctionalitysuchasversioninganda RecycleBinforsingleitemrecoverybyendusers,butwhataboutacompletedisaster? How,exactly,areyousupposedtoprepareforawholeserverfailure?Andforthatmatter, howareyousupposedtogoaboutrecoveringdatathatstoooldtobeinaRecycleBinorin theversioningsystem?Whatifyouneverturnedonversioningandneedtorecoveranold file? Partofthechallenge,ofcourse,isthatSharePointisahugeandcomplexproduct(Figure 6.1illustratesthevariousindependentcomponents)thatreliesonnumeroustechnologies outsidethecontroloftheSharePointdevelopmentteamatMicrosoft.ItreliesonspecificIIS configurationsettings,andIIShasnevermadeitespeciallyeasytobackupthose configurations(although,asofIIS7,itsgotteneasierWebsitesettingsareinasimple web.configfilethatcanbebackeduplikeanyotherfile).SharePointsitesoftenhave customizedfiles,whichresideontheserversfilesystem.SharePointdatalivesinSQL Server,whichhasitsownbackupandrecoverymodel.Worse,muchofSharePoints configurationdataasIvedescribedreliesonbeinginexactsyncwiththedatastores,so youprettymuchhavetobackupeverythingatonce.Fromabackupandrecovery standpoint,SharePointjustisntwellthoughtout,unfortunately.
Figure6.1:ThemanycomponentsofSharePointServer. SohowdoesBackup1.0workinaSharePointenvironment?
100
Don Jones
IntheOldDays
TheweaknessesofBackup1.0arereallybroughtintosharpreliefbyacomplexproduct likeSharePointServer.Microsoftsrecommendation?Buyabackupsolution,suchasthe oneMicrosoftwillbehappytosellyou.ManybackupsolutionssupportSharePointServer, butalldosofromaveryBackup1.0perspective:pointintimesnapshots,lengthybackup windows,andevenlongerrecoverytimes.
BackupTechniques
Forstandardbackups,youhavetwochoices:Usethemishmashoftoolsthatarenativeto SharePointoravailableforfree,orbuysomethingthatcanbackuptheentireSharePoint infrastructureinonego.Eitherway,largeSharePointinstallationscanhaveincrediblylong backuptimesthefactthatthenativetoolshavespecificfeaturestoaccommodate backupslongerthan17hoursshouldtellyousomething. IknowalotofSharePointadministratorswhoruntheirSharePointServerinstallations SQLServerandallinavirtualmachinesimplybecausethatallowsthemtograbcopiesof thevirtualmachinesvirtualdiskfile,thusbackinguptheOSandalltheSharePointbitsin onepiece.Figure6.2illustratesthebasicidea.
Figure6.2:EasierbackupswithvirtualizedSharePoint. Theproblemwiththisapproachisthatitstoughtoactuallyusethosebackupsforanything exceptdisasterrecovery.Yourestuckrestoringtheentireserver,andifyouonlyneededto recoveronefilewell,thatsdefinitelyabitofoverkill.
101
Don Jones
OtherBackup1.0stylesolutionstrytoduplicatetheapproachofthenativetoolsbut consolidateeverythinginonetoolforyou.Inotherwords,thesesolutionsdiginto SharePointandgrabtheIISconfiguration,thecontent,theSharePointconfiguration,andso onbutratherthangivingyoueighttoolstodoit,theydoitallatonce.Thatcanbe beneficial,butitsstillapointintimesnapshot,andyouvealwaysgotdataatrisk. Performingrecoveryfromthatkindofbackupcansometimesbetricky,too,dependingon howthebackupsolutionisbuilt. Bytheway,IdontwanttoimplythatMicrosofthasntthoughtaboutallthesecomplexities. Theyhave;oneofthecompletebackupandrecoverysolutionsthatyoucanbuyismade byMicrosoft.
RestoreScenarios
Recoveryscenariosusingthenativetoolsetarejustascomplicatedasperformingthe actualbackupusingthenativetoolset.Infact,asIwasresearchingthischapter,I commentedonFacebookthatIcouldntbelievehowcomplexthenativetoolsetwas;oneof myfriendsresponded: Itsmoreliketheyneverexpectthesystemstoeverfail&thusneedarestore!HA!I amsohappytobeoffoftheSharePointteam! Ouch.ButIcouldntagreemore:Natively,SharePointdoesassumethatthesystemwill neverfail,andthatRecycleBinsandversioningwillprovidealltherecoverycapability youlleverneed.Ifyoudoexperienceacompletesystemfailure,theofficial recommendationistoessentiallystartfromscratchandreinstalleverything,thenrestore yourcontentdatabasefromastandardSQLServerbackup. Ofcourse,Microsoftisntthatnavethecompanyabsolutelyrecognizestheneedforfull systemrecovery,andthatswhytheymakeanaddonproductthatwilldoitforyou. However,onceyouvedecidedthatyouhavetospendmoneytoprotectSharePointand, frankly,youdoyoushouldstartlookingbeyondMicrosoftandcomparetheirofferingsto theothersolutionsoutthere. ThoseothersolutionsincludingtheoneMicrosoftwillsellyoucansometimesdoa betterjobathandlingthetworecoveryscenariosyoullruninto:singleitemrecoveryand disasterrecovery.Illgetintodisasterrecoverynext;fromasingleitemrecovery perspective,mostsolutionsworkbypullingthedataoutofSharePointandstoringthe backedupdatainawaythattheycanalsoaccess.Itsanapproachsimilartothewaysome solutionsbackupExchangeServer:Ratherthanattemptingtograbtheentiredatabasein onechunk,theygrabeachindividualcontentitem.Thatway,youdontneedacopyof SharePointtoaccessindividualitemsfromthebackupyoujustusethebackupsoftware itself.
102
Don Jones
IndividualItemBackup:GreatforDeduplication Invariably,SharePointsiteswindupwithalotofduplicateddatacopiesof fileskeptinmultipleplaceswithinthesite,forexample.Asolutionthat accessesindividualitemsforbackuppurposescancomparethemwitheach otherandeliminateduplicatesfromthebackup,makingthebackupssmaller andsavingspace.SomuchspacecanbesavedinatypicalSharePoint installationthatdeduplicationhasbecomeamajorsellingpointfor SharePointbackupandrecoverysolutions. Othersolutionstakeaslightlydifferentapproachtohowtheymaketheirbackup,butthe practicalupshotisthis:Youwantasolutionthatwillallowyoutoaccessoneormore individualitemswithouthavingtorestoretheentiredatabasetoaliveSharePointServer. Thereisactuallyawholemarketoftoolsthatcanaddthiscapabilitytootherbackup solutions.Forexample,toolsexistthatcanopenandbrowsethebackupfilesmadeby Microsoftsownextracostbackupsolution,enablingyoutobrowsetheSharePoint repositorieswithoutactuallyperformingadatabaserestore.Figure6.3showsonesuch solution,whichisafreetoolavailableat http://www.appassure.com/applications/docretriever/.ItcanconnecttoSQLServer backups,STSADMexports,andotherbackupsnapshots;letyoubrowsethedatabase;and retrieveindividualitemsfromit.
Figure6.3:DocRetrievercanrecoverindividualitemsfromaSharePointbackupfile.
103
Don Jones
DisasterRecovery
AsIvealreadysaid,theofficialdisasterrecoverypolicyforSharePointistoreinstall, reconfigure,andthenrestoreyourcontentdatabaseaprocessthatwill,inmostcases, takeadayortwoatleast(includingtimetoapplypatchesandservicepackstotheOSand applicationsoftware)andthatsassumingyouvedoneeverythingbythebook,packaged anycustomizationsasdeployablesolutions,andsoon.Ifyouhaventdonethose thingswell,sorry.Youreinforalongrecoveryproject. Ivealsomentionedvirtualizationasawaytomakedisasterrecoveryeasier.Bybackingup avirtualdiskfile,youcanrestorethatfileinasingleoperationthusrestoringyourOS, SharePointbinaryfiles,databases,configurations,andeverythingallinasingleoperation. Ofcourse,youmaybelimitedinhowoftenyoucaneffectivelygrababackupofavirtual diskfile:Togetaconsistent,reliablecopyofthefile,youregoingtoneedtoshutdownthe virtualmachine.AlargeSharePointinstallationmayusevirtualdiskfilesthataredozensof gigabytesinsize,whichmaytakesometimetocopyalthoughyoucanmakeadisktodisk copy,thenbackupthecopytodiskwhileyourestartthevirtualmachine. ForExample IhaveonecustomerwhoseSharePointsiteincludesabout110GBoffilesand data;theirvirtualmachinesvirtualdiskfilestotalabout146GB.Evenona faststorageareanetwork(SAN),ittakesafewhourstomakeadisktodisk copyofallthevirtualdiskfiles,andittakesseveralhourstobackthecopyup totapeforoffsitestorage. Ifyoureusinganaddonbackupsolution,oddsarethatitprovidesadisasterrecovery path.Somerequirethat,ataminimum,youreinstallWindows,andmayrequirethatyou reinstallSQLServerandSharePointalso;othersbackuptheentireserverandcanrestore theentireserver.Fromadisasterrecoveryperspective,Iprefersolutionsthatcanrestore thecompleteservertobaremetalbecausethatstheeasiest,mostfoolproofwaytogetthe serverupandrunningagainasquicklyaspossible.
BackupManagement
ManagingbackupsinaSharePointServerworldcanbecomplicatedabitmoresothan theusualbackuptapeshufflethatwereallfamiliarwith.Becausethevariousbitsof SharePointconfigurations,content,databases,files,andsoforthareallsotightly interconnected,youhavetomakesurethatallthevariousbackupsmatchup.Inother words,youneedtofiteverythingfromasinglebackupontoasinglesetofbackuptapes. BecauseSharePointbackupscanbesotimeconsuming,administratorstendtorelyon techniquessuchasdifferentialandincrementalbackups,whichallowyoutomakemore frequentbackupswithoutsuchlengthybackupwindows.However,allthoseincrementals anddifferentialsalsocomplicatebackupmanagementandmakerecoveryprocesseslonger andmorecomplex.
104
Don Jones
IrememberoneparticularlyunrewardingexperiencewithSharePoint,wherewehada completebackupfromaboutamonthbackandincrementalbackupssincethen.Wewere usingathirdpartybackupsolution,andwemadeanincrementalbackupeverynight.One day,theserverdiedandcorruptedalotofdiskfiles,soweneededtorestoreeverything. ThatswhenwefoundoutthattheincrementalbackupoftheSharePointconfiguration databaseatiny,tinylittlefilefromaboutaweekbackwascorrupted.Thatmeantallthe backupsofthatsamedatabasesincethenwereuselessbecausetheyallreliedonhavinga continuoussequenceofincrementalbackups.Becausetheconfigurationissocloselytiedto thecontentandeverythingelse,webasicallyhadtostoprestoringdataatthatpoint meaningwelostoveraweekofdata,eventhoughwehadmorerecentbackupsforthe actualcontent.Wetriedseveraltimestoextractthedatainvariousways;ultimately,we woundupusingathirdpartysolutiontoconnecttothecontentdatabasebackup,and manuallyextractedtheindividualitemsthathadchanged.Notfun,andittookalmost2 weekstogeteverythingalmostbacktonormal.ThatsBackup1.0foryou.
RethinkingSharePointServerBackups:AWishList
CanBackup2.0makeSharePointbackupsanylesscomplicated?ItseemslikeBackup2.0 mayhavemetitsmatch:ThisisthefirsttimeIvelookedataserverproductthathasits ownbinaries,reliesonIIS,reliesonSQLServer,andstoresfilesanddatainahalfdozen differentplaces.TheBackup2.0concepthasitsworkcutoutforit.
IvementionedthatsomeadministratorsvirtualizetheirSharePointinstallationssimplyto makebackupslesscomplicatedanditturnsoutthatthoseadministratorsareontheright path.Notwithvirtualizationperse,butratherwiththeideathatbackinguptheindividual componentsofSharePointisoverlycomplex;therightideaistobackuptheentireserver, grabbingSharePointsdataalongwitheverythingelse. TakealookatFigure6.4.IveredrawnFigure6.1tosimplifyitabitandtoillustratethat althoughSharePointhasmanycomponents,allofthemstoretheirdataontheserversfile system.Disksare,afterall,theonlypersistentformofstorageinatypicalserver.
105
Don Jones
Figure6.4:EverythinginSharePointendsupondiskeventually. ThetricktomakingSharePointbackupseasieristoforgetaboutSharePoint.Ignorethe databases,ignorethefiles,ignoretheconfigurations,ignoreitall.Instead,simplybackup everyblockofdiskspace.Asdataischanged,thatdataiswrittentodiskintheformof changeddiskblockshaveanagentontheservergrabthosechangesandsendthemoffto abackuprepository. Thosebackedupdiskblockscanalsobetimestamped,meaningtheBackup2.0solution cankeeptrackofwhichchangescamefromwhatpointintime.Thatimmediatelysolves theproblemwithkeepingtheconfigurationssynchronizedwiththecontent;solongas youregrabbingallthediskblocksuptoaspecificpointintime,youllhaveaconsistent SharePointrestore.Thatalsomeansyoucanchoosetorestoreuptoanypointintime, effectivelyrollingbackSharePointtosomeearlierpointintime,ifneeded.Figure6.5 showshowanagentmightcapturethechangeddiskblocksandsendthemtoatime stampedrepository. ThebeautyoftheBackup2.0approachisthatthebackupsolutiondoesntreallyneedto knoworcarewhatsrunningontheserver,orhowcomplicateditis.Everythingthat matterstousashumanbeingswillendupondisk;solongaswecaptureeverysingle changeddiskblock,wevegotacompletebackup.Becauseadiskblockistinynomore than64kilobytes,andassmallas4kilobytesbackingupasinglediskblockisntvery timeconsumingorintensive.Streamingthosechangesoverthenetworkusing compression,ofcourseisentirelypossible,andgetsallthebackedupdatatoasafeplace almostinrealtime.
106
Don Jones
Figure6.5:Streamingdiskblockstothebackupserver. Deduplicationofdataisalsopossible,meaningbackupscanbesmaller,takeuplessspace, andevenrequireabitlesstimetorestore.
AsIsaidbefore,youreallygetintotworecoveryscenarioswithSharePoint:needingto recoverasingleitemandneedingtorecoveranentireserver.ThelatteriswhatIcall disasterrecovery,andIllgettoitnext.Fornow,doesBackup2.0improvesingleitem recovery?
107
Don Jones
Absolutely.Remember,becausethosebackedupdiskblocksaretimestamped,youcan recovertheentireservertoaspecificpointintimeoryoucanexposethedatastoresas theyexistedatacertainpointintime.ThisisaconceptIvementionedbefore,butits particularlyapplicabletoSharePoint,soletmerunthroughanexample. LetssayyourgoalistoretrieveasingledocumentfromSharePoint,asthatdocument existedatnoononthepreviousMonday.Thedocumentlivesinthecontentdatabase, whichisaSQLServerdatabase.SQLServerdatabasesarefilesthatsitonthefilesystem; typically,acertainamountofdiskspaceisallocatedtothedatabase,whetherthedatabase actuallycontainsthatmuchdataornot.Astheallocatedspaceisfilled,thedatabasefileis expandedtomakeroomformoredata.Soletssaythatthisparticulardatabaseoccupies diskblocks10,000through790,000(forthisexample,weliveinaperfectworldwhere diskfragmentationdoesntexist,sothedatabaseoccupiesacontiguoussequenceofdisk blocks).Figure6.6showsaportionofthisdiskspaceblocks11,100through11,123.
Figure6.6:Diskblocksonthefilesystem. WhenwestartourBackup2.0stylebackup,wetakeasnapshotofeverydiskblockonthe serverasourbase,orstartingpoint.Then,wesimplycapturediskblocksastheychange. Figure6.7showsapartialexample:thebasesnapshotforourrangeofdiskblocksaswell aschangeddiskblockscapturedfromthreetimeperiods.
108
Don Jones
Figure6.7:Backingupdiskblocksfromthefilesystem. Whenthetimecomestoperformourrestore,weneedtoreconstructthefilesystemasit wasat,say,12:02.Sothebackupsolutiongoesandstartswiththebasesnapshot,then substitutesanydiskblocksthathavechangedbetweenthetimethatsnapshotwastaken andtherecoverytimewespecified.Intheeventthatadiskblockhaschangedmultiple times11,102isagoodexampleweuseonlythelatestversion,uptotherecoverytime specified.Letssaythatwereonlyconcernedwithdiskblocks11,100through11,107 becausethoseholdthedataforthefilewewanttorecover.Figure6.8showshowthe backupsolutionwouldapplyblockstorecreatetheserversfilesystemasof12:02.
109
Don Jones
Figure6.8:Recoveringtoaspecificpointintime. Oncethebackupsolutionhasreconstructedthispointintimefilesystem,wehavetwo choices.Wecouldsimplyrestoretheentireservertothatpointintimeperhaps recoveringtoavirtualmachinesothatwecoulduseSharePointitselftoactuallyretrieve thefile.Or,thebackupsolutionmightbeabletoexposethispointintimeviewofthefile systemasamountableimage,meaningwecouldbrowseitsfilesystem,attachthedatabase toaSQLServerinstance,orevenusearecoverytool(likethefreeoneImentionedearlier) thatcanattachtotheSQLServerdatabasewithoutneedinganactualrunninginstanceof theSQLServersoftware. Thisisamuchbetterrecoveryscenario:Itseasy,prettyintuitive,anddoesntrequirealot ofwork.Abackupsolutionmightbeabletoconstructthispointintimesnapshotalmoston thefly,meaningwecouldgetinandbrowsespecificpointsintimewheneverweneededto.
110
Don Jones
DisasterrecoveryiseasierunderBackup2.0,aswell.Rememberourmissionstatementfor Backup2.0: Backupsshouldpreventusfromlosinganydataorlosinganywork,andensurethat wealwayshaveaccesstoourdatawithaslittledowntimeaspossible. Becausewerestreamingthosediskblocksinnearrealtime,wewontloseverymuchdata atallafewsecondsworth,perhaps.Wecanaccessourbackedupdataalmostinstantly, asIvedescribed,andwecanrestoreanentirepointintimeimageanytimeweneedto performdisasterrecovery. SharePointisprettypickyaboutitsconfiguration.IfyousetupaSharePointServernamed SPOINT1,thenneedtorestorethatserver,therestoredserverhadbetterbenamed SPOINT1orSharePointmightnotworkproperly.WithaBackup2.0solution,were restoringtheentireOSduringdisasterrecoverysothingsliketheservernamewill remainintact.However,becauseweresimplywritingdiskblocksbacktoadisk,werenot limitedinwritingthembacktotheoriginalphysicalserverwhichmayhavefailed.We canwritediskblockstoavirtualmachine,allowingustobringSharePointbackonlineeven inanoffsiterecoveryscenario.
EasierManagement
Managingbackuptapesismucheasierbecauseinsomeinstancesyoumightnotevenhave backuptapes.Personally,IlovetheadditionalfeelingofsecurityIgetfromhavingcopiesof mybackupsoffsite,andtapesareagreatmediumforit.Thus,yourBackup2.0solution shoulddefinitelybeabletomovecopiesofitsdiskblockrepositorytotape.Butshuffling tapesisalotlesscriticalbecausefromatapeperspective,youreonlybackinguponething: thebackuprepository.Abackupofthebackup,ifyouwill;thus,thetapesareliterallyjust PlanBintheeventyourbackupserverdiesorbecomesinaccessibleduetoadatacenter problemorotherfacilitydisaster.
SharePointSpecificConcerns
SharePoint,asIvepointedoutpreviously,isnotwithoutitslittlequirks,andanybackup solutionusedwithSharePointhastoembracethosequirksandcatertothem.Backup2.0is nodifferent,soletsexaminesomeofthemajorSharePointspecificconcernsandseehow Backup2.0mightdealwiththem.
SingleObjectRecovery
AlthoughSharePointsversioningandRecycleBinfeaturescanprovideenduserswiththeir ownsingleitemrecovery,thosefeaturesonlygosofar.Youlldefinitelywanttheabilityto retrievesingleitemsfromyourbackups,andaBackup2.0solutionusingatimestamped diskblockrepository,asIvedescribedshouldbeabletoprovidethatcapability.By mountingaspecificpointintimeasareadablefilesystem,youcanattachallkindsoftools tobrowsetheSharePointrepositoryandretrieveindividualdocuments;inaworstcase scenario,youcouldrestoretheentireservertoaphysicalorvirtualmachineanduse SharePointitselftoaccessthefilesyouneed. 111
Don Jones
SomeBackup2.0stylesolutionsmayevencombineallthefunctionalityyoullneed: mountingthepointintimesnapshot,attachingtotheSharePointdatastores,and presentingtheSharePointdatainagraphicaluserinterface(GUI)whereyoucanbrowse therepositoryandselectthefilesyouwanttoretrieve.
ProtectingDependencies
SharePointsconfigurationanddataneedstostayproperlysynchronized;youcantrestore aconfigurationdatabasefrom2monthsagoandrestoreyesterdayscontentdatabaseand expecteverythingtowork.ThatswhyMicrosoftsofficialguidancerecommendsagainst actuallybackinguptheconfigurationandinsteadsuggeststhatyoudocumentthe configurationandrecreateitmanuallyintheeventofadisaster.Becausemanuallyis whywehavecomputersinthefirstplace,right? Backup2.0avoidsthatcomplexitybysimplytreatingtheentireserverasasingleunitof management,andbyallowingyoutorestoretheentireservertoaspecificpointintime. Thatmeansyoucanalwaysrestorethecontentdatabaseandtheconfigurationdatabaseas itexistedatthatexactsamepointintime.ItsjustinthenatureofhowBackup2.0works: Yougeteverything,allthetime.
ComingUpNext
Inthenextchapter,Illtackleasubjectthatsnearanddeartomyheart:virtualization backups.Virtualizationhaschangedthewaywethinkaboutourdatacenters,andhas enabledanumberofnew,flexiblecomputingscenariosthataresavingbusinessesmoney andhelpingthemtobemoreagile.Atthesametime,however,virtualizationhasupset manyofthetriedandtrueIToperationspracticesthathavebeendevelopedoverthe years.Inmanyways,virtualizationisComputing2.0,andBackup1.0justisntagoodfit. IllexplainhowpeoplehavebeentryingtomakeBackup1.0getalongwithvirtualized infrastructures,andsuggestsomewaysinwhichBackup2.0coulddoamuchbetterjob andhelpfullyrealizethepromiseofvirtualization.
112
Don Jones
Chapter7:VirtualizationServerBackups
Virtualizationisthehotnewcodewordfortodaysbusinesses,anditschangingeverything wethoughtweknewaboutbackupandrecovery.IactuallyconsiderthattobeaReally GoodThing,becauseitmeansatleastwithregardtovirtualizationwedonthavetoun learnasmanyBackup1.0habitsinordertoseehowaBackup2.0techniquemightbemore effective.
NativeSolutions
Oddly,itsalmostlikevirtualizationvendorsknowthatbackupsarecomplicatedandthat nativesolutionsareusuallydeficientinkeycapabilities,becausemostvirtualization vendorsweretalkingCitrix,VMware,andMicrosoft,heredontreallyprovideany nativebackupcapabilitiesatall.Sure,mostofthemhavebackupsolutionstheyllsellyou, butmostofthosearestillgoodoldBackup1.0style,getitdoneduringtheeveningbackup windowsolutions. Butletstakeastepbackandlookatwhyvirtualizationbackupshavethepotentialtobe morechallengingthanaphysicalserverbackup.
AsillustratedinFigure7.1,avirtualizedserverrunsonavirtualizationhostlikeVMware vSphereorWindowsHyperV.Thehostcontrolsthehardwareofthephysicalmachine, whilethevirtualizedserverhasitsown,virtualizedhardwareenvironment.Thevirtual serversharddisksaregenerallyjustfilessittingonthephysicalhost.Irealizethisis probablynothingnewtoyou;Imjustsettingsomecontext.
113
Don Jones
Figure7.1:Virtualserverrunningonavirtualhost. Thiswholearchitecturemeansthat,unlikewithanormalphysicalserver,wevegottwo placeswecouldrunbackups.Thefirstplaceisonthevirtualizationhostitself,wherewed simplybackupthefilesthatrepresentdrivesforthevirtualservers.Inotherwords,ifwe grabtheCDrive,DDrive,andEDrivefiles,werereallybackingupthecomplete contentsofwhatthevirtualserverseesasitsC,D,andEdrives.Thatwouldseemtobea goodideabecausewegeteverythingthevirtualserverhas. Theproblemisthat,usingpurelyBackup1.0technologies,doingthistaskiskindoftricky. Thoseharddiskfilesarealwaysopenandalwaysinusewhilethevirtualserverisrunning. Thesimplestthingwouldbetotemporarilyshutdownthevirtualserver,backupallofits files,thenstartitupagainandthatsexactlywhatmanycompaniesdo.Butthattakesus backtobackupwindows,pointintimebackups,andotherBackup1.0downsides. Thesecondpossiblescenarioistorunbackupsoftwarewithinthevirtualserver.The backupsolutionwouldseeaC,D,andEdrive,andwouldbackthemupexactlyasifthe backupsolutionwererunningonaphysicalserver.Thatmeanswewouldbackup Exchangeexactlyaswealwayshave,backupSQLServernormally,andsoon.Nothing changes,really;thebackupsolutionisunawarethatitsrunninginsideavirtualmachine (VM),soitbehavesasifvirtualizationdoesntreallyexist.
114
Don Jones
ThereareprosandconstoboththeinsidetheVMandtheoutsidetheVMscenario. WhenyourebackinguptheentireVMfromoutside,youregettingeverything.Essentially, thosevirtualdiskfilesarediskimages,andbybackingupthosediskfiles,youreenabling prettystraightforwardrestoreanddisasterrecoveryscenarios.However,youloseanykind ofgranularityyourebackinguptheentireserver,andthatsallyoucanrestore,toothe entireserver.Thefactthatitsavirtualservermakesitveryeasytorestoretoadifferent locationjustpickadifferentvirtualizationhost. Theinsideapproachgivesyougranularity,meaningyoucanbackupindividualfilesand soon.However,youpickupsomeofthedownsidestoanyphysicalserverbackup:Inorder torestoretheserverfrombaremetal,youhavetoinstalltheoperatingsystem(OS)and yourbackupsolution,ataminimum. TheresatwisttoVMstoragethatcomplicatesallofthistheStorageAreaNetwork(SAN). Figure7.2illustratesthedifferences.
Figure7.2:AvirtualserverusingaSAN.
115
Don Jones
Virtualserversdonthavetostoredataonvirtualdisks.Theyrealsocapableofconnecting toaSAN,andstoringtheirdatadirectlyonitwithoutcreatingthefictionofavirtualhard disk.Butdoesthatcreatenewpossibilitiesforbackups?Notnecessarily.SANsgenerally assignbigchunksofdiskspacetoaparticularserverforaccess,meaningwhateverSAN diskspaceisbeingusedbythevirtualservercanonlybeusedbythevirtualserver.Many SANsdosupporttheabilityforbackupsolutionstoonlyreadthediskspaceassignedtoa server,meaningthebackupsolutioncouldgrabthevirtualserversfilesandfolderswhile thevirtualserverwasrunningbutthatsaddingalayertoyourbackupandrecovery scheme. Boththeinsideandoutsidetechniqueshaveprosandcons,butwhichoneisbest?Most importantly,whichonecomesclosesttoourBackup2.0manifesto? Backupsshouldpreventusfromlosinganydataorlosinganywork,andensurethat wealwayshaveaccesstoourdatawithaslittledowntimeaspossible. Toanswerthosequestions,letslookmorecloselyatbothtechniques.
IntheOldDays
LetslookatvirtualserverbackupfromapurelyBackup1.0perspective,whichiswhat mostcompaniesareusingtoday.Ivealreadyoutlinedboththeinsideandoutside techniques;nowletsdiveintoabitmoredetail.Tokeepthingssimple,Imgoingtofocus onMicrosoftsHyperVvirtualserverhypervisor;obviously,thetechniquesImdiscussing aresubstantiallysimilaronothervirtualizationplatforms,butthisbookhashadanice Microsoftfocustothispoint,soIllkeepitthereforthespecifics.
BackupTechniques
Letsconsidertheinsideapproachfirst,whichmeansyouhaveabackupsolutionrunning insideeachandeveryVM,grabbingthefilesandfoldersanddatafrominsidethatVM.This isastraightforwardtechniquethatalotofcompaniesusebecauseitsexactlythesameas thetechniquestheyvealwaysusedforbackingupservers.Abackupsolutioncangrab individualfilesandfolderssubjecttoalltheBackup1.0limitationsthatIvediscussedin theprecedingsixchapters,ofcourse. Adownsideisthatyourereallydevotingabiggerchunkofphysicalprocessingpowerto runningbackups.ImagineyouhaveasmallerHyperVhostrunningamerefiveVMs.You runyourbackupsduringaneveningbackupwindow,andyourbackupsolutionconsumes about10%ofeachVMsprocessingpowerwhichisntmuch.Ataphysicallevel,however, thoseindividual10%bitesmightadduptoaround50%ofthehostmachinescomputing powerjusttoperformbackups.Figure7.3illustrateswhatImtalkingabout.
116
Don Jones
Figure7.3:Processingpowerusedforrunningbackups. Imdeliberatelysimplifyingthenumbersherejusttomakeapoint,whichhopefullyisclear enough:RunningbackupsinsidetheVMsis,inaggregate,usinganoticeableamountof physicalprocessingpower.Itskindofawaste,butitsgettingyoueverysingleindividual fileandfolderfromwithintheVMs,meaningyoucandoverygranularrestorestothe singlepointintimewhenthebackupwasmade,ofcourse.Itwouldprobablynotbe efficienttorunbackupsallthetimefromwithintheVMs;theaggregateuseofphysical processingpowerwouldbetoohigh. WhataboutrunningaBackup2.0stylebackupsolutionwithintheVMs?WhatIve suggestedisanagent,runninginsidetheVM,thatgrabschangestoindividualdiskblocks astheyarewrittentodisk.Suchanagentmightadd2to3%overheadtotheserver,which isabsolutelyminimal.Again,however,thatsperVM;that2to3%wouldadduptoa sizablechunkofphysicalprocessingpower,whichstillseemslikeawaste.Idontmind dedicating2to3%ofmycomputingpowertobackups,butdedicating10%ormoreofa HyperVhostscomputingpoweritjustseemsliketheressomethingbetterIcouldbe doingwiththatpower. Okay,soletsmovetotheoutsidetechnique,wherewerunsomekindofbackupsolution rightontheHyperVhost.Imonlygoingtobelosing2to3%overheadtotalnow,which makesmefeelalotlesswasteful.ButusingpurelyBackup1.0technologies,allIcandois grabentireVMharddiskfiles.Thatswonderfulfordisasterrecovery,becausetheentireVM isexactlywhatIwantinthatscenario.Butrecoveringanindividualfileorotherpieceof datagetsreally,reallytedious.YoualsohavetotakeeachVMofflineinordertofullyback upeachvirtualdisk,whichisahugeinterruptionandtiesyoutoamaintenancewindow ifyourVMscant,forbusinessreasons,beofflineeverynight,thenyouwontbegrabbinga backupeverynight.
117
Don Jones
HyperVVSSWriter HyperVdoessupportMicrosoftsVolumeShadowCopyService,which slightlyexpandsyouroptionsforoutsidetheVMbackups.Essentially,VSS allowsHyperVtonativelymakesnapshotsofVMdiskimageswhilethe VMsarerunning.Thisdoesexactlywhatyoumightsuspect:Itmakesa completecopyofthevirtualdiskfiles,andyoucantheneasilybackupthat copywhiletheVMcontinuesrunning.Youneedtoplanfortheextradisk spaceontheHyperVservertotakeadvantageofthis,ofcourse.HyperV evenmakesasnapshotofthevirtualserversmemory,sowhatyoure backingupisliterallyasnapshotoftheVM,includingitscurrentoperating state. Afterthefirstsnapshot,subsequentonesarejustdeltas,meaningtheyonly includethebitsthatchangedsincethefirstsnapshot.Thathelpsconserve diskspace,andVSScanexposetheresultingsnapshotbycombiningthebase snapshotwiththemostrecentdifferences. Thistechnologyisntquiteasmagicalasitmaysound.Virtualization productshavehadtheabilitytotakesnapshotslikethisalmostforever; VMwareWorkstationwasprobablythefirst,anditwassuperhelpfulwhen youwereusingyourVMstodoproductdemonstrations.Youdget everythingsetupthewayyouwanted,andtakeasnapshot.Aftereachdemo, youdrollbacktothesnapshot,leavingyourselfcompletelyreadyforthe nextdemo.BybuildingthecapabilityintoHyperV,Microsofthassimply leveragedthisyearsoldtechniqueforbackuppurposes. Butsnapshotsarestillveryspecificpointsintime.Youcantrelyonthe snapshotsbeingkeptentirelyontheHyperVserver;ifthewholeservergoes kaput,youlosethosesnapshots,too.Snapshotsareonlybackupswhen theyrecopiedoffserver,typicallytotape.ManybackupsolutionsareVSS enabled,meaningtheycanseethesesnapshotsandgrabthemduringa backupoperation.Soyourestillatriskforanydatathathasntbeenmoved offtotape;thebigadvantageisthatyoucancontinuouslymoverecent snapshotsofftotapebecauseyourecopyinginactivefiles. Ofcourse,IvenevermetanyonewhobacksuptheirHyperVserversmore thanoncepernight,andmostdoitonceaweek.Sothatsalotofdataatrisk. Soitseemslikethesetwotechniquesbothhavetheirupsides,buttheybothhave downsides,too.Doeseitherofthemoffergreatrestorescenariosthatwouldmakeupfor thedownsides?
118
Don Jones
RestoreScenarios
Again,usingpurelyBackup1.0techniques,boththeinsideandoutsidetechniques actuallyofferprettypooroptionsforrestoringdata.Usingtheinsidetechnique,youstill havetohuntdowntherightbackuptape,spintotherightfile,andrestoreit.Intheeventof anExchange,SQLServer,orSharePointbackup,youhavetorestoreanentiredatabaseto analternativelocation,openitusingthenativeserverapplication,findthebityouwanted torestore,exportittoastandalonefile,andsoon.Tediousandtimeconsuming.Although Backup1.0iseasytopulloffinsideaVM,itstilldoesapoorjobofmeetingactualbusiness needs.Youstillcantrestoretopointsintimeotherthanyourbackup,meaningyoustand toloseagreatdealofdata.Whensomethingdoesgowrong,youllbespendingalotoftime onrecoveryprettymuchtheantithesisofouridealsituation: Backupsshouldpreventusfromlosinganydataorlosinganywork,andensurethat wealwayshaveaccesstoourdatawithaslittledowntimeaspossible. Usingtheoutsidetechniqueseemslikeitwouldbeabetterdeal.Yourespendingless overallprocessingpowerbecausethebackupsolutionisrunningontheHyperVhost. Yourealsograbbingtheentirediskimageinyourbackup,withouttheneedforspecial Exchangeaware,SQLServeraware,SharePointaware,orwhateverawareagents.Allof yourdataisprotected.Butyourestilldealingwithbackupwindowstypically,evenif youreusingHyperVsVSSintegration,yourestillonlygrabbingonebackupanightorper week,whichmeansyouhavealotofdataatrisk.Also,restoringdatameansrestoringthe entireVM.Althoughitseasytorestorethattoanalternativelocationyoudohaveaspare HyperVserversittingaroundforthatpurpose,right?youhavetowaitfortherestoreto run,startuptherestoredVM,getwhateveritisyouneeded,andsoon.Doingallthatto recover,say,a5kilobyteemailthatsomeoneaccidentallydeletedisahorriblewasteof time,akintorebuildinganentirehoteljusttofigureoutwhichbitofcarpetingcaughton fireandburnedtheplacedown. MountableImages TheresalittlerayofsunshinewhenitcomestoHyperVthatIllletyouin on.HyperVVMsstoretheirdiskimagesas.VHDfiles.Microsofthasopened the.VHDfileformatspecification,andtheyvecreatedanumberofutilities thatletyoumountandbrowseoffline.VHDfiles. Soheresthescenario:Youregrabbingpointintimebackupsofthe.VHD filesbyrunningabackupsolutionontheHyperVhost.Perhapsyoureusing VSStobackuplivesnapshotsgreat.Whenrestoretimecomes,youdont necessarilyhavetorestoretheentireVM,startit,andgointoittorecover whateveryouwanted.Instead,youcanjustgetthelatest.VHDfileoffoftape, thenmountthat.VHDasareadonlydiskdriveonanotherWindows computerWindows7canactuallydothisnatively.
119
Don Jones
Thisstillisntaperfectsituation:Yourerestoringfromapointintime,andif yourbackupsarefarapart,youstillhavealotofdatapotentiallyatrisk.You stillhavetospinthe.VHDfileofftapeandfindaplaceforittolive.Butyou canmountitwithouthavingtoactuallyruntheoriginalVM,whichcansavea lotoftime.Forexample,iftheoriginalVMhadthree.VHDfiles,youonlyneed torestoretheoneyouwantsomethingoffofratherthanrestoringallthree sothatyoucanstarttheVMnormally. SovirtualizationbackuptechniquesintheBackup1.0worldarentanybetterthanthenon virtualizationbackupsinthatsameworld.MaybeBackup1.0canofferabetterdisaster recoveryscenario?
DisasterRecovery
Infact,itcan.Hooray,onepointforoldschoolbackuptechniques!Intheoutsidebackup scenario,whereyourebackingupentireVMdiskimages,youvegotgreatdisaster recoveryoptions.Becauseyouvebackeduptheentireserver,possiblyevenincludingits memorystateataspecificpointintime,youcanrecovertheentireserverprettyeasily. WhatsevenbetteristhefactthatVMsactuallyhaveverylittleideawhatphysical hardwaretheyrerunningon.Remember,allaVMsees,forthemostpart,isanabstract, virtualizedhardwareenvironment.SoifyourentireHyperVserverdies,youcanjust restoreallyourVMstoanotherHyperVserverorevenspreadthemoutacrossmultiple otherHyperVservers,evenatdifferentlocations.Oncerestored,theVMswillstartwith littlecomplaintbecauseasfarastheOSsinsidetheVMsareconcerned,nothinghas changed. ThisabilitytorelocateaVMtoanyavailableHyperVhostistheexactthingthatmakes livemigrationpossible.Inalivemigration,arunningVMismovedtoadifferentHyperV host,oftensothattheoriginalhostcanbetakenofflineformaintenance.Itworks. Well,itworkstoapoint.Yourestilllimitedtowidelyseparatedpointsintimeforyour disasterrecoverybecauseyourbackupsarepointsintime;theyrenotcontinuous.Itsgreat thatyoucanrestoreaVM,evenincludingitsmemorystate,butifwhatyourerestoringis5 daysold?Youvestillgotsomebadnewstodelivertotherestofthecompanybecauseodds aretheresalotoflostdata. Theresmorebadnews,too.WhenIwriteaboutdisasterrecovery,Imreallywritingabouta completeserverfailuremeaningyouvelosttheHyperVhost,too.Ifyourerelyingon backupsofVSSsnapshots,thendisasterrecoverywillstillhavetostartwithreinstalling WindowsandHyperV.AlthoughnewerversionsofWindowsServerinstallfasterthan ever,itsstillnotinstantaneous.ItdbefastertorestorethoseVMimagestodifferent, alreadyupandrunningHyperVservers,butthatassumesyouhavesomeofthosewith availableprocessingpower,diskspace,memorytoallocate,andsoon.Ifyoudont,then yourdisasterrecoverywillstillbealittleslow.
120
Don Jones
BackupManagement
ManagingbackupsisthesameoldBackup1.0story.Youmustshuffletapes,makesureyou knowwhichonesarethelatest,andifyouredoingfull+incremental+differentialbackups, youstillhavetomakesureeverythingstaysintherightorderforasuccessfulrestoreor disasterrecovery.
LetsonceagainforgetBackup1.0.LetsevenforgetBackup2.0.Iwanttowriteaboutwhat Iwishvirtualizationbackupscouldlooklike.Itmightnotbepossible,buttheonlywaywell eventuallymeetallourbusinessneedsistopushthetechnologyalittle.Mywishlistis simple: Backupsshouldpreventusfromlosinganydataorlosinganywork,andensurethat wealwayshaveaccesstoourdatawithaslittledowntimeaspossible. Now,morespecifically,hereswhatIwanttoenablemywishlist: ContinuousbackupsofVMswithaslittleaggregateoverheadaspossibleThat meansIdprefernottohavetorunbackupsolutionsinsidetheVMsbecausethatis essentiallyduplicatedeffortandwastedcomputingpower. Full,asfastaspossiblebaremetalrecoveryofanentirevirtualizationserver, includingthebaseOS,hypervisor,andalltheVMs TheabilitytograbsinglefilesandfoldersfromaVMbackup,preferablywithout havingtorestoretheentirething,starttheVM,andsoon TheabilitytorollbackasingleVM,ortheentirevirtualizationhostserver,toa particularpointintimeImhavingtroublethinkingofspecificscenarioswhenI mightwanttorollbacktheentirehost,butImnotlettingmylackofimaginationget intheway:Iwantthecapability. Idontwantandthisisabigpointformetohavetowaitfortapedrivestocopy filesbacktodiskHonestly,Idontliketape.Sure,youhavetouseitbecauseitsthe easiestwaytogetdataoffsite,butIprefertapestobePlanB.Inotherwords,I wantmyPlanAbackuptobeonnice,fastdiskspacesomewhereinmydata center;ifmywholedatacenterfloodsorwhatever,Imhappytodealwithtape.
121
Don Jones
WhytheTapeHate? Itsjustauserperceptionthing.Ifauserwantsafile,theydonttendto understandwhyittakes6hoursformetogettherighttape,restoreafile, andsoon.If,however,theentiredatacenterfloods,everybodyinthe companytendstobeabitmoreunderstandingaboutwhythingsaretaking solongitsclearlybeyondanyonescontrolatthatpoint.ThatswhyI preferahierarchicalstoragesystem:diskbasedbackupsfordaytodayuse, andtapesasabackuptothat. SowhatdoesvirtualizationBackup2.0looklike?Isittechnicallypossibletomakethiswish listareality?
Ithinkitis.ThinkingabouttheBackup2.0techniquesIveoutlinedinpreviouschapters, hereshowIdimagineitworking: Youdstartwiththeoutsidetechniquerunningthebackupsolutionrightonthe HyperVhost,andrunningnothingspecialinsidetheguestVMs.Thistechnique involvesthelowestprocessingoverheadprobablylessthan3to4%ofthehosts totalcomputingpower. Youdneedtorelyondeduplicationandcompressionbecausetheymakedisk basedbackupsmorepractical(diskspaceisstillmoreexpensivethantapespace). Bytakinguplessdiskspaceonabackupserver,youreenablingmorebackupstobe keptforlonger. Youdsimplymonitorforchangestothediskblocksonthehost,capturethose changes,andtransmitthemofftothebackupserver.Thisisgoingtocaptureevery changetoboththehostOSandtotheVMdiskimages,innearrealtime.Figure7.4 illustratesthis.Essentially,everythingthathappensonaHyperVhostcomesdown toblocksstoredonthehostsphysicaldiskstorage.Byinterceptingthosechangesat theWindowsNTFSfilesystemlevelbymeansofafilesystemfilteryoucan capturethechangeddiskblocks(whichareonlyafewkilobyteseachinsize)and transmitthemtoabackupserverforlongtermstorage.
122
Don Jones
Figure7.4:Diskblockcaptureinavirtualizationhost.
123
Don Jones
Thebackupserverknowswhichdiskblocksgowithwhichfile.Therefore,itsnotjust keepingagiantrepositoryofdiskblocks;itknowswhichblockscomprisethehostOS, whichblocksmakeupagivenVMs.VHDfiles,andsoon.ThereareafewkeyfactsthatI needtohighlightaboutthis: Wehaveacopyofeverydiskblockthathaseverchangedonthefilesystem. Weknowwhattimeeachchangewasmade. Weknowwhatfileeachchangewentto. Thepreviousthreefactsmeanwecanreconstructanyfileatanypointintime, simplybygrabbingthediskblocksassociatedwith(a)thatfileand(b)thatpointin time. Windowsalreadyincludesthetechnologyneededtomounta.VHDfileandbrowseit asanactivedisk.
Thesefivefactsleadtovastlyimprovedrestorescenarios.
WecouldusearestorescenarioliketheoneillustratedinFigure7.5.Here,weveusedthe backupsolutiontoassemblethediskblocksforaspecificVHDfileataspecificpointin time.WeverebuiltthatVHDfileontoafileserveronthenetwork,thenmountedthatVHD filetoaworkstation.ThatprovidesaccesstothefilesinsidetheVHD,whichcanbecopied toarealdiskdrive,thenmovedtowhereveritistheyneedtogo.Thesametechnique couldbeusedtorecoveraVHDandmountitasadriveonaSQLServer,andthenSQL ServercouldattachadatabasefromthatVHDaneatwaytoaccessanearlierversionofa database.
124
Don Jones
Figure7.5:Restorescenario. ThisisagreatrestoretechniquebutBackup2.0candobetter.Remember,computersare supposedtosaveuswork,notcreateintermediatestepsforus.Whynotsimplydesigna backupserverthatcoulddirectlyexposeaspecificsetofdiskblocksasafilesystem?In otherwords,wedontneedtoextractthediskblockswewantandreconstructafile;we shouldsimplybeabletodynamicallyaccessthedesireddiskblocksastheysitinthe backuprepository,exposingtheresultingfilethroughstandardfilesystemmechanisms. And,whilewereatit,whynotsimplybuildinthetechnologyneededtomounttheVHD?In thisfashion,theVHDwouldappearasashareddriveonthebackupserveritself.Figure 7.6showswhatImtalkingabouthere.
125
Don Jones
Figure7.6:Faster,easierstepstorestoringfiles. Thestrikingthingaboutthisisthatitdoesntinvolveanyradicalnewtechnologies. EverythingIvesuggestedherecanalreadybedoneinindependentways;werejust combiningthemtocreateabetterbackupandrestorecapabilityforourselves.Bysitting downandthinkingaboutwhatweactuallywantfrombackups,asopposedtowhatvendors havetraditionallygivenus,wevebeenabletodrawtogetherexistingtechnologiestocreate somethinginfinitelymorevaluable. ItstheBlockthatChangesEverything IfyoulookatwhatImsuggestinginFigures7.5and7.6,itsreallynotthat differentfromwhatHyperVandVSScanalreadydo.VSSgrabssnapshotsof VMs,whichyoucanthenbackuptodiskortape.Youcanrestorethose, mountaVHD,andrecoverfilesexactlyasIvesuggested.Suchabackup solutionwouldrunontheHyperVhost,notwithintheguestVMs,andwould useaminimumofcomputingpower(albeitalotofdiskspacebecauseVSS doesntsupportdeduplicationorcompression). VSSonlydownsideisitsscope:ItfocusesonsnapshotsoftheentireVM.You cantjustconstantlytakesnapshots;therestoomuchoverheadinvolvedand waytoomuchdiskspace.
126
Don Jones
Mysuggestionisjusttogetmoregranularandgrabdiskblocksnapshots insteadofentireVMsnapshots.Thatway,yourecontinuallygettingalow overheadstreamofdata,andyourebackingupthelatestchangesona continualbasis.Thatbasicallyminorchangeinscopeindividualdiskblocks versusentireVMsmakesthenearrealtimebackuppossible,andthats whatletsusoperatewithmuchlessdataatrisk.ThatsBackup2.0: continuousbackups,notpointintimebackups.
Disasterrecoveryrecoveringanentire,baremetalserverintheeventofadisasteris straightforward.Simplygrabthemostrecentdiskblocksfortheentireserveranddump thembackontotheserversdisk.IimagineyoudhavesomekindofbootableCDorDVD withaminimalOSonit,capableofmountingtheserversphysicaldisksandreceiving streamedindiskblocksfromthebackupsolutionasshowninFigure7.7.Youdgetthe entireserverhostOS,registry,alltheVMs,andeverything,towhateverpointintimeyou needed.
Figure7.7:Abaremetalrecovery. Theneatbitisthatyoucouldrecovertootherhardware,too,providedthenewhardware wasofthesameprocessorarchitecture,hadthesamebasicconfiguration,andsoon.That makesrealdisasterrecoverylikeoffsiterecoveryarealpossibility.
127
Don Jones
ThisisthesamebasicdisasterrecoverymodelIveproposedinpreviouschapters.Itoccurs tome,however,thatbyaddingalittlereplicationaction,youcouldreallyenablesome interestingdisasterrecoveryscenarios.Forexample,byreplicatingthecontentsofthe backuprepositorytoanotherphysicalserverinadifferentphysicallocation,youdbe assuredofdisasterrecoveryevenintheeventofatotalfacilityloss.Youcouldpotentially replicatethosediskblockstoaninthecloudserviceprovider,whocouldpotentiallygive youtheabilitytorecoverentireserverstoVMsthatrunattheserviceprovidersfacility creatingacloudbaseddisasterrecoveryfacility.Youdneedtodosomemathonthe bandwidthrequired,butrememberthatdatadeduplicationandcompressionwouldhelp reducethenecessarybandwidthsomewhat,andyoucouldhaveacontinuousreplication streamatallhoursofthedayandnight.Itbearssomeadditionalthoughttoseeifthatsa practicaladditiontoBackup2.0ssuiteofcapabilities.
EasierManagement
ManagementoverheadkindofgoesawaywithBackup2.0,oratleastgetsalotsimpler.Im notsuggestingthatyoullneverneedtapesagain,becauseyouabsolutelyshouldbebacking upthediskblockrepositorytotapeforoffsitestorage(unlessyoucangetthereplication thingfiguredout,whichwouldbeverycool).Figure7.8illustrateshowtapebackupswould fitintothescheme;thethingtorememberisthatthoseareredundantbackups.Iimagine makinganightlybackupofthedaysnewestdiskblocks;theresnobackupwindow becauseyourenotbackinguplivedataortakingproductionserversoffline,sohowever longittakestowritethedatatotapeisfine.
Figure7.8:Makingaredundantbackuptotape.
128
Don Jones
VirtualizationSpecificConcerns
Anytimeyoustarttalkingaboutvirtualization,therearespecificconcernsthatcometo mind.Virtualizationisjustabitdifferentthannormalcomputing,andtherearefactors involvedthatwewanttomakesurewevefullyconsidered.
Performance
Performanceisabigoneofthosefactors.AsIpointedoutearlier,simplyrunningabackup solutioninsideeachVMseemslikeawasteofcomputingpower.Allthethingsyouneedto backupareononemachine,butyourespendingmultipletimestheneededcomputing powersimplybecauseeachVMisresponsibleforthedataunderitscontrol. Ithinktheoutsidescenarioprovidestheanswertothis.ProvidedyoucanbackuptheVM imagesfromthehostOSlevelandyoucanwhilestillhavinggranularaccesstothestuff insidetheVMimagesandIthinkyoucandothat,toothenthisisthewaytogo.Youre gettinganaggregatecomputingoverheadinthelowsingledigits,whichiscompletely acceptable.
GranularRecovery
Hereitisagain:granularrecovery.IdislikealotoftraditionalBackup1.0,thatis techniquessimplybecausetheyreprimarilygearedtowardfullfailuredisasterrecovery models,whereasmostofwhatwedealwithonadailybasisisrecoveringafileortwo. IthinkIveshownhowBackup2.0techniquesgrabbingdiskblocksinnearrealtime canprovideanylevelofgranularityyouneed,fromafullbaremetaldisasterrecoveryto recoveringasinglefile.GranularrecoveryshouldandunderBackup2.0,doesrequire minimaleffort;theabilityofaBackup2.0stylesolutiontodynamicallyexposeabackedup volumebydynamicallyassemblingtherightdiskblocksondemandwell,thatcovers minimaleffort.
GuestsandHosts
ItsnotenoughtojustbackuptheVMVHDfilesandrelatedconfigurationfiles.Idorealize thatthoseVHDfilesarewhatcontainthedatayouactuallycareabout;withouta virtualizationhosttorunthoseVHDfiles,however,theyrenotnearlyasusefultoyour business.Backupandrecoveryisallaboutgettingbackonlineasfastaspossible;simply backingupVHDfilesdoesnotaccomplishthatifittakesacouplehourstorebuildahost forthoseVHDfilestoliveon. Iveworkedwithanynumberofconsultingclientswhodidntquitegetthatpointuntil theydidhaveacompleteserverfailure.Then,alltheVHDbackupsintheworldwere uselessuntiltheydbuiltanewhost. IveshownhowBackup2.0caneasilyscaletoafulldisasterrecoverymodelbystreaming diskblocksbacktowhateverphysicalserveryouwant.ThisrebuildsnotonlytheVMVHDs butalsotheentirehostOSandconfiguration,meaningyoucansimplyrestarttheserver andpickupbusinesswhereyouleftoffwithverylittlemanualeffort.
129
Don Jones
ComingUpNext
Atthispoint,wevecoveredWindowsServer,SQLServer,SharePointServer,Exchange Server,andevenvirtualizedservers.WhatelseisleftforBackup2.0toconquer?Plenty. Todaysbusinesseshavelotsofotherconcernsandneedswithregardtobackup,andinthe nextchapter,welladdresssomeofthem.Forexample,restoreisaverydifferenttask thandisasterrecovery,soIneedtolookathowBackup2.0mighthandlethelatter.And whatabouthighavailability?Youcertainlycantaffordforyourbackupsolutiontobedown whenyouneedit!Thereareotherissues,too:security,compliance,dataretention,mobile infrastructure,andmore.AndwhatifBackup1.0isntcompletelydead?Howcanyou integratethe1.0and2.0techniquesintoaconsolidatedbackupplanandwhyinthe worldwouldyouwanttodoso?Thatsallinthenextchapter.
130
Don Jones
Chapter8:OtherConcernsandCapabilities
Theresmoretoabackupstrategythanjustgrabbingtherightfilesandmakingsureyou canrestoretheminapinchalthoughthatsobviouslyabigpartofit.Asolidbackup strategyalsoconcernsitselfwithdisasterrecoveryinavarietyofscenarios.Youneedto makesureyourbackupsystemitselfhassomeredundancynothingsworsethanbeing withoutabackupsystem!Becausebackupsinherentlyinvolvedataretention,inthisday andage,youalsohavetoconcernyourselfwiththesafetyandsecurityofthatdataaswell asanylegalconcernsaboutitsretention.Thatswhatthischapterisallabout:Dealingwith theextrasthatsurroundabackupstrategy.IlllookathowtraditionalBackup1.0 techniquesaddressedtheseextras,andsuggestwaysinwhichwemightrethinkthemfora Backup2.0world.
DisasterRecoveryandBareMetalRecovery
Ivealreadywrittenquiteabitaboutdisasterrecovery,orbaremetalrecovery,whichis whatyoudowhenanentireserverdiesandyouneedtorestoreit.Inthebad,bad,badold days,disasterrecoveryalwaysstartedwithreinstallingtheserversoperatingsystem(OS) fromscratch,theninstallingsomekindofbackupsolution,thenrestoringeverythingelse fromtapeatimeconsumingprocessbecausespinningdataoffoftapeisntexactlythe fastestactivityintheworld. EventheBackup1.0mentalitygotsickofthatprocess,though.Today,mostthirdparty backupsolutionsprovidesomekindofrestoreCD,whichcanbeusedtobootafailed server.ThestrippeddownOSontheCD,oftenbasedonDOS,WinPE,oraproprietaryOS,is smartenoughtofindthebackupserverandreceivedatabeingstreamedacrossthe Internet;dependingonthesolution,itmightalsobesmartenoughtoreaddatadirectly fromanattachedtapedrive.Figure8.1showsanexampleofoneoftheserecoverydisksin action.
131
Don Jones
Figure8.1:Recoveringaserverbyusingarecoverydisk. Note SomevendorsmightmakeabootableUSBflashdriveorsomeotherformof removablemediabootable.Thatsfine,too,andactuallycanbefasterthana disk,providedyourserverscanbootfromtheselectedtypeofmedia. Noteverybackupsolutionreliesonadisk,thoughafterall,somefolksdoconfiguretheir serverswithoutopticaldrives.Otherbackupsolutionscaninsteadinstalltheirbaremetal recoverysoftwaretoaseparatepartitionoftheserversbootdisk,makingtherecovery softwareavailableasabootoption.Idontfavorthisapproachbecauseifyouloseawhole server,itsentirelypossiblethatyoulostthedisks,too,inwhichcaseyourrecovery softwareisgoneaswell.StickwithsolutionsthatprovideabootableCDorDVD,andsimply makesureyourservershaveanopticaldriveinstalled.Eventodayslowprofile,2U rackmountserverscanalmostalwaysbeconfiguredwithaslimlineopticaldriveusually thesameslimlinedrivesthatthemanufacturerusesintheirlaptopcomputers.
132
Don Jones
AThirdOption:PXE AthirdoptionistohaveserverswhoseBIOSandnetworkcardssupport networkboot,usingIntelsPXEprotocol.Inthisscenario,yourbackupserver softwarewouldneedtosupportnetworkboot,meaningthattheserver wouldbeabletofinditonthenetwork,requestabootOSimage,and downloadthatimagefromtheserver.Thatimageisusuallythesamething thatwouldbeonarecoverydisk. MicrosoftmakesgreatuseofPXEfortheirautomateddeploymenttools;even theolderRemoteInstallationServices(RIS)supportednetworkboot scenarios.ItslesscommontoseePXEusedinconjunctionwithabackupand restoresolution,though,simplybecauseyourehopefullynotusingthe featurethatoften,sousingarecoverydiskissimpler,requireslesssetupand infrastructure,andiseasiertouseunderawidervarietyofsituations. Backup2.0isntmuchdifferentintermsofprocess:Youllusuallyuseabootdiskofsome kind,andthatdiskissmartenoughtolookforthebackupserveronthenetwork.Youmay beabletobrowseforaspecificserverimagetorestore,thentherestorationbegins.Other times,youmightbootusingthedisk,thengotothebackupserversconsoletoselectan imagetopushtotheserveryourerecovering. ThedifferencewithBackup2.0iswhatgetsrestoredbacktotheserver. BecauseaBackup2.0solutionwasstreamingdiskblockchangesinnearrealtimeupuntil theminutetheserverfailed,youregoingtolosealotlessdatathaninaBackup1.0style scenariowhereyourmostrecentfullserverbackupmayhavebeenfromaweekagoor longer.SoBackup2.0getsyoubackonlinewithlessdatalossiftheresevenanylossat all. Also,rememberthatBackup2.0reliesprimarilyonadiskbasedbackupstore.Thatoffersa coupleofadvantages:First,youwontbesittingaroundwaitingfordatatostreamofftape, whichevenwithtodaysimprovedmagnetictapeanddrives,isstilltimeconsuming.A Backup2.0solutioncanstartrestoringyourserverimmediately. Finally,Backup2.0doesntmessaroundwithfull,incremental,anddifferentialbackups.Or technically,Iguess,itdoes:Itgrabsafullbackupofyourserverinthebeginning,then continuallygrabschangeddiskblocks,whichistechnicallylikeanincrementalbackup.But youdonthavetokeeptrackofanyofthat,orfindtherighttapes,orrememberthecorrect ordertorestorethosetapes.TheBackup2.0solutionsimplyasksyouwhatpointintime youwanttorestoretheserverto,anditgetsthecorrectdiskblocksautomatically.
133
Don Jones
RecoveryTimes
Now,youmightthinkthatstreaminganentireserversworthofdiskblocksacrossthe networkwouldbeprettytimeconsuminginandofitself.Imean,justtryingtocopya2GB fileacrossthenetworkcantaketime,andserversmighthavehundredsofgigabytesworth ofdiskblocks.ThatswhereyouwanttodigintosomeofthedetailsofaBackup2.0style solutionbeforeactuallybuyingone. Laterinthisguide,Illbediscussingcompressionanddeduplication,twotechniquesthat helpabackupsolutionstoreserverdatausinglessdiskdatathanthatdataoriginallytook ontheserveritcamefrom.Lessspaceondiskalsomeanslesstimeonthenetwork,so exactlyhowabackupsolutionimplementsitsrecoverycanmatteralot.Figure8.2shows whatyoudontwant:Asolutionthatexpandscompressedanddeduplicateddiskblockson thebackupserver,streamingeachdiskblocktotherecoveringserverexactlyastheyllbe writtentodisk.Inotherwords,aserverwith200GBofstoragewillhave200GBofdata transmittedacrossthenetwork;evenwithadedicated1000Mbpsnetworkconnection, youreprobablylookingatanhourorso.
Figure8.2:Uncompressingandreduplicatingdataatthesource. Figure8.3showsaslightlysmartersolution:Leaveeverythingcompressedandde duplicateduntilitgetstotherecoveryserver.There,thebackupsolutionsrecoverydisk softwarecandecompressandreduplicatethedata.Lessdataistransmittedacrossthe networkthisway.
134
Don Jones
Figure8.3:Uncompressingandreduplicatingdataatthedestination. Thesesortsoffinedetailscanhelpspeedtherecoveryprocesstremendously.Asyoure evaluatingsolutions,askvendorsforbenchmarksforbaremetalrecoverytimes.They mightbeabletotellyou,forexample,howmanyminutesittakestorecoveryxgigabytesof dataacrossanetworkconnectionofymegabitspersecond.Iftheycant,itscertainlya benchmarkyoucanfigureoutonyourowninalabusingtrialsoftware.
RecoveryLocations
Akeycapabilityforanybaremetalrecoverysolutionistheabilitytorestoreaservers imagetoadifferentservereitheradifferentphysicalboxortoavirtualmachine.Both capabilitiesphysicalserverandvirtualmachineareneeded.Insomecases,aserver mightfailentirely,leavingyouwithnochoicebuttogetanewphysicalmachinetoreplace it.Inothercases,youmightonlyneedafewdaystorepairaphysicalserver,meaning restoringtoavirtualmachinewillprovidealessexpensiveinterimoption.Beingableto restoretoavirtualmachinealsoprovidesvastlymoreflexibilityfordisastersthataffect yourentiredatacenter:Ratherthanneedinganoffsiterecoverylocationthathasdozens ofserversreadytogo,youcanjustdumpallyourserversintovirtualmachines. Expectsomelimitationsaroundthesecapabilities.Forexample,youobviouslyshouldnt expecttorestorea64bitOSontoolder,32bithardware.Serverhardwarewithvastly differentconfigurationsmaypresentproblems,too,althoughtheWindowsOShasbecome moreadeptatdealingwiththosetypesofchanges.
135
Don Jones
Tip HaveyourWindowsserialnumbersorlicensekeyshandy,andifyoureusing internalVolumeActivationlicenses,makesurethatanactivationserveris partofyourrecoveryplan.WhenWindowsdetectsmassivehardware changes,itmaydemandimmediatereactivation. Thefeaturetolookforasyoureevaluatingsolutionsisdissimilarhardware,whichoften meansthevendorhasprovidedtechnologiesformakingtherecoveryprocessonto differenthardwareorontoavirtualmachineabitsmoother.
WhyWaitforaDisaster?
HeresanideaworthyofourBackup2.0letsrethinkeverythingmotto:Whywaitfora disastertostartyourdisasterrecovery?Whynotjusthaveaspareserverreadytorunfor everyoneofyourservers? Hearmeout.Intherecent,badolddays,yousimplycouldntaffordtokeepaspareserver intheclosetforeveryproductionserveryouowned.Notpractical.However,withthe adventofhighperformance,reliablevirtualmachines,youcankeepaspare.Hereswhata reallyslickBackup2.0solutioncouldoffer:Onceanight,say,thesolutioncouldconstructa readytorunvirtualmachinefromyourcurrentbackups.Youdjusthavesomebighard disksomewhere,filledwithvirtualmachinefilesthatwereconstantlybeingrecreatedor updated.Whendisasterstruck,youwouldntneedtorecoveranythingyoudjustbootone ofthevirtualmachines.Weretalkingarecoverytimeofafewminutes! Dependingonhowyouvebuiltoutyourvirtualizationinfrastructure,thevirtualservers mightnotperformaswellasthephysicalones.Butslowerisfarbetterthancompletely gone,right?Andwithtodayslivemigrationcapabilitiesofferedinmostmajor hypervisors,providedyouveboughttherightmanagementtoolstoenablethefeature youcanalwaysshiftvirtualmachinestodifferentvirtualizationhosts,balancingthe workloadtogetthebestperformanceyoucanfromthemostcriticalresources.Andthose virtualserversmightwelljustbeaninterimstrategywhileyourepairaphysicalmachine orgetanewphysicalmachineinplace.
HighAvailability,InstantRecovery,andReplication
Aserverdies.Auserismissingafile.Someonesemailiscorrupted.Youconfidentlyturnto yourBackup2.0solution,havingreadthisbookandfullyembracedtheBackup2.0 lifestyle. Andyourbackupserverisdead. Whoops.
136
Don Jones
Highavailabilityisanappropriatefeatureforanymissioncriticalfunctionwithinyour business,andyourbackupsolutionshouldcertainlyqualifyasmissioncritical.Therearea fewdifferentwaysinwhichabackupsolutioncanbemademoreredundantandmore resistanttosinglepointsoffailure.
Hardware
Obviously,runningyourbackupsolutiononredundanthardwarecanhelp.Redundant powersupplies,redundantdrivecontrollers,redundantnetworkcards,RAIDarraysfor disks,redundantfans,andsoonthosefeaturesareallwidelyavailablefrommostserver manufacturers,andareusuallyworththeextraprice. Dontforgetabouttheinfrastructurethatyourbackupsolutionrelieson,though.Having redundantnetworkcardsisnice,butitslessniceiftheybothplugintothesameswitch, andthatswitchfails.Someredundancyinyourinfrastructurecanhelp,too.Ofcourse,you havetodecidewherethetradeoffisforyourcompany:Redundancycostsmoney,andat somepointyoullhavetodecidehowmuchredundancyandexpenseispractical.
Clustering
Redundantsoftwaresimplymeansthatyourbackupsolutionisrunningonmorethanone server,eitherinsomekindofclusterconfigurationorusingreplication.Acluster,forme, isntthemostefficientuseofresources.Hereswhy:Lookattheclusterarrangementin Figure8.4.
Figure8.4:Typicaltwonodecluster. Inthistypeofcluster,youhavetwocompletephysicalservers(theycouldbevirtual,too,I suppose),connectedtosomekindofsharedstoragelikeaStorageAreaNetwork(SAN). Yourbackupsoftwarewouldrunoneachserver,althoughonlyoneserveratatimewould beactive.Intheeventofafailure,theotherserverwouldtakeover,andwouldhaveaccess toallthesamestorage,soitcouldpickupwheretheotheroneleftoff. 137
Don Jones
Forabackupsolution,thisiskindofinefficientandyoustillhaveyourstoragenetworkas asinglepointoffailure(althoughthoseareoftenredundantinandofthemselves,using RAIDarraysandthelike).Forabackupsolution,replicationoftenoffersabetterformof highavailabilityandbringswithitadditionalflexibilityforsolvingotherbusiness problems.
ReplicationforRedundancy
Withreplication,yourbackupsoftwareiscontinuouslystreamingcopiesofitsbackedup datatoanothercopyofthesamesoftware.Thatmeansyourbackupdatalivesintwo places,soifoneofthoseplacesgetsattackedbydinosaursorsomething,theotherplaceis stillabletoproviderestoreandrecoveryservices.Thiscanbeagreatwaytogetdataoff siteentirely,especiallyforsmallerbranchofficesthatmightnotbeabletomessaround withtapebackups.Figure8.5showswhatitmightlooklike.
Figure8.5:Replicatingbackupdata.
138
Don Jones
Inthisexample,eachbranchofficehasitsownbackupserver,whichbacksupitslocal servers.Thebackupdataisreplicatedtothemainoffice,whosebackupserveralsobacks uptheproductionserversinthatoffice.Themainofficewouldlikelyhaveaccesstotape backupdrivesandanITstaff,sothemainofficesdataisredundantthroughtapebackups. ThebranchofficeswouldntneedtapebackupsorITstaffersbecausetheirdataisbeing maderedundantthroughreplication.Inaworstcasescenario,abranchofficeservercould berestoredfromthemainofficesbackupserver.Mightnotbeideal,andwouldntlikelybe asfast,butslowisbetterthanoutofbusiness. ThedownsideofthisapproachisthatyoumightwellbereplicatingdataacrossWAN links,mindyouthatyoudontcarethatmuchabout.Inotherwords,maybeabranch officehastwoservers,andyourebackingupthemboth,butyoureonlyreallysuper worriedaboutoneofthem.SoIllsuggestamodification:Yourbackupsolutionshouldlet youspecifywhichserversbackupdatawillbereplicatedupstream.
ReplicationforOffsiteRecovery
Theresanotherbig,big,bigadvantagetothiskindofreplication.TakealookatFigure8.6.
Figure8.6:Replication+virtualization.
139
Don Jones
Server4,oneoftheserversinthefirstbranchoffice,hasdied.Restinpeace.Fortunately, Server4wasbeingbackedup,anditsbackupdatawasbeingreplicatedtothemainoffice. And,bystrangecoincidence,themainofficehasavirtualizationhost.Eachnight,themain officebackupservercreatesafreshsetofvirtualmachinesonthatvirtualizationhost.So whenServer4dies,someoneatthemainofficepushesabutton,andVirtualServer4starts up.Totaldowntime?Howeverlongittakestomakeaphonecallandpushabutton. VirtualServer4isstillbeingbackedup,sowhentherealServer4isfixed,someonepushes anotherbuttonandthecurrentServer4imageisrestoredtotheServer4hardware.Virtual Server4isshutdown,andServer4resumesitsexistence.Thatfailbackcanbedone duringamaintenancewindow,ofcourse,toavoidfurtherinconveniencingthebranch officeusers. Inthisinstance,havingthebackupdatareplicatedwasntcoveringforafailureinthe backupsystem,itwasofferingafasterandmoreconvenientrecoveryscenarioforafailed server.Thatswhatscalledinstantrecovery,anditsaveryusefulcapability. ChangingtheNatureofOffSiteRecovery Icanseethistypeoftechnologyreallychangingthewaycompaniesthink aboutoffsiterecovery.Today,manycompaniesinvestalotofmoneyin disasterrecoveryfacilities.Typically,companiesthatofferoffsiterecovery alsoofferoffsitetapestorage,andtheykeepavariedinventoryofhardware onhand.Ifyourdatacenteristoast,youpackupyourstaffandheadtothe offsitefacility.Yougetyourlatesttapesandsitbackforanicelongday(and night)ofrestores,gettingyourserversupandrunningontheloaner hardware.TypicalBackup1.0mentality,right? Imagineinsteadthatyourepayinganoffsitefacilitysimplytohostacopyof yourbackupsolution,withenoughdiskspacetoreplicateyourbackupdata. Allyouneedthemtoprovide,intheeventofadisaster,isabunchofHyperV servers.Whensomethingbadhappens,youmightnotevenhavetopackup thewholeteamjustgetontoaWebinterface,pushafewbuttons,andyour precreatedvirtualmachines(whichyourbackupsolutionhasdutifullybeen makingfromyourreplicatedbackupdata)moveovertoaHyperVhostand startup.YouestablishVirtualPrivateNetwork(VPN)connectionssothat youremployeescangettotherecoveredservers,manglesomeIPaddresses andDNSentriesperhaps,andyourebackonline.Heck,Iimaginesomeday therewillbeabackupsolutionthateventakescareoflittledetailslikeDNS entries. Offsitedisasterrecoveryfacilitiessuddenlybecomealotlessexpensive, becausealltheyneedissomegoodWANbandwidthandabunchofHyperV serversinadatacenter.Theycangetbywithalotlessvarietyofhardware, andtheycanenablefasteroffsiterecoverywithoutactuallyrequiringmany (ormaybeevenany)ofyourteammemberstoactuallygooffsite.Callit cloudrecovery,perhaps.Itsnothereyet,althoughyoucancertainlybuild yourownprivateversionofit;itllhappensomeday,though.
140
Don Jones
DataRetentionHereCometheLawyers
Backupscontaindata.Kindofthewholepoint,really.Butalotofcompaniesthesedaysare undersomeprettystrictregulationsonhowlongtheymustkeepcertaintypesofdata,and howlongtheymaykeepcertaintypesofdata.Workingthosedataretentionpoliciesinwith yourbackupsolutioncanbeawfullytricky.Inatapebased,Backup1.0world,itusually meanscyclingtapesoffsite,andkeepingthemforhoweverlongyourerequiredtokeep thedata.InaBackup2.0world,youcangetjustabitmoreflexibility. RememberthateachdiskblockbackedupinaBackup2.0stylesolutionistimestamped, andthatthebackupdatalivesprimarilyondisk,nottape.Itsthereforeveryeasytospecify minimumormaximumretentiontimesfordifferenttypesofdatathesolutionsimply makessurethateachdiskblockstaysaroundforthatlong,andremovesanythingthatsno longerneeded. ThatideagetsabitmoreexcitingwhenyourememberthataBackup2.0solutionmightuse diskblocksasitslowestformofgranularity,butthatitalsorelateseachdiskblocktothe realworlddataitrepresentslikeafile,oramessage,oradatabasetable.Figure8.7 showshowaBackup2.0solutionmightreassemblediskblocksintoanExchangemessage store,mountthatstore,andthendisplaytheindividualmailboxesandtheirmessages.By combiningthiscapabilitywitharetentionpolicy,asolutionmightallowyoutospecifythat allmessagesbekeptfor1year,ratherthanworryingaboutwhatthatmeansatadisk blocklevel.
Figure8.7:Managingindividualmessagesfromabackupstore. Obviously,dataretentioncanincreasetheamountofdiskspacethatyourbackupsolution requiresbutifitsabusinessrequirement,itssimplysomethingyouhavetoplanfor.
141
Don Jones
ComplianceandSecurityHereCometheAuditors
Nowthetricky,trickypart:Makingsureyourbackupsarecompliantwithyoursecurity policies,andwithanyexternallyimposedsecurityrequirements.Evenmanymidsize companiestodayfacetoughindustryandlegislativesecurityrequirements: Ifyoureinthemedicalindustry,youreprobablydealingwiththeHealthInsurance PortabilityandAccountabilityAct,commonlyknownasHIPAA. AnyonedealingwithfinancialserviceshastocomplywiththeGrammLeachBliley Act,orGLBA. AnypubliclytradedcompanieshavetomeettherequirementsoftheSarbanes OxleyAct,orSOX. AnycompanywhoacceptscreditcardshastodealwiththePaymentCardIndustry (PCI)DataSecurityStandard(DSS),whichisenforcedbycompanieslikeVisa, MasterCard,andAmericanExpress. FederalcontractorsofanysizeareoftensubjecttoChapter21oftheConsolidated FederalRules(21CFR)andvariousFISMArequirements.
Andthatsjust(mainly)intheUS;manyothercountrieshavesimilarrulesandlaws. Typically,theseallboildowntoafairlycommonsetoftechnicalrequirements: Yourdatahastobeprotectedagainstunauthorizeddisclosure.Thatbreaksdown intoafewpieces: o Youusuallyneedtoencryptyourdatabothondiskandintransitacross yournetwork o Youneedtobeabletoplaceaccesscontrolsonyourdata,sothatonly authorizedindividualscangettoit Youhavetoprovethatthecorrectaccesscontrolsremaininplaceonyourdata. Thatnormallymeansyouhavetocaptureanychangestoaccesscontrols,suchas filepermissions,inatamperevidentauditlog Youhavetotrackwhoactuallytouchesyourdata,whichalsomeansatamper evidentauditlog
Auditors,thefolkswhocomealongtocheckandmakesureyouredoingallthesethings, arefunpeoplewhocompletelyunderstandifyourbackedupdatadoesntcomplywith theserules.Youwish.Inreality,dataisdata;auditorsdontcarewhereitisorwhyits there,butithastofollowtheserules.
BuildingComplianceIslands
Onepopulartechniqueformakingiteasiertocomplyiscalledislanding.Figure8.8 illustrates;basically,youdesignateoneormoreserversthatwillcontainallthedatathat youhavetofollowtheruleson,andeveryplaceelseyoudontfollowtherules.See,these variouslawsonlycovercertaintypesofdata:HIPAAcovershealthcareandpatient information;PCIDSSconcernsitselfwithcreditcardholderdata.
142
Don Jones
Figure8.8:Creatinganislandofcompliance. Theideaisthatthelawsorotherrequirementsonlycareaboutsomekindsofdata,soyou sequesterallthatdataandjustmanagecomplianceonafewservers.Ofcourse,ifyoure caughtwithdataofinterestoutsidetheisland,youreprobablyintroubleandmightpay somefinessoitpaystobeveryclearwitheveryoneinthecompanyaboutwhatis supposedtolivewhere. Withthisislandingtechnique,youcanhaveaseparatebackupserverbecauseyour backedupdatawillhavetobecompliantjustlikelivedatais.Tomakesureyourbackup solutioncanbemadecompliant,takealookandseeifitcandothefollowing: Encryptdataondisk.Ifyoursolutioncantencryptdataondisk,thenatleastseeif itiscompatiblewithWindowsEncryptingFileSystem(EFS)orBitLocker technologies,whichcanhandletheencryptionforyou. Encryptdataonthenetwork.Thisreferstobackedupdiskblocksthatareflying fromserverstothebackupsolution,orfromthebackupsolutiontoaserverunder recovery.Again,ifthebackupsolutioncantprovidethis,youcanalwaysuse WindowsnativeIPSecurity(IPSec)featurestocreateencryptednetworkchannels betweenserversalthoughIPSecimposesadditionalprocessingoverhead. Note ItspossibletoobtainnetworkcardsthathandleIPSecencryptionin hardware,placinglittleornooverheadontheserveritself.Thesecanbe usefultohavewithinyourislandofcompliance.
143
Don Jones
Auditallaccesstodata.ABackup2.0solutionmakesiteasytogettodata,soits goingtoneedtokeepanauditlogofeveryaccesstothatdata.Unfortunately,simply loggingtotheWindowseventlogsisusuallynotsufficient,simplybecausethatlogis prettyfarfromtamperproofortamperevident;administratorscanclearthelog veryeasily,erasinganyevidenceoftheirownwrongdoing.Typically,acceptable logsareinadatabaselikeSQLServer,whichcanbeindependentlysecured, encrypted,andsoon. Note Logconsolidationsolutions,suchasMicrosoftsAuditCollectionService (MACS;partofSystemCenterOperationsManager),mayprovidean acceptablewaytoturnWindowsSecurityeventlogintoatamperproofor tamperevidentauditlog. Securethedata.Youcantsimplyallowallusers,orevenalladministrators,access tobackedupdata.Youneedawaytosecurethedatasothatonlyauthorized individualscangettoit.Insomecases,thismaymeanconfiguringabackupsolution sothatonlymembersofaSecurityAdministrationteamcanaccessbackedup data. Note Onewaytoapproachthisistocreatededicateduseraccountsthathave permissiontoaccessbackedupdata.Theseaccountswouldhavelong, complicatedpasswords,whichwouldbebrokenintotwoormorepiecesand oftenwrittendownandstoredinasafe.Thatway,twoormorepeopleare neededtoaccesstheaccount,andthustoaccessthedata,ensuringthatno onepersoncangettothedataunobserved. Auditconfigurationchanges.Becausetheconfigurationofyourbackupsystemis criticaltomaintainingcompliance,youalsoneedtoauditanychangestothat configurationandstorethosechangesinatamperproofortamperevidentlog.
Ifyourcompanyisntsubjecttoexternalsecurityrequirementorsimilarlystrictinternal securitypolicies,thenyoureinluck.Youwonthavetoworryaboutanyoftheseconcerns.
144
Don Jones
IntegratingBackup2.0withBackup1.0
WithallthegloriesofBackup2.0,whyintheworldwouldyoueverwanttointegrate Backup1.0?Weliveinaworldthatrequiresflexibility.Youmay,forexample,simplywant anoldschooltapebackupofyourserversonanoccasionalbasisasanextralayerof redundancythebeltandsuspendersapproach,ifyouwill.Thatmayseemparanoidto some,butIsaygoforitthemorebackups,themerrier.Nopieceofsoftwareis100% perfect,meaningwhateverbackupsolutionyoureusingevenBackup2.0style softwaremayhavesomeimperfectionthatcausesaproblemforyou.Havinganold fashionedtapebackup,madeusingdifferentsoftware,providesaPlanBandprevents youfrombeingcompletelyinthelurchifaproblemdoescomeup. TheresnothingaboutBackup2.0sdiskblockbasedtechniquethatintrinsicallyprevents youfrommakingaparallelBackup1.0style,fileandfolderbasedbackuptotape;inmany instances,thetwocanrunsimultaneously.Butnotalways.Keepinmindthatmostbackup solutionsrelyonanagentofsomekind,whichbundlesupdataandshipsitofftothe backupserver.ManyagentsrelyonWindowsVolumeShadowCopyService(VSSorVSC, dependingonwhoyoutalkto)tocopyinusefileslikeSQLServerdatabases.VSSworksby creatingatemporarysnapshotondisk,anditsthatsnapshotthattheagentgrabsforits backup.Sometimes,evenwhennotusingVSS,agentshavetomakelocalcopiesofdataso thattheycanperformcompressionorencryptionorwhatever. Localcopies.Snapshotsondisk.Yes,yourBackup1.0backupagentsarewritingfilestodisk, andyouresimultaneouslyrunningBackup2.0agentisgoingtoseethosechangeddisk blocksandtrytobackthemup.Soyourebackingupthetemporaryfilesusedtomakethe backupofthebackupwithofImlost.Figure8.9showshowchaoticthiscanbe.
Figure8.9:Doubleandtriplebackups.
145
Don Jones
TheresnoreasonfortheBackup2.0solutiontobegrabbingtempfilesandVSSsnapshots; itsalreadybackedupthatdatawhentheoriginaldiskblockshitthediskdrive;bybacking upthesetempfiles,itsjustdoingunnecessarywork. Thatswhy,ifyouregoingtotakeanormalbackupofaservertotape,itsoftenbestto suspenddiskblockimagingonyourBackup2.0server.Ideally,youcanhavetheBackup1.0 agentwritingtoanareaofdiskthatyoucantelltheBackup2.0agenttosimplyignore;that waytheresnooverlap. Note AsIimaginethemworking,itmightnotbeenoughtosimplyshutoffa Backup2.0agent;itwouldlikelyscanforchangesitmissedassoonasyou turneditbackon.ItdependsabitonhowyourBackup1.0agentworks:Ifit cleansupitstempfiles,theywontbetheretobeseenwhenyour2.0agent comesbackonline;ifyour1.0agentleavesstufflayingaroundindisk, though,your2.0agentwillseeitandbackitupunlessyoucanexcludea certaindirectorystructureorfiletypeorsomething.
BackupsasaFormofMigration
Withvirtualizationbecomingincreasinglypopular,werealsoseeinganincreaseinthe availabilityoftoolstohandlePhysicaltoVirtualmigrations,orP2Vmoves.Theresalso V2V,P2P,andV2P,ifyouwanttobepreciseinotherwords,movingaserversimageto whereveritneedstobemoved. Traditionalx2xmigrationtoolsworkprettymuchlikeanimagebasedBackup1.0solution: Theytakeasnapshotoftheserversharddrive,copyitelsewhere,andconvertitinto whatevervirtualformatorphysicalimagelayoutisneeded.Thedownsideisthattheyoften requirethesourceservertobecompletelyoffline,oratveryleastnotunderverymuchuse, sothattheycangetalltheserversfilesinaquiescentstate. ABackup2.0solution,incontrast,wouldmaketheperfectx2xmigrationtool,providedit hadsomeprovisionsforrestoringserverimagestodissimilarhardware(somethingI identifiedasusefulinthedisasterrecoverysectionofthischapter,also).ABackup2.0 solutioncouldeasilybackupeitheraphysicalorvirtualserver,andcandosowhileits running,grabbingchangesastheyhappenalmostinrealtime.Themigrationpartcomes whenyourestoretheserver,againeithertoaphysicalorvirtualmachine.Becausemanyof themajorvirtualizationvendorshavemadetheirvirtualdiskimageformatspublic,a Backup2.0solutioncouldevenproduceacompletevirtualmachineimagethatsreadyto beloadedontoahypervisorandstartedupacapabilityIvementionedseveraltimesin thischapter. Havingabackupsolutionthatcandodoubledutyasamigrationsolutionparticularlya migrationsolutionthatcandoondemandproductionofphysicalorvirtualimagesisa veryusefulbitofflexibility.
146
Don Jones
ComingUpNext
WeveexploredBackup2.0inalmosteverydetailexceptforsomeoperationalrealities. AssumingyouobtainasolutionthatimplementsthetechniquesandtechnologiesIve discussedinthisandpreviouschaptershowdoyougoaboutactuallymanagingthat solutiononadailybasis? Inthenextchapter,Illlookatoneofthemostimportantaspectsofanybackup infrastructure:storagearchitecture.Asyoumightimagine,weregoingtodiscardthe notionthatmagnetictapesarethebeall,endallofbackupstorage,andlookatBackup Storage2.0,whichwillinvolvedisks,SANs,andyes,tapesaswell.Illalsoexplainhowsome newtechnologieslikededuplicationcombinewithsomeoldertechniqueslike compressiontochangethereasonsbehindmanyofourstoragechoices.
147
Don Jones
Chapter9:KeepingYourBackupsStorage Architecture
Storagehaslongbeenadifficultcompanionforbackups.Thefirstbackupswerestacksof punchedcards,althoughmagnetictapequicklycameontothescenetostoremoredataand permitsomewhatfasterrecovery.Eversince,wevestruggledwithwheretoputour backups.Buyinganewterabytefileserverinevitablymeantbuyinganotherterabyteof backupcapacity;somecompaniesusedandstillusefilefilteringtechnologiestoreduce theamountofdataontheirfileservers,primarilytohelpcontroltheamountofdatathat hastobebackedup. Thatstheuglythingthathappenswhentechnologyanditslimitationsstarttodrivethe businessratherthanthebusinessdrivingthetechnology.Sure,keepingerrantMP3filesoff yourfileserversmightbeagoodideaforanynumberofreasons,butingeneralshouldnt usersbeabletoputanybusinessrelateddataontoafileserverwithoutworryingthatit mightnotbebackedup?Isntallourbusinessdataworthbackingup? ThisiswherestoragecomesintotheBackup2.0picture.Forallthegreatthingsthat Backup2.0candointermsofbackingupourdataandallowingfastandflexiblerestore operations,itsuselessifitneedsmorespacethanwecangiveit.
WhereBackupsGoStorageinBackup1.0
Letsconsiderbackupstorageinthe1.0world,whichtraditionallyinvolvedmagnetictape. Today,evenBackup1.0solutionsareabitmoreflexible,allowingmultipleformsofstorage andevenahierarchyofstorage.Isthe1.0conceptofbackupstoragesuitablefora2.0 world?
WhatBackedUpDataLooksLike
InaBackup1.0world,backedupdatareflectstheintrinsicnatureofthebackupitself.In otherwords,thinksnapshots.Atypicalbackupfileconsistsofasinglefile,orperhapsa collectionofrelatedfiles,thatcontaindatastreamedfromasource.Figure9.1illustrates thisprocesswhichyouprobablywatchhappensooftenthatyouvestoppedthinking aboutit.
148
Don Jones
Figure9.1:Traditionalbackupprocess. Sotherearereallytwoproblemshere:First,werebackingupdataasitexistsataspecific pointintimemakingasnapshot,inotherwords,whichistheintrinsicproblemwith Backup1.0.Theotherproblemrelatestohowwerestoringthedata:Wereessentially dumpingeverythingintoonelogicalstructure.Eventhoughthatmaybesplitacross multiplephysicalfiles,itsasingle,usuallyflat,logicalentity.Theresareasonforthat,and itscalledmagnetictape.Magnetictapeisasequentialaccessstoragedevice;itdoesnt supportrandomaccessthewayaharddiskdoes.Becausemostbackupsarewrittento tape,itmakessensetostoretheminasinglestructurethatcanbesequentiallywrittento, andreadfrom,magnetictape.Andtheproblemwithsequentialaccessandmagnetic tapeisthatitsincrediblyslowcomparedwithrandomaccess.
149
Don Jones
WhenIsSequentialAccessFast? Oneareawhereyouregularlyusesequentialaccessandgetreallyfastaccess speedsiswithopticalmediaCDROMs,DVDROMs,andsoforth.Those discsarewrittenmuchlikeanoldvinylrecord,withasingletrack,andthe laserhastofollowthatsinglepathsequentiallytogettowhateverdatayou want.Thereasonitssofastissimplythatthediscisspinningreally,really fast(around1600RPMforaDVD)sothelasercanmovefromthe outermostareaofthedisctotheinnermostinaflash. Tape,unfortunately,doesntsupportthatkindofspeed.Althoughmodern tapedrivescanwindandunwindtapeawfullyrapidly,theystillcantdoso fastenoughtosimulaterandomaccessthewayaDVDROMcan.Thetapeis moredelicatethanaDVDdisc,andtheresusuallyalotmoreofit,sincetapes havetostoremanygigabytes(600GBisntunusual,and1600GBis consideredstateoftheartrightnow)ofdata,whileaDVDonlystoresupto 8GBorso(evenaBluRaydisconlystoresabout50to60GB). ModernBackup1.0solutionstypicallykeepadditionalsupportingfiles,indexesofasort, thathelpthemquicklylocateagivenbitofdatawithinthatmonolithicbackupfile structure.Thus,whenthebackupfileislocatedonrandomaccessstorage,abackup solutioncantypicallydiveinandfindagivenmailboxmessage,orwhatever,prettyquickly. Theproblemisthatmostofusdontstoreourbackupfilesonrandomaccessstorage.In fact,inmanycases,thatbackupfileisbeingwrittendirectlytotape,whereitwillalwaysbe fairlyslowtoretrieve.
TapeDrives
Thatisultimatelythedownsideoftapebasedstorage:theirspeed.Capacitycanbean issue,too,butthatsaquestionofspendingmoremoney:16petabyte(PB)tapelibraries arewidelyavailable.Tapedrivesaretypicallymeasuredinhowmuchdatatheycanmove inanhour,andevenatthehighendofthecurrenttechnologyaround450GB/hour thatsasnailspace,especiallywhensomeonesanxiouslywaitingforafile,orwhenthe bossistappinghisfootwaitingforyoutobringafailedserverbackonline.Keepinmind thathourisntthecompleterecoverytime:Oncethefilesarepulledofftape,youstillhave tohaveabackupsolutiongothroughthefileandlookfortheexactbitsofdatayouneed, andwritethembacktotheservertheycamefrom(ortoanalternativelocation).Heck, evenfindingtherighttapeespeciallyifitsoffsitecantakealongtime. Youhavetousesomecaution,too:Tapedrivespeedsareusuallymeasuredincompressed data.Soa430GB/hourtapedriveisntreallymoving430GB/hourofrawdata,itsmoving thatmuchdataaftercompression.Sothedatawillstillneedtobeuncompressedbeforeit canbeused.Capacitieslike1600GB(1.6TB)areoftenexpressedwithcompression assumed,andtypicallya50%compressionratioisassumed.Thatmeansnativecapacities maybesomethinglike800GB,andthe1.6TBnumberassumesyoureachieving50% compression,whichmeansona430GB/hourdrive,youdbemoving215GB/hourofraw data.Itsfunmath.
150
Don Jones
Somecompaniesgetfrustratedenoughwiththesespeedsandcapacitiesthattheycreatea sortofhierarchicalstorageplan:Backupserverswritetheirfilestolocalharddiskspace, thenstreamthatbackupfiletotapeafterthebackupiscomplete.Diskscanmovedataalot fasterthanatapedrive(think6Gbpsfordisksversussomethinglike.5Gbpsfortape makingdisksatleasttwelvetimesfaster,nottomentioncapableofrandomaccess).An advantageofthisapproachisthatthelatestbackupfileoreventhelatestfewbackup files,ifyouhavethediskspacearehandyonfasterdiskbasedstorageifyouneedthem. Ofcourse,ifyouneedsomethingolder,yourestillbacktotape.
DiskStorage
Itsnotunusual,intheBackup1.0world,toseeadministratorsconsideringpurelydisk basedstoragesolutions.A1TBSATAdiskdriveisfarcheaperthana146GBtapedrive.Its alsofarfaster.Whynotjustrelyexclusivelyondisktodiskbackup? Tworeasons:Disksaremorefragilethantape,anddisksarelessportablethantape.Disk drivesdontdowellwithshockwhenbeingtransported;tapedoesntcare.Disksaremore susceptibletomagneticfieldsthantapes.Diskscanonlysurviveinanarrowertemperature rangecomparedwithatape.Foraneffectivebackupplan,youneedtomovecopiesoffsite, andtaperemainsthebestwaytodothat. ThatswhytheBackup1.0worldoftensettlesfordiskandtape,asFigure9.2shows.
Figure9.2:Tiered,orhierarchical,backupstorage.
151
Don Jones
Thisapproachgivesyoufastaccesstorecentbackupfiles,andtheabilitytoeasilyand safelycarrycopiesofthosebackupsoffsiteforsafekeeping.Infact,thistiered,or hierarchicalbackupstorageapproachhasbeeneffectiveinsolvingsomeofthetraditional problemswithBackup1.0,suchasrecoveryspeed.Itsstillconstrainedbytheinherent snapshotbasednatureofBackup1.0,butthiskindofstorageapproachmayserveuswell forBackup2.0.
RethinkingStorageforBackup2.0
Rethink.Iusethatworddeliberatelybecause,likebackupsthemselves,weintheITworld havebeendoingthingsthesamewayforalongtimeoutofsheerinertia.Letsstart,infact, byrestatingourmissionforBackup2.0: Backupsshouldpreventusfromlosinganydataorlosinganywork,andensurethat wealwayshaveaccesstoourdatawithaslittledowntimeaspossible. Thatsgoingtodriveafewprioritiesforbackupstorage.Weneedtokeepourbackups: Onfast,randomaccessstorage,becauseonlythatofferstheaslittledowntimeas possiblecriteriathatwerelookingfor. Offsite,aswell,becausefastrandomaccessstorageisnt100%reliable. Potentiallyforalongtime,sincewemayneedtorollbacktoafairlyoldpointin time,orretrievedatafromanoldpointintime.
Letsnotusewordslikediskandtaperightnow;letsfocuslessonthetechnologyfora momentandfocusinsteadoncapability.Sure,tapemightbeonewaytogetourdataoff sitebuttheremightalsobeotherways,soletsexplorethose.Disksaregreatrandom accessstoragedevices,butletsfocusonthatrequirementfast,randomaccessstorage overtheimplementationorthemedium.
WhatBackedUpDataShouldLookLike
InaBackup2.0world,wereconstantlycollectingchangeddiskblocksfrommultiple differentcomputers.Weneedtostoreeachdiskblock,alongwithanidentifierofwhereit camefrom,andthetimestampofwhenwecollectedit.Butwecantlosesightofthefact thatwewanttobeabletoretrieveallthediskblocksassociatedwithaparticularmachine, uptoaparticulartimestamp,ondemand.Inotherwords,stickingallthisdataintoatape centricsequentialfilestructureisntgoingtoserveourneedsverywell.Itsmorelikelythat thesediskblockswillneedtogointosomekindofdatabaseFigure9.3illustrateswhat Imtalkingabout.Letsnotworryforthemomentabouthowwellstorethatdatabase althoughdiskstorageisasafebetbutfocusinsteadonthecapabilitiesthisdatabasegives us.
152
Don Jones
Figure9.3:Creatingabackup2.0backupdatabase. Thistypeofstructureallowsanygivendiskblockchangetoberecalledveryquickly,and allowsalltheblocksforanentiremachineuptoacertainpointintimetoberetrievedvery quickly.Thistypeofapproachhasasingledownside:Ithasthepotentialtotakeupgobsof diskspace.Thisis,infact,whereBackup2.0goesfrombeinggreatonpapertopossibly beingnotsogreatinpractice:Ifyouaddupthetotaldiskspaceinvolvedineverydiskblock changeacrossallyourserverscanyoucomeupwithenoughspacetostoreallofthat? Whataboutgettingitoffsiteforsafety?Whataboutkeepingalotofitforalongperiodof time?Giventhesheeramountofdatainvolved,canBackup2.0meetourthreeobjectives forbackupstorage? Fast,randomaccessstorage Offsitestorage Longarchivalperiod
Wellhavetosee.Thisiswhereweneedtostartreallylookingatthestoragetechnology ortechnologiesthatthisisgoingtouse.
PhysicalStorage:Tapes,SANs,NAS,theCloud,andMore
Letsgetbacktophysicalstoragemedia,andletsbehonestwitheachother.Thereareonly twopracticalkindsofmassstoragedevicesavailabletoustoday:diskandtape.Were goingtohavetofindawaytomakethemwork,becauseitsallwevegot.
153
Don Jones
Butdisksinparticularcomeinawidevarietyofimplementations:Relativelycheaplocal storageinsideaserver(orinanattachedstorageexpansioncabinetofsomekind), inexpensivenetworkattachedstorage(NAS)appliances,andsomewhatmoreexpensive storageareanetworks(SANs).AlloftheseofferRAIDcapabilitiestohelpprotectusinthe eventoneortwodrivescrashonus,andallofthesearewellunderstood,robust,mature technologies.iSCSIinparticularismakingNASandSANstoragelessexpensive,better performing,moreflexible,andsoon. Tapeswell,tapesaretapes.Mostbusinesseshaveautoloadersortapelibrariesthatcan automaticallychurnthroughseveraltapestostoretonsandtonsofdata,andsometape librariesareevendirectlyconnectedtoanetwork(althoughitsstillcommontohavea tapedriveorlibraryhangingoffthebackofadedicatedserverortapelibrarycontrollerof somekind). Youcanstartdraggingcloudsintotheconversationbecausecloudstorageisbecoming fashionable,butitsjustmoreharddrivesandtapeslocatedsomewhereelse;itisntreallya differentkindofstorage. Whatreallymattersisspeed,cost,andlocation.Remember,wehavethreerequirements forourbackupstorage: Fast,randomaccessstorage Offsitestorage Longarchivalperiod
Thefirstrequirementaddressesspeed.Thesecondaddresseslocation.Thethirdimpacts costthemost(afterall,ifwewerehappyonlybeingabletorecovertolastnight,we wouldntneedtostoreverymuchbackupdata,andsothepotentialcostofourBackup2.0 solutionwouldntbeveryhigh). AswithBackup1.0,theanswerisgoingtobeamixofbackuptechnologies.But,aswedid withBackup1.0inthefirstplace,weneedtorethinkwhatahierarchicalbackupstorage setupmightlooklikeandweneedtoconsiderallourmodernoptions.
RethinkingHierarchicalBackupStorage
LetsstartbyacceptingthefactthatourBackup2.0solutionisgoingtobebackingupdata todisk.Thatsgoingtogetusourfast,randomaccessstorage.Wecandecidehowmuchwe wanttospendondiskspace,andthatllconstrainhowlongofanarchivewecanmaintain infast,randomaccessstorage.Figure9.4showstheinitialphaseofourstorage architecture.
154
Don Jones
Figure9.4:Initialstoragearchitecture. Wehavefourserversthatwewanttobackup,andacentralbackupserveralongwith somediskstorageisholdingitall.Again,ourabilitytoprovidemorediskspaceisthe onlylimitationonhowmuchdatawecanstoreonsite,meaningthemorediskspacewe canthrowattheproblem,thelongerwecankeepbackupsonsite,andthefurtherinthe pastwecandivewhenweneedtorecoversomething. Backup<>Archive Dontthinkofyourbackupsystemasanarchive.Inotherwords,ifthereare specificpiecesofdataquarterlyfinancials,lastyearstaxes,personnel records,andsoonthatyouneedtopermanentlyarchive(orevenarchive foranumberofyears),yourbackupsolutionisnttheplacetodoit.Those archivesshouldbesomeplaceelseoftenontape,whichisprettydurable anddoesntconsumepowerwhenitisntbeingused.Youmightburnfilesto opticalmedia,whichisequallydurableandlowpower.Butjustbecauseyou needtokeepcustomerrecordsfor,say,10years,doesntmeanyourbackup systemneedstobecapableofstoring10yearsofdata. Mygeneralruleisthis:IfigureouthowfarinthepastIcangoinabackup systemuntilIstartrunningintodataIwouldnever,everrestore.Thatsall thedatamybackupstoragesystemsneedtohold.Inmanycases,Ifindthat tobeamonthortwo,possiblyevenaslongasaquarter,butanythingfurther backtendstobecompletelyuseless,sotheresnoneedtobackitup.Archive it,perhaps,butitdoesntneedtoliveonthelivebackupsystem.
155
Don Jones
Yourservicelevelagreements(SLAs)withyourusersplayanimportantrole here.Wecanretrieveanythingupto3monthsold,andwecandoitinan hourorso,isagoodSLA.Anythingolderthanthat,wecansearchthe quarterlyarchivesforbutthatmaytakeafewdays,becausewehaveto bringtheminfromoffsite,andtheresacostassociatedwithdoingso. ThatsagoodnextstepinanSLA:Establishingwhatcanbedone,andwhatit willcost,andhowlongitwilltake. Ournextstepistogetacopyofourbackupsoffsite.Notforlongtermstoragebutbecause wewanttobecoveredintheeventthatourdatacentercontainingourbackupserver burnsdownorsomethingawful.Wewanttogetacopyofourbackupdataoffsitefairly frequently,butwedontneedtokeepcopiesoffsiteformorethanouronsitebackupdata retentionperiodsay,3monthsorso. Onewaytodosois,ofcourse,tape,asFigure9.5shows.
Figure9.5:Movingdatatotape. AsIveacknowledged,tapeisagreatwayforgettingdataoffsiteinbulk.ThetrickisthatI dontneedmytapestobeanarchiveImjustusingthemtocovermyselfintheeventofa backupserverdiskfailure.SoifImonlykeeping3monthsofdata(inmyexample)inthe backupserver,thenmytapesonlyneedtobegoodforabout3months.Tapesdorepresent asnapshotofmybackupserversdata,soifIdolosemybackupserver,thenImatriskfor losingdatabecauseIvelostwhateverbackupsoccurredbetweenthelasttapebackupand thebackupserverfailure. Anotheroption,then,istomigratedatatothecloudtoachievemyoffsiteprotection. Figure9.6showswhatthismightlooklike.
156
Don Jones
Figure9.6:Movingdataintothecloud. Thisaccomplishesthegoalofgettingthedataoffsite,anditisntasimpracticalasitmay soundatfirst.Tobeginwith,cloudbasedstorageisgettingcheaperallthetime:Compareit withthecostofmakingtapebackupsandstoringthemoffsite,andyoumightfindthatthe twotechniquesarentworldsapartintermsofprice.Solutionsalreadyexisttomovedata offsiteinnearrealtime,usingadvancedcompressionandbandwidththrottling techniquesthathelpconserveWANbandwidth.Thistypeofbackupwouldbemoreupto datethanatapebackup,solessdataisatriskmeaningtheremightbejustificationfor somewhathigherpricingthantraditionaltapes.Thereareevensolutionprovidersout therewhocantakeyourbackedupdiskimagesandmovethemtotheirownvirtual machinehostserversalsolocatedinthecloud.Thatcanprovideanexceptionallyquick recoverytime,ifthatssomethingyoufeelyoullneedforyourbusiness. Youcouldalsoconstructyourownprivatecloud,replicatingbackupcontentfrom multiplebackupserverstoyourownbackupofthebackupserverperhapslocatedina differentoffice.Figure9.7illustratesthisidea.
157
Don Jones
Figure9.7:Privatecloudbackup. ThebackupreplicawouldactuallybeusingBackup2.0techniquestobackupthefrontline backupservers,meaningverylittledatawouldbeatriskatanygiventime.Ifaserverfails, itsbackupservercanrecoverit;ifabackupserverfails,thecentralbackupreplicacould recoverit.Yourcostsgoup,butyourabilitytosurvivemultipledisastersalsoincreases,so youregettingsomethingforthemoney. Havinganoffsitereplicacangreatlyimproveyoursituationintheeventofatotallocation failure,suchasaflood,fire,lossofutilitypower,meteorstrike,andsoon.Thebackup replicacanbeusedtorapidlyreconstructallyourlostservers,perhapsasvirtual machines,inanotherlocation.Rapidlyisthekey,there:Youwontbespinningdataoffofa dozentapesforhoursandhours,youllbeimmediatelybuildingvirtualmachinedisk imagesandgettingthemrunningonavirtualmachinehostperhapsevenoneyouve leasedforthisparticulardisaster. Thebackupreplicacouldalsoserveasasourceforarchivalactivities.UsingBackup2.0 techniques,thebackupreplicaitselfwouldbeabletomountdiskimages,meaningyou couldgrabfiles,folders,databases,andwhathaveyouandcopythemtotapetocreatea permanentarchiveofwhateverdatayourbusinesshasthatneedstobepermanently archived.Figure9.8illustratesthisaddition. 158
Don Jones
Figure9.8:Addingtapearchivaltothemix. Again,thepurposeofthesetapesistopermanentlyarchivefilesthatneedtoberetained beyondyourfastrestorecapabilities.Typically,archivedfilesdontneedtobeaccessed veryoften,andwhentheydoneedtobeaccessed,itsforreferenceorreadonly purposes.Opticaldiscsmightalsobesuitableasanarchivalmedium,andyoucouldstore copiesonsiteaswellasoffsite,ifdesired.
159
Don Jones
Thepointofallthisisthatyoucanachievewhateverbusinessneedsyouhave.Thekeysare toclearlydifferentiatedifferentneeds: Usefastdiskstoragetocreateafastrecoverycapability.Establishareasonable periodoftimethatyoullretaininfastrecovery,ortier1storage. Getthatdataoffsitequicklyandfrequently.Ideally,getitoffsitetoanotherfast recoverysolution.Lessideally,getitoffsiteinsnapshotformlikelyontape. Periodicallymigratefilestoslower,tier2storageforlongtermarchivalpurposes. Again,archivefilesarentneededintheeventofafailure,theyreneededasalong termorpermanentreference.Archivingisntpartofyourbackupstorage architecture,althoughyourbackupsmaybeastartingpointforcreatingarchive copies.Youdontchoosetoretaintapecopiesfor5yearsbecauseyouthinkyoull needtorecoverthosefilesandbeginmakingchangestothem;youdosobecause someonemightneedtorefertothosefilesatsomedistantpointinthefuture. Backup<>Archive,Redux Abackup,forme,issomethingyoukeepbecauseyoumightwanttoreturn yourproductionenvironment(orabitofit)tothepointintimethatthe backuprepresents.Youmaywanttorecoveranaccidentallydeletedfile,for example,oryoumightwanttorestoreafailedserver,oryoumaywantto recoveracrasheddatabase.Backupsareforwhenthingsgowrong. Archivesarethingsyoukeepbecausethedatainthemhasinherentvalue. Youllneverwanttomakeittheworkingcopyagain,butyoumighthave needtorefertoit.Archivesarentusedbecausesomethingwentwrong,but becauseyouneedtolookatsomethingupfromthepast. WhenIrunacrosscompanieswhosavebackuptapesforayear(ormore),I diealittlebitinside.Doesanyoneseriouslycontemplaterollingbackthe productionenvironmenttoayearago?No,theydonot.Whattheyvedoneis createdanarchiveoutofabackupandlikely,thatarchivecontainsdata theywillneverneedtoreferto,likecopiesofoperatingsystem(OS)or applicationfiles. OnereasonImtoldthatdiskstorageisntsuitableforbackupstorageis becauseitstooexpensivetokeepenoughdatafarenoughintothepast.My problemwiththatisthatyoushouldntbekeepingbackupsforthatfarinto thepastatsomepoint,thebackuphasnovalueasalive,production system,anditsturnedintoanarchive.Whenitdoesso,itshouldbetrimmed downtothedatayoullactuallyneedtoreferto,andwrittentosecondor thirdtierstorage. Carefullyconsideryourbusinessspecificneedswithregardstobothbackupsandarchives. Thatllhelpyouproperlyplanfortherightamountofdiskspace,storagetiers,offsite storage,andsoon.
160
Don Jones
SavingSpace
Ofcourse,diskspaceisstillnotfree,andsothelessofitwehavetouseforourbackups,the better.ThatswhyaBackup2.0solutionshouldalsoincludetwomajorfeatures compressionanddeduplicationtoreducediskcostsasmuchaspossible.Lessdiskusage onthebackupservermeanslessstorageusedateverysuccessivetierwhetheryoure movingdataoffsiteontapes,toacloudprovider,ortoanoffsiteprivatecloud.
DeDuplicationofData
Deduplicationisahotnewtopicintheworldofstoragemanagement,andtheresno reasonBackup2.0shouldntbeonthatbandwagon.Theideaissimple:Ratherthanstoring multiplecopiesoftheexactsamedata,yourstoreitonce,alongwithsomenotesaboutall theidenticalcopiesthatwouldnormallyexist. Youwouldprobablybeappalledattheamountofduplicatedataonyourservers.EveryOS file,forexample,isduplicatedacrosseachservernoneedtostorethemallindependently whenonecopywilldo.Sometimes,theOSwillwriteablockofdatawiththesameactual contentsaswerealreadyondisknoneedtostorethatdiskblockasachangebecause theresalreadyacopyofthesamedatainstorage. Withoutdeduplication,ourbackupstoragedatabasecouldlooksomethinglikeFigure9.9, withmultipleblocksofduplicateddatabothbetweenserversandevenwithinthesame servers.
Figure9.9:Duplicateddatainthebackupstoragedatabase. Deduplicationsavesusroom,byeliminatingtheduplicateddataandjustkeepingtrackof wherethemastercopyofthatdatalives.Figure9.10showswhatthatmightlooklike.
161
Don Jones
Figure9.10:Deduplicateddatainthedatabase. Theideaisthatthepointertotheactualcopyofthedataoccupieslessspacethanthedata itself,sotheresanetsavings. Therearenumeroustechniquesfordeduplicatingdata.Atahighlevel,withabackup solution,youcouldsimplylookatdiskblocks,whichareusuallyrelativelysmall.With largerdiskblocks,suchasan8KBblock,youmightuseatechniquecalledchunking,where youbreakdowneachblockintoasmallerchunksay,1KBchunks.Itllbeeasiertofind duplicationatthatlevel,meaningyoucansavespaceincrementally. Thetradeoffrelatestothepointersize.Ifduplicateddatawillbereplacedwitha12byte pointer,theresnosenseinlookingforduplicationatthe12byteorsmallerlevelbecause anyspaceyousavedwouldbecompletelyoffsetbythepointeryouhadtocreate. Note Inbackupsolutions,therearetwokindsofdeduplication,whichIllcall SourceandTarget.InSourcededuplication,thebackupsolutionwillde duplicatedatafromaparticularsource,likeaserver,butitwillallow duplicatedatatoexistbetweensourceslikebetweentwoservers.Youcan savemorespacewithTargetdeduplication,whichdeduplicatesalldata regardlessofwhereitcamefrom. Youwanttolookforasolutionthatoperatesatthesmallestpracticalchunksize,which maybethesizeofatypicaldiskblock.Somesystemsmayonlydeduplicateentirefiles, meaningyouremuchlesslikelytofindasmanyduplicatesandsavespace.
162
Don Jones
CompressionofData
Compressionis,ofcourse,alongusedtechniqueforreducingstoragerequirements,and Backup2.0shouldmakegooduseofit.Infact,agoodsolutionwillactuallysupportmany compressiontechniques,asdifferenttechniquesworkbestacrossdifferentkindsofdata. Allcompressioninabackupsystemislossless,whichmeanstheoriginaldatacanbe completelyreconstructedupondecompression.Infact,compressionalgorithmsarereally justaveryadvancedformofdatadeduplication,functioningacrosssmallerandlargerdata setsandacrosspatternsaswellaspurelyduplicateddata.Forexample,supposeadisk blockcontainedthisdata,expressedinhexadecimal: 1730000AE4627DBCBCBCBCBCBC733475475475475 Acompressionalgorithmmightreducethatto: 173(0x4)AE4627D(6xBC)733(4x475) Theideaistotakerepetitionsoveracertainlengthandreplacethemwithsometokenthat allowsthepatterntoberecreatedduringdecompression.Shorterpatternsmightnotbe replacedatall,ifthereplacementtokenwouldbelargeroreventhesamesizeasthe patternbeingreplaced. Thenatureofdataitselfisusedbyothercompressionalgorithms.Forexample,asimple textdocument,storedasaplaintextfile,typicallyusesaverylimitedcharacterset.These characterscanbeexpressedin4bits,anddonotneedanentire8bitbyteyettheyare storedinfullbytesondisk.Sixcharactersmightlooklikethis: 00000110 00001000 00001101 00000110 Theupper4bitsofeachbyteareessentiallywasted;acompressionalgorithmcanbyte packthese4bytesinto2bytes,fora50%reductioninspace.Thissimplyinvolvescopying thelower4bitsofonebyteintotheupper4bitsofthepreviousbyte: 10000110 01101101 Solongasthedecompressionalgorithmknowshowtoreversethisprocess,itworks perfectly.Ofcourse,thisdoesntworkwithmorecomplexdata,illustratingtheneedfora compressionsystemtosupportmanydifferentkindsofalgorithms.
163
Don Jones
Commonlosslessalgorithmsinclude: Runlengthencoding(RLE) LZW(andvariantsLZ77andLZ78),whicharedictionarycoders DynamicMarkovCompression(DMC) Variousformsofentropyencoding Resource Ifyoureinterested,youcanreadaboutthespecificsoftheseandotherforms ofcompression http://en.wikipedia.org/wiki/Data_compression#Lossless_data_compressio nisagoodplacetostart,asitprovidesanindextosomeofthemore commonlyencounteredalgorithms. Betweencompressionanddeduplication,abackupstoragesystemcansavealotofspace. Itsrelativelystraightforwardforcompressionalgorithmstoachieve50%compression (whichiswhysomanytapebackupdevicemanufacturersassumethatrateofcompression intheirstatistics);deduplicationcaneasilyachieveanadditional20to30%reductionin space,meaningyourbackupstoragecanuseasmuchas80%lessspacethanthedataitis backingupactuallyconsumesinproduction.
WorkingwithBackedUpData
Thelastconcern,ofcourse,isbeingabletouseyourbackedupdata,andthestorage architectureobviouslyplaysanimportantrole.InaBackup2.0solution,usingthedata meansbeingabletoreassemblethediskblocksforanentireserverharddrive,orjusta portionofit,quickly.Ideally,youshouldbeabletopushthoseblocksbacktotheiroriginal location,ortoanalternativelocation(suchasavirtualmachine)forrecoverypurposes. Youmightalsowanttheabilitytomountthoseblocksasaworkingdiskimagesothatyou canbrowseitandretrievedatafromitwithouthavingtopushtheblockstoarealserver. Lookforasolutionthatwillgiveyouflexibilityinhowyouaccessandutilizeyourbacked updata.
ComingUpNext
AlotofmydiscussiononBackup2.0thusfarhasfocusedonbackupandrestore:Making sureyoucanretrieveasinglefile,orasinglemailbox,orasingleotherpieceofdatawhen youneedto.Alongtheway,Ivemadementionofdisasterrecoverywhichiswhatyoull bedoingwhenanentireserver,oranentiredatacenter,crashesbutinthenextchapter, Imgoingtofocusexclusivelyonthatimportanttopic.Disasterrecoveryisadistinctsetof concernsfromnormalbackupandrestore,anditsanotherareawherewellreallyhaveto examineourassumptions,carefullyreconsiderouractualbusinessobjectives,andseewhat a2.0approachcanoffer.
164
Don Jones
Chapter10:WhenEverythingFails:Whats YourDisasterRecoveryPlan?
Muchofthisbookhasfocusedonbackupandrestoreratherthandisasterrecovery.The difference?Iregardrestoringassomethingyoudowithasinglefile,oragroupoffiles,or asingleemailmessage,oranentiremailboxsomethinglessthananentireserver.It mightbeadisasterthatafilewasaccidentallydeleted,butitstypicallyadisasterforone ortwopeoplenottheentirebusiness.Atruedisaster,inmyview,iswhenanentire servergoesdownorworse,whenanentiredatacenterisaffected. Thereasonmuchofthisbookhasfocusedonrestoresisthat,frankly,itswhatwespend moretimedoing.Itsnotallthatcommonforanentireservertofail,orforanentiredata centertoencounteradisaster.Itdefinitelyhappens,butwhathappensalotmoreis someoneneedingyoutopullasinglefileormailboxfrombackups. Inthischapter,however,Imgoingtofocusentirelyondisasterrecovery.Disastersdo happenfloods,hurricanes,powersurges,andsoforthcantakeoutentireserversoreven entiredatacenters.Onetime,Ihadtodealwithacompleteweeklongpoweroutagewhen someonerantheirpickuptruckintothetransformeronthecornerofouroffices propertytalkaboutadisaster.Infact,Illusethatstoryasakindofrunningexampleof whereBackup1.0reallyletmedown. WeregoingtostickwithourBackup2.0manifestobecauseitsjustasapplicableto disasterrecoveryasitistoasinglefilerecovery: Backupsshouldpreventusfromlosinganydataorlosinganywork,andensurethat wealwayshaveaccesstoourdatawithaslittledowntimeaspossible.
OldSchoolDisasterRecoveryPlanning
Youhavetoplanfordisasterrecovery.ItsbecomesocommonplaceforusITfolkstotalk aboutthecompanysdisasterrecoveryplanthatweoftenforgetthatalotofplanningis involved.Infact,areallylargenumberofbusinessleveldecisionshavetogointotheplan beforewetechnologistscanevenbeginourendoftheplanning.
165
Don Jones
Primarily,yourbusinessleadersneedtodecidehowmuchofadisastertheywanttobe abletosurvive.Herearesomeexamplesofcompletelydifferentscenarios: Aserverfailsbecauseofahardwareissueorcatastrophicsoftwareissue.Thisisnt theendoftheworld,obviously,butdependingontheserver,itcanbeprettybadfor business. Youloseutilitypowerforashortperiodoftime.Simplyhavingalotofbatteriesora backupgeneratormightbeagoodwaytomitigatethiskindofdisaster. Youloseutilitypowerforlongerthanyoucanpracticallymitigateusingbatteriesor agenerator.Thisusuallymeansyourelookingatbringingatleastsomeofyour applicationsbackonlineoffsite. Anaturalormanmadedisasterfire,flood,tornado,hurricane,orsomething elsestrikes,renderingyourdatacenteruseless.Again,youreprobablylookingat anoffsiterecoverytosurvivethis.
Thereasonyourbusinessleadersneedtoconsidertheseisbecausesomeofthesolutions canbeprettyexpensive.Theyllneedyourhelpinfiguringouthowexpensive,sothatthey candecideifitsworthittohavearecoveryplaninplaceforanygivenscenario. Youllalsoneedtorateyourdatacentersservices.Whatcanyoureallydowithoutinthe eventofafailure?Willeveryoneneedtoaccessresources,orwillthecompanyberunning onaskeletoncrew?Itsobviouslycheaperandeasiertoplanforextremedisasterscenarios thatinvolverestoringahandfulofservers,asopposedtoscenariosthatwillrequireyour entiredatacentertosomehowbereplicatedoffsite. Thesekindsofdecisionsalsochangedrasticallybetweencompaniesofdifferentsizes.If youworkforalarge,globallydistributedcompany,thenyoualreadyhaveoffsitedata centers;youjustneedtofigureouthowtoensurethatonedatacentercantakeoverfor anotherintheeventofadisaster.Smallercompaniesthatoperateoutofasinglelocation, however,cantgetoffsiterecoverywithoutpayingadditionalforitandthatcanbe expensive,dependingonhowyouchoosetodoit.Inastrangeway,itseemsthatifyou spendenoughonITlikehavinggeographicallydistributeddatacentersinvarious locationsdisasterrecoverycanactuallybecomealotcheaper.
166
Don Jones
ThecompanyIllbeusingasmyBackup1.0casestudywasaretailer,with asingleheadquartersanddistributioncenterinthemidAtlanticregion. Wehadasingledatacenter,whichhousedeverysingleITassetwe ownedrightdowntotheofficephonesystem.Ourstoreswere,of course,independentandcouldoperateforsometimeifthehomeoffice datacenterwasoffline,butwithoutthatdatacenterwewouldntknow whatproductsstoreshadsold,andcouldntpracticallygenerate restockingshipments.Ourexecutiveswantedtobeabletosurvive anythinguptoandincludingacompletelossofthedatacenter,andso wecontractedwithanoffsiterecoveryfacility.Thebasicdealwas,when wecalled,theydhaveacertainsetofhardwarereadyforus,alongwith telecommunicationsservices.Wedhavetotakeitfromthere.
BareMetalRecoveryandServerRebuilds
IntheBackup1.0world,wehadtwowaysofdealingwithafailedserver:Rebuildit,or recoverit. Rebuildingisahorriblethingtohavetodo,andIdontknowofanyadministratorwho looksforwardtoit.Youretalkingaboutstartingfromscratch:Installinganoperating system,installingapplications,andsoon.Presumablyyoucanrestoreapplicationdata frombackups,oryouvebasicallylosteverything.Veryfewcompaniesmaintaindetailed enoughdocumentationonserverconfigurationstoensureacompletelythorough,accurate rebuild,whichmeanstherearealwaysafewconfigurationitemsthatgetmissed.Ionce hadtorebuildanExchangeServercomputerthisway,andfortwoweeksafterwardnoneof ourremoteuserscouldaccessthecomputer.Wefinallyrealizedthatwedforgottentore configuretheservertousethenondefaultportnumbersthatourfirewallwasallowing through(whichis,bytheway,abigargumentinfavorofstickingwiththedefaultslessto worryaboutifyoudohavetorebuildfromscratch). Rebuildingtakeshours,ifnotdays,anditlocksdownsomeofyourmostskilledhuman resourcesforthatentireperiod.Becauseyoureusuallyrebuildingasfastaspossible,and underagooddealofstress,yourealotmorelikelytomakemistakesandmesssomething up,too. Youalsoneedsomeplacetorebuild.Intheeventofaserverhardwarefailure,thatmay meaneitheranewpieceofhardwareyoudidhaveacompleteserverjustsittingaround waiting,right?oravirtualmachine,assumingyouhaveavirtualizationinfrastructure, andavirtualizationhostwithavailablecapacity.
167
Don Jones
TheretailerIworkedwithactuallydidmaintaincoldspareservers, meaningwehadtwoorthreeserverssittinginclosets,waitingtobeused incaseaproductionserverupanddied.Wetriedtominimizethe numberofservermodelsinourdatacenterwhichfranklyreducedour flexibilityagooddealsothatwecouldminimizethenumberofspares wedneedtokeep.Thiswasafewyearsago,andvirtualizationwasnt reallyanoption.Wedidhavemaintenancecontractsonallourserver hardware,butforafewspecificserverswecouldntreallyaffordany downtime,sohavingthesparewasawaytocutthatdowntimebyas muchaspossible.Itwasalsoexpensive,andwehadtohavemaintenance contractsonthespares,too.Wewerepayingformaintenanceon hardwarethatwedidntnormallyevenuse. Howdoesrebuildingfitourgoals? Backupsshouldpreventusfromlosinganydataorlosinganywork,andensurethat wealwayshaveaccesstoourdatawithaslittledowntimeaspossible. Prettypoorly.Youredefinitelynothittingaslittledowntimeaspossiblewitharebuild, andyoureprobablygoingtobemissingsomedata,dependingonhowoldyourmost recentapplicationdatabackupsare. Baremetalrecoveryistypicallyfaster.Theassumptionisthatyouhaveacompletebackup oftheentireserver,andyouwanttojustdumpthatontoafreshservereitherhardware orvirtualizedtogetyourserverbackonline.Ofcourse,howmuchdatayoulosedepends entirelyonhowrecentyourmostrecentbackupis.Howmuchdowntimeisinvolved dependsentirelyonhowyoumadethosebackupsinthefirstplace.Forexample,lets supposeyourbackupsareallontapedrives,andyouneedtorestoreanExchangeServer. LetssaythatExchangeandWindowstogethertakeupabout10GBofdiskspacefor operatingsystemandapplicationfiles,andyouvegotanother400GBofmailboxdata. Thats410GBtotal.Illgiveyouthebenefitofthedoubtandsupposethatyouhavethevery latestinDLTtapebackups,andcanfitanentirefullbackupononetape.Illassumeyouget typical2:1compression,meaningyoullbeabletopullallthatdataoffoftapeinaboutan hour(assumingatapedatatransferrateabout215GB/hourofraw,compresseddata). Assumingyoucanquicklylayhandsontherighttapeandthatyouhaveallofyourrecovery bootdiscshandy,youcanprobablyhavetheserverbackonlineinacoupleofhours losingonlythedatathatwascreatedormodifiedsincethetapebackupwasmade,of course.
168
Don Jones
Isthatreasonableforyourcompany?Twohoursisntbad,butIveneverbeenabletogeta CEOtoseeitthatway.Worse,mostofusintheBackup1.0worldareonlygrabbing backupseverynight,atbest,meaningiftheserverdiesatlunchtimeyouvelostaroundhalf adayswork.Nobodyisgoingtobepleasedaboutthat;manyofthemmayacceptitbecause theyvebeenconditionedtobelievethatitsthebestthatcanbedonebuttheyrenot happy. Ofcourse,ifyourenotgrabbingafullbackupeverytimeyoubackupaserverandmost ofusarentthentherecoveryisgoingtotakelongerasyoushuffletapes.Lastweekends fullbackup,afewincrementalsfromduringtheweek,andsoforthitaddstime,nomatter howfastyourtapedrivesare.Andofcourse,ifyourenotrunningthelatestspeeddemon tapedrives,yourenotgoingtobepullingthat420GBofdataoffoftapeinanhour. WhyIncrementalandDifferentialBackupsAreNoFun Ifyouthinkaboutit,bothincrementalanddifferentialbackupssaveustime onlyduringthebackupphase;whenthetimecomestousethosebackups, theyactuallyslowusdown. Youhavetostartwithyourmostrecentfullbackup,whichmeansyoure completelyrecoveringtheserversay,400ishGBofdatainmyExchange Serverexample.Thenyoustartinwiththemostrecentdifferential,orstart applyingincrementals,dependingonhowyourbackupplanworks,Every incrementalisoverwritingdatayouvealreadyrestored,andeachsuccessive incrementalisprobablyoverwritingdatafromthepreviousincrementals thatis,afterall,thewholepointofincrementals.My400GBExchange exampleusuallyresultedinabout10GBofincrementalbackupdataevery night,meaningthataThursdayafternooncrashmeansIhavetorecover around460GBofdata,increasingmytimeandtheamountoftapeloadingI havetodo. Fullbackupsarethebestchoicebutgettingafullbackupofeveryserver, everynight,isusuallyimpractical.Andyourestillatriskforallthedatathat getsmodifiedorcreatedduringtheday,evenifyouareabletograbafull backupofeveryservereverynight. Sowhatstheproblemwiththisplan?Ithonestlytakestoolong.Anhourisyourabsolutely bestcasescenarioforrestoring400GBorsoofdata.Iusedtoknowcompanieswhowould actuallyspinupmoreservers,justsoeachservercontainedlessdata,andcouldbe recoveredmorequickly;whentheyeventuallyrealizedhowmuchmoretimeandmoney theywerespendingonserversandmaintenance,theyconsolidatedeverythingandjust decidedtotryandfindawaytomakebackupsfaster. Theunderlyingproblemwiththisplanstarts,ofcourse,inhowthebackupsaremade. Recoverytakesalongtimebecausewetakeshortcutsonthebackupsideofthings,inorder togetallourdatabackedupduringamaintenancewindoworsomething.Backingup400+ GBofdataisaprettybigdealwhenyouhavemultipleserverslikethattoworryabout;we havetotakeshortcutslikeonlydoingweeklybackups,orusingincrementalbackups,just togetitallbackedupduringthetimewehavetoworkwith. 169
Don Jones
Butthinkaboutsomething:Letssayyouhaveaserverthatgeneratesabout10GBofdata eachnightforanincrementalbackup.Thatmeansyourecreatingorchangingaround 10GBofdataduringtheworkday.Letsgoworstcase,andsupposethatthosechanges occurredduringasixhourperiod(everyoneshoweduptoworklate,tookalonglunch, andwenthomeearlyhappensallthetimeinyouroffice,right?).Thatmeansyoure changingabout1.6GBperhour,orabout.02GBperminute.Thatisntactuallythatmuch data.Assumingthechangesareevenlyspreadout(whichIrealizetheyrenot,butpretend tomakethematheasyonme),yourechanging.0004GBofdatapersecondthats500KB persecond,ifImgettingmydecimalplacescorrect,andbackingup500KBeverysecondis hardlyevenworkonamodernnetwork.Assumeeachserverisincludedonadedicated networkthatsonlyusedforbackupdata;ifeveryserverwasgenerating500KBofdata eachsecond,aspeedy10GbEthernetnetworkcouldhandleafewhundredserverswith ease(actually,somethinglike2,000serversifIhaventmisplacedadecimalpoint,butfew hundredseemsmorepractical).Figure10.1showswhatImtalkingabout,withseparate networkstocarryusertrafficandbackuptraffic.
Figure10.1:Creatingadedicatedbackupnetwork. Mypointisnotreallyaboutprecisionmath;itsthatwerenottalkingaboutimpossibleto achievenumbers,andthisisexactlywhatBackup2.0proposes:Continuousdataprotection, capturingchangesastheyoccur,ratherthanwaitinguntiltheybuildupintoanenormous pileofdatathatwesomehowhavetograbduringaonehourmaintenancewindow.Plus, bygrabbingdatacontinuously,wereatriskforlosingverylittledataintheeventofa failure.
170
Don Jones
OffSiteRecovery
Whentheentiredatacenterisaffected,offsiterecoverycanbeasolutionforgettingat leastsomeofthebusinessbackonline. HereshowoffsiterecoveryworkedatthatretailerImentioned:We droveovertotherecoveryfacility(whichbythewaywasinthesame state,soaregionaldisasterwouldprobablyhaveputitoutof commissiontoo).Wewouldhavecalledthemtoletthemknowwewere coming,andsincetheyhandledourbackuptapestorage,theywould startrootingaroundandgettingourlatesttapesforus.Wedshowup andstartfeedingtapestotapelibraries,andstartrestoringaboutahalf dozenservers.Wedidnthaveagoalofrestoringeveryserverfromour datacenterwehadidentifiedserversthatthecompanyneededto operateinasortofworstcasescenario,andweworkedonthoseservers first.Anythingelsewouldberestoredlater,aswegotthetime. Inourpracticeruns,thistookabouteighthours.Weweredelightedto getanew,fastersetoftapedrives(aftermakingsuretherecoveryfacility hadthem,too)thatcutthetimedowntofourhours.Forourneeds,this wasbasicallysufficient:Itgotusupandrunningquicklyenoughtostart collectingdatafromourretailstoresagain,andtostartprocessing restockshipments.Assumingoursoledistributioncenterwasnt completelyunderwaterorsomething. OurtechniquewaswhatInowcallacoldrecoverysite.Inotherwords,wedshowupand havenothingbuthardware,sometelecomlines,andaboxfullofmagnetictapes.Wedtake itfromthere,essentiallyreconstructingaportionofourdatacenteralmostfromscratch, usingfairlyrecentbackuptapes.Wesentdatabackuptapesoffsiteeverymorning,anddid aweeklycompletebackupofeveryserver,soourworstcasescenariohadusmissingabout adayofdataatmost. Wewereprettypleasedwithourselvesoverthisplan:Adayofdataatriskandadaytoget backonlineseemedprettyawesome.Ourexecutiveswereokaywiththeplan,too, becauseagaintheydbeenconditionstoacceptthatasbeingthebestthatcouldbe accomplished.Atthetime,whichwasyearsago,itwasaboutthebestthatcouldbedone. Theproblemisthatmanycompaniesstilloperatethatwaytheyhaventreevaluated theirconditioninginlightofnewtechnologiesandtechniques.Theyhaventsatdownand statedtothemselves: Backupsshouldpreventusfromlosinganydataorlosingany work,andensurethatwealwayshaveaccesstoourdatawith aslittledowntimeaspossible. Andthenaskediftheycancomeanyclosertothesegoals.
171
Don Jones
Backup2.0sDisasterRecoveryApproach
LetsquicklysummarizehowBackup2.0solutionsshould,inmyview,operate.Youinstall somekindofagentsoftwareonyourservers.Thisshimstheoperatingsystemsfile system,givingtheagentasortofpreviewofeverydiskblockchangethattheoperating systemiswritingtodisk.Theagentclonesthatdiskblockinmemory,andtransmitsitto acentralizedbackupserver.Thatserverkeepsadatabaseofallthediskblocks,tagging eachonewithatimestampandtheserveritoriginallycamefrom.Thebackupserveralso doessomecompression,deduplication,andothermagictohelpreducetheamountofdisk spaceitactuallyneedstoconsumeforthatdatabase(asIdiscussedintheprevious chapter). Theresultisthatyoucanpushabutton,andqueryfromthedatabaseallthediskblocks thatgowithacertainserveratacertainpointintime.Yourenotlimitedtothegranularity ofmaintenancewindowsandtapebackups;everychangeiscaughtnearlyinrealtime,and youcanchoosetorestoretoanyparticularmoment. Allofthosediskblocksliveonfast,randomaccessdiskstorageinrealityprobablyaRAID cabineteitherdirectlyattachedtothebackupserverorperhapslivingonaStorageArea Network(SAN).Iimaginethatyoudkeepsomespecifiedperiodworthofdiskblocksin thatfashionsayseveralweeks.Youdperiodicallydumpolderdiskblocksonesthathad sincebeenoverwrittenbyneweronestoatapeforoffsitestorageorarchival,andyoud deletethosediskblocksfromthediskofthebackupserver.Ifranklycantimaginewanting torecoveracompleteservertomorethanafewhoursinthepast,letaloneentiredaysor weeks,butImsuretheresabusinessscenarioouttheresomewherethatwouldjustify retainingtheabilitytorestorethatfarinthepast.
BareMetalDisasterRecovery
Letsagreethattherebuildtheservermanuallyapproachistootimeconsumingandstick withbaremetalrecovery.InaBackup2.0world,allofyourdataissittingonnice,fast, randomaccessdisks.Soyourtimetogetthedataoffoftapeisexactlyzero:Youjustneed togetthedatafromyourbackupserversdiskstothediskofaphysicalorvirtualserver.In otherwords,abaremetalrecoveryisreallyjustabig,fancyfilecopy.Now,afast 430GB/hourtapedriveisanicethingtohave,butthatsonlyabout7GBperminute.A 10GbEthernetnetwork,ontheotherhand,ismuchfasterand10GbEthernetisntatall uncommonintodaysdatacenters,havingbeenintroducedinlate2002.Infact,both40Gb and100GbEthernetarecurrentlyindraftspecifications.Butamature10GBEthernet networkmoves10gigabitspersecond.Thereareeightbitsinabyte,sothats1.25gigabytes persecond.Inaminute,thats75GBabouttentimesasfastasasuperfasttapedrive. Evenassumingthenetworkisonly80%efficient,yourestillat60GBperminutes,orabit morethan8timesfasterthanaveryfasttapedrive.Ifyouneededtorestore420GBtoa server,itwouldtakeaboutsevenminutesacrossanetworklikethat.Sevenminutesto restoreacompleteserver,eithertophysicalhardwareortoavirtualmachine.When40Gb Ethernetisavailable,justarestorewouldtakeundertwominutes.Twominutes.Canyou imaginethatconversation?Theserverswhat?Down?Oh,canIputyouonholdforlike twominutes?<musicplays>Tryitnow.
172
Don Jones
ANoteonMath Bytheway,IrealizethatthenumbersImtossingaroundhererepresenta perfectworld,andtheyreperhapsglossingoversomeassumptionslike assumingyouhavebuiltadisksubsystemforyourbackupserverthatcan pulldataoffthedriveat60GBaminute,andthatyourbackupservers operatingsystemandsoftwarecanspitdataontothenetworkthatfast.The pointisreallytoillustratethevastperformancegulfbetweentapedrivesand disk+networkdatatransmission;obviously,oneofthethingsyoushould evaluateasyoustartlookingatBackup2.0solutionsishowfasttheycan performinrealworldconditions. ThepointisthatBackup2.0srecoveryscenariolooksbetterbecauseitsbackupscenariois better.Bygrabbingdataasitchanges,wegetacompleteuptotheminutebackupforallof ourservers.Wecanthensendthatdatafromanygivenpointintime,nolessbacktoa server,ortoanalternateserver,wheneverwewantto,inlesstimethanittakestomakea propercupofcoffee.
VirtualizationasaDisasterRecoveryStrategy
Onceyouveturnedbaremetalrecoveryintosomethingapproximatingagiganticfilecopy operation,youcanreallystarttogetcreativewithyourrecoveryoptions. Forexample,whynotjustengineerapad,orextracapacity,intoallofyourvirtualization hostservers,sothateachonecouldsupportafewmorevirtualmachinesthanthey normallyran?Ifaphysicalserverdies,youcouldjustspinupanewvirtualmachine whichtakesafewsecondsandrestorethefailedservertothatvirtualmachine.You mightnotgetamazingperformancethatway,dependingonwhatthefailedserverdid (someapplicationsjustdontrunwellonvirtualmachines),butyoudbeupandrunningin areducedstateuntiltheserversoriginalhardwarecouldberepaired.Thenyoudjust restoretheserverbacktothatoriginalhardwareduringamaintenancewindow,andyoud besettogo. Virtualization,combinedwithBackup2.0techniques,offersapracticallyendlessarrayof recoveryscenarios.Recoverphysicalmachinestovirtualones.Recovervirtualmachinesto differentvirtualizationhosts.Recovervirtualmachinestophysicalmachines,ifneeded. Yougetatonofflexibility,anditcanallbedonequickly,providedyouvebuiltan infrastructuredesignedforthiskindofoperation.
RestoringApplicationstoSomeplaceElse
Butwhatifyourentiredatacenterisaffected?HowcanBackup2.0servethen?Thefirst keyisgettingthatbackupdataoffsite,becauseifyouloseyourdatacenterthenyoure losingyourbackupserverandallitscontents,too.Tapeisobviouslyonewaytogetthat backupdatasafelyoffsite,butitsrevertingtoaBackup1.0,snapshotstyleapproach, meaningyoullalwayshavedataatrisk.Thatmightbeokayforyourorganization,andits certainlyaneconomicalapproach.Ioutlinedthisideainthepreviouschapter;Figure10.2 isareminderofwhatthismightlooklike.
173
Don Jones
Figure10.2:Movingbackupdataoffsiteviatape. Therecoveryplanhereistorunoutandgetyourtapes,andthentakeittowhereveryou plantoexecuteyourdatacenterrecovery. ColdRecovery ThatscenarioisessentiallythesameasthecoldsiterecoverythatIoutlinedearlier. Yourestillgoingtowaitfordatatostreamofftapeasyourecoveryourbackupserver,and thenyoullbecopyingmultiplesetsofdataacrossanetworktogetyourserversupand runningagain. Aslightmodificationofthisapproachistonotbackupyourbackupservertotape,but rathertoexport,fromyourbackupserver,diskimagesoftheserversitsbeenbackingup. Thatway,thetapecontainsimagesthatcouldinstantlybecomevirtualmachines,orbe usedtorecovertoaphysicalmachine,assoonasthedatagetsoffofthetape.Figure10.3 illustratesthisapproach.
174
Don Jones
Figure10.3:Exportingdiskimagestotapeforfasterrecovery. Imnottryingtopresentthisasanidealsolution,becauseIdontthinkitis.Itisan economicalsolution,though,anditwillworkwiththewidestpossiblevarietyofoffsite recoveryfacilities.Basically,iftheycanprovideyouwithatapedriveandavirtualization host,youreinbusiness. WarmRecovery Withthisapproach,youreplicateyourbackupdataoffsiteoveranetworkconnection, ratherthanontape.Therearetwomainadvantageshere: Youroffsitedataismuchmoreuptodatethanitwouldbeifyouusedtapes. Recoveryisfasterbecauseyouwontnecessarilyberestoringfromtape;youllbe recoveringfromdiskbasedstorageattherecoveryfacility. Figure10.4showshowthismightwork.
175
Don Jones
Figure10.4:Replicatingbackupdataoffsite. Therearesomeprettyobviousconcernswiththisapproach,firstandforemostofwhichis probablyWANbandwidth.Youreobviouslygoingtoneedalot,butyoucanmitigateand managethisabit.Forexample: Compressionanddeduplicationwillreducehowmuchhastogettransmittedby asmuchas80%,accordingtosomevendorclaims. ThrottlingcanreducetheimpactonproductionWANusage.Letthebackupserver transmitusingleftoverbandwidth. Thebackupservermightnotreplicateeverydiskblock.Forexample,alotofserver operationswillcontinuouslywritethesamediskblockmanytimesinsuccession. Thebackupservermightwaitforanidleperiod,andthentransmitthelastversion ofaparticulardiskblock.Youlllosesomegranularityofrecoverythatway,butyou mightnotcarewhenitcomestooffsiterecovery. IusedInternetinmymodel,butyoumightnotactuallyusetheInternet.Dedicated WANbandwidthtotherecoveryfacilitywouldprovideadedicatedpathforthedata tobeuploaded,althoughyoudobviouslyhavetobeabletojustifythecostofthat dedicatedbandwidth.
176
Don Jones
Letssayyougenerate100GBofbackupdataaday.Ifyoureducethatby60%for compressionanddeduplication(eventhoughmanyvendorsclaim80%),thats40GBto transmit.Ifjust5%ofthatwasrepeateddiskblocks,youcouldpotentiallyhavetojust send38GB.Assumethatsspreadevenly(oratleasttransmittedevenly)overa24hour periodandyouget1.6GBperhour,meaningyoudneedanuplinkspeedofabout3.5Mbps, whichisroughlyacoupleofT1linesintheUS(orevenanSynchronousDSLlineinsome areas).Notcheap,certainlyabout$6500/yearinmanyUScitiesusingdualSDSLlines butclearlynotimpossible,andforsomebusinessesitwouldbeworththeprice.Think aboutit:YoucouldcallyourrecoveryfacilityandletthemknowtoactivateMission Recovery.Theircopyofthebackupdatacouldimmediatelystartstreamingtovirtual machines,andinafewminutesyourmostcriticalserverswouldbeonlineatanoffsite facility,withperhapsafewhoursofdatalossbeingyourmaximumrisk.Youcanengineer thiskindofsolution. Note Exactlyhowyoureplicatethisdatadependsalotonthesolutionstackyouve assembledforbackups.Somebackupsolutionsmighthavenativesupportfor thiskindofoffsitereplication,includingbandwidththrottlingandother WANfriendlyfeatures.Inothercases,youmightneedaseparatesolutionto handlethedatareplication.Youllneedtoconsultwithyouroffsiterecovery vendortodeterminewhattechnologiestheycanworkwith,aswell. HotRecovery Hotrecoverygoesastepbeyondwarmrecovery,asthenameimplies.Ratherthanmerely storingyourbackupdataattheoffsitefacility,theyrerestoringthatdatatophysicalor (moreoften)virtualmachinesasyoutransmitittothem.Thatmeansrecoveryisjusta matterofstartingthosemachinesyoudontevenneedtowaitforarecoveryoperationto complete,becauseitsbeingdoneconstantly.Figure10.5illustratesthisidea.
177
Don Jones
Figure10.5:Ahotrecoverymodel. Thisisobviouslyaprettyexpensivechoice.Youlllikelyneedmoreupstreambandwidthto therecoveryfacilitytomakethiswork,andtherecoveryfacilityisobviouslygoingtowant tochargeabitmoretomaintainthiskindofarrangement.However,forsomeservicesin theirdatacenter,somecompaniesmayfindthepricejustified. Note Serviceisreallyanaccuratetermhere.Thingslikeemail,particular applications,andsoforththeseareallservicesthatyourdatacenter deliverstoyourusers.Someservicesmaybeworthyofexpensive precautionslikeahotrecoverymodel(althoughthatbecomesalotmore feasibleifyourcompanyalreadyhasitsownmultipledatacenters);other servicesmightwarrantwarmrecovery.Someservicesmightnotbepartof yourdisasterrecoveryplanatall.Italldependsonyourexactneeds. Today,mosthotrecoveryvendorsutilizevirtualization,whichhasmarkedlyloweredthe costofsuchservicesandputthemwithinreachofamuchbroaderrangeofbusinesses.
178
Don Jones
Note Backup2.0isnttheonlywaytoachieveahotsiterecoverymodel. Dependingontheapplicationsandservices,youmaybeabletouse geographicallydispersedfailoverclusteringandothertechniques.Thoseare beyondthescopeofthisbook,butIdowanttomakesureyoureawarethat therearemanyotheroptions,especiallyifyouroffsitefacilityissimply anotheroneofyourowndatacenters.
DisasterRecoveryasaMigrationStrategy
Believeitornot,theabilityforBackup2.0tohandlebaremetalrecoverycanmakea wonderfulmigrationtool.IfyourealreadybackingupyourphysicalserversusingBackup 2.0techniques,youcaneasilypusharecoverytoavirtualmachine.Figure10.6shows whatImtalkingabout.
Figure10.6:Disasterrecoveryasamigrationtechnique. Becausethebackupserverdoesntcarewhereitrestoresto,youcanenablesomepretty neatmigrationoptions: V2V,orVirtualtoVirtual:Migrateserversbetweendifferentvirtualizationplatforms orbetweenvirtualizationhostservers. P2V,orPhysicaltoVirtual:Migratephysicalcomputerstovirtualmachines. V2P,orVirtualtoPhysical:Migratevirtualmachinestophysicalmachinesgreat whenyourealizethataparticularserviceneedsawholecomputertoitself. P2P,orPhysicaltoPhysical:Migrateservicesfromonephysicalmachinetoanother, helpingeliminateolderhardwareandmovetonewerhardware.
179
Don Jones
YouwouldntneedtomessaroundwiththeP2Vtoolsprovidedbyvirtualizationvendors, whichsometimesrequirethesourceservertobetakenoffline.Instead,youcanbasethe migrationoffoftheuptotheminutebackupavailableonyourbackupserver.Infact,you canevenrepeatthemigrationasmanytimesasneeded,doingtrialmigrationsoverand overuntilyouresatisfied,becausethesourceserverdoesntneedtobeinvolved.
ComingUpNext
Wevecoveredquiteabitofgroundintheprecedingtenchapters.Itstimetocircleback andlookatsomeoftheoriginalproblemswitholdschoolbackups,andseewhatwemay havesolvedwitha2.0approach.Itsalsotimetolookmorepreciselyatwhatsinvolved inredesigningyourbusinessbackupstrategy,andoutliningamethodologyfor determiningtherealcostofaredesign.Weshouldalsolookatsomeofthemorehumanor, shallwesay,politicalfactorsinvolvedinaredesign.Thatsallcomingupinthenext chapter. Then,inthefinalchapterofthisbook,Imgoingtolookatsomerealworldstoriesof Backup2.0techniquesinaction.Hopefullyitllgiveyouafeelforthevarietyofwaysin whichcompanieslikeyoursareutilizingBackup2.0,andinspireyoutoinvestigateyour ownsolution.
180
Don Jones
Chapter11:RedesigningBackup:IsItReally WorthIt?
WhatsitgoingtocostyoutoimplementBackup2.0,andisitworththetimeandmoney? Obviously,Icantgiveyouspecificnumbersbecausethosenumberswilldependonexactly whatyourebackingup,howdistributedyournetworkis,andanumberofotherfactors. ButwhatIcandoisshowyouhowtocalculatethatcost,andtocalculatethecostofjust stayingwithyourexistingbackupinfrastructure.Youcandothemathandfigureout whethera2.0inspiredredesignisgoingtobebeneficialforyou.
LetsReview:WhatisBackup2.0?
Beforewedothat,letsquicklyreviewwhatBackup2.0isallabout.Asalways,were focusedonabusinessgoalhere,andwerenotconcerningourselveswithpasttechniques ortechnologies.Hereswhatwewant: Backupsshouldpreventusfromlosinganydataorlosinganywork,andensurethat wealwayshaveaccesstoourdatawithaslittledowntimeaspossible. Theproblemwitholdschoolbackuptechniquesisthattheyrelyprimarilyonpointintime snapshotsandonrelativelyslowtapebasedmediaforprimarybackupstorage.That meansrecoveryisalwaysslowerthanitshouldbe,andwealwayshavealotofdataatrisk. Backup2.0proposesthatweusecontinuousdataprotection.Inatypicalimplementation, youwouldinstallagentsoftwareonallyourservers,andtheywouldbackupyourdatatoa centralbackupserver.Figure11.1illustratesthisarrangement.Thebackupserverwould keepyourbackupinformationondisk,makingitmorereadilyavailable.Withafast10Gbps Ethernetnetworkconnection,youcouldrestoreanentireserverinafewminutesfar fasterthanyoucouldfromeventhefastesttapedrive.
181
Don Jones
Figure11.1:Backinguptoacentral,diskbasedbackupserver. Ratherthangrabbingsnapshotsoftheentireserverorevenentirefiles,theagentwould shimtheoperatingsystems(OSs)filesystem,enablingittoseeeverychangebeing writtentodiskatadiskblocklevel.Becausetheagentwouldbeaccessingthisdatabelow thelevelofafile,itwouldnthavetoworryaboutopenfilesoranyothercommonbackup restrictions;itcouldsimplycopyeverydiskblockchangeasitwasbeingwrittentodisk. Figure11.2showshowthismightwork.InStep1,anapplicationmakesachangetodisk perhapssavingarowtoadatabase,writinganewmessagetoamessagestore,or modifyingafile.InStep2,thosechangeddiskblocksarecapturedbythefilesystemshim, andinStep3,theyaretransmittedtoabackupserver,whichsavesthediskblocksina database.
182
Don Jones
Figure11.2:Backingupdiskblocks. Becausethediskblocksaretimestamped,thebackupservercouldreconstructthe originalserversentirediskatanygivenpointintime.Byexposingthatinformationasa mountablevolume,andbyprovidingutilitiesthatunderstandhowtonavigatedatabases suchasSQLServerorExchangedatabases,youcouldretrievesingleitemsfromanypoint intime.Figure11.3showshowauserinterfacemightexposethatkindoffunctionality.
183
Don Jones
Figure11.3:Recoveringsingleitemsfromthebackupdatabase. Toreducethesizeofthebackupdatabase,thebackupservercouldusefileorblocklevel deduplicationalongwithcompression.Forexample,asillustratedinFigure11.4,the servercoulddetectdiskblocksthatcontainedthesameinformationand,asshownin Figure11.5,removetheduplicatesandsimplyprovideanoteofwhatblocktheywerea duplicateof.Vendorsimplementingthistechnique,alongwithcompression,claimupto 80%reductionsindatasize.
Figure11.4:Detectingduplicatediskblocks
184
Don Jones
Figure11.5:anddeduplicatingthemtosavespace. Magnetictapewouldntnecessarilybeeliminated.Tapestillprovidesaconvenientwayto movedatasafelyoffsite.However,tapeswouldbeusedtobackupthebackupdatabase, ratherthanserversdirectly,asFigure11.6shows.Thiswouldeffectivelyeliminatebackup windows:Youcouldruntapebackupsalldaylong,ifyouwantedto,becauseyouwouldnt beimpactingyourservers.
Figure11.6:Movingbackupsoffsiteviatape.
185
Don Jones
However,becauseBackup2.0dealsindiskblocksthatarecapturedinnearrealtime,you couldalsostreamthosediskblocksoffsitetoasecondbackupserverbymeansofaWAN orInternetconnection.Offsitemightrefertoanotherfacilityofyourownortoa contracteddisasterrecoveryfacility.Figure11.7illustratesthistechnique;itcanbevery usefulincreatingawarmrecoverysitethatcanbeginrestoringkeyserverstovirtual machinesintheeventofatotaldisasterinyourmaindatacenter.
Figure11.7:ReplicatingbackupcontentoffsitebymeansofaWANorInternet connection. ThesearethebasiccapabilitiesthatBackup2.0offers,andthecoretechniquesthatituses. Itsdesignedtoprovidebothrapidsingleitemrecoveryaswellasspeedywholeserver recovery,givingyoutheflexibilityofrestoringentireserverstothesamehardware,to differenthardware,oreventoavirtualmachine.
186
Don Jones
LetsBeClear ThecapabilitiesIvewrittenaboutherearelargelywishfulthinkingonmy part,meaningthatImnotwritingaboutanyspecificvendororsolution. Thatsdeliberate:Idontthinkweshouldacquiretechnologybasedonwhats available.Rather,Ithinkweshoulddefinewhatwewant,thenpushthe marketplacetodeliver. Thereareabsolutelyvendorswhoofferatleastsome,ifnotall,ofthe capabilitiesIvewrittenabout.SomeofthemusethephraseBackup2.0and somedont.Thekeyforyouasatechnologypurchaseristofocusonthe capabilitiesyourbusinessneeds.Whenyoufindavendorwhocandeliver exactlythosecapabilitiesforthepriceyoufeeltheyreworth,youvefound yournewbackupsolution.
ABackup2.0ShoppingList
LetmequicklysummarizethekeycapabilitiesofaBackup2.0solution: Continuousdataprotectionisthekeyelement,leavinglittleornodataatriskin theeventofafailureofanykind.Therearemanywaysinwhichtoimplementthis, butmypreferenceistofocusonstreamingchangeddiskblocks.Doingso,rather thangrabbingentirefiles,bypassestheneedtoworryaboutfilesthatareconstantly openandprovidesmoregranularrecoverycapabilities. Mountablerecoveryimagesareanotherkeyelement.Youshouldessentiallybe abletobrowseyourbackupsasiftheywereafilesystem.Dependingonyour environment,thismightincludetheabilitytomountspecifickindsofdatabases suchasSQLServer,SharePoint,orExchangesothatyoucanrecoversingleitems withouthavingtorestoretheentiredatabase. Variablerecoverytargetsoffertheabilitytorecoveraservertodissimilar hardwareoreventoavirtualmachine.Thiscapabilityprovidesmaximumflexibility forrecoveryscenarios. Datacompressionanddeduplicationreducetheamountofdiskspaceconsumed bythebackupsolution. Tieredstorageallowsyoutotransferorcopybackupdatatotapeforoffsite storage. Replicationallowsyoutotransferorcopybackupdatatoanotherbackupserver forredundancy,offsitestorage,centralization,orothergoals.
Yourbackupsolutionhastoprovidethesecapabilities,plusanyothersthatyouve identifiedasimportanttoyourbusiness.Redesigningyourbackupinfrastructuremeans acquiringacompliantsolution,thenredesigningifnecessaryyourdatacenterto supportthatsolution.
187
Don Jones
WhatsInvolvedinaRedesign?
Ifyourelikemostcompanies,youprobablyalreadyhaveabackupplaninplace.Thatmost likelyinvolvesoneofthreebasicscenarios: Backuptapedriveshangingdirectlyoffservers,whichbackuptothatdrive Backuptapelibrarieshangingoffadedicatedbackupserver,whichusessoftware agentstograbsnapshotsandstreamthemtotheserver,whichthenstreamsthe datatotape Acombinationofthesetwo,asFigure11.8shows
Thethirdoptionisactuallyfairlycommon;companiesstartwiththecentralizedtape backupscheme,thenattachtapedrivesdirectlytoafewkeyserverstoenablemore backupstotakeplacewithinagivenbackupwindow.
Figure11.8:Typicaltapebasedbackup. MyfirstITjobwasasanAS/400operator,andmymainjobwasbacking upthesystemeverynight(tothisday,Ihatenightshifts).Iremembermy elationwhenwepurchasedanewtapebackupdrivethatwascapableof backingupthewholesysteminhalfashift,usingjustonerackoftapes.It replacedasystemthattookalmostthewholenightandtworacksof tapes.ThatswhenIfirststartedtothink,Wow,ourexpectationsfor backupandrecoveryareawfullylowifthisiswhatexcitesme.Thatwas thebeginningoftheBackup2.0conceptforme:Backupsthatactually meetbusinessrequirementsratherthanfitwithintechnologylimitations.
188
Don Jones
Itsnotevenunusualtohaveasortofhybridmodel,wherethebackupservercollects backupinformationondisk,thenwritesittotape,thusprovidingmorebackupcapacityin ashortertimeframe.Thebackupservertypicallydoesntstoremorethanadayortwoof snapshots,butthatsoftenenoughtoproviderecoverytothemostrecentimage. RedesigningthisschemetofittheBackup2.0modelisntdifficult.Youreobviouslygoingto needtoacquireaBackup2.0continuousdataprotectionsolution;feedcontinuousdata protectiontoyourfavoritesearchengineandseewhatcomesup.Youcanalsojusttry searchingonBackup2.0,asseveralvendorsarestartingtousethattermtodescribetheir products. Caution PleasedontassumethatavendorsuseofthetermBackup2.0guarantees thattheydelivereverysinglecapabilityIvewrittenabout.Icantcontrolthe marketingdepartmentsofeveryvendoroutthere;usethisbooktodevelop yourownlistofcriteriaandrequirements,andrevieweachsolutionthat youreconsideringaccordingly. Youwillhavetotouchallyourservers,orusesomecentralsoftwaredeployment mechanism:YouregoingtowanttouninstallordisableyouroldBackup1.0styleagents oneachserver,andinstallagentsforyournewBackup2.0solution.Inalllikelihood,you cancontinueusingyourexistingbackupserver,withitsattachedtapelibrary.Youllwant tochangeanyserversthathavedirectlyattachedtapedrivesandhavetheminstead streamdiskblockstoyourbackupserver.Figure11.9showsthenewarrangement;at worst,youmightneedtoadddiskstoragetoyourbackupserver. Note Frankly,determiningtheamountofdiskstorageyourcentralbackup serverwillneedisprobablythebiggestchallengeinmovingtoBackup2.0. Everysinglevendorisdifferentinthisspace;workwiththemtodetermine somediskspaceestimates.Justrememberthatyourplanwillprobablybeto movebackupdatatotapeonsomescheduledbasis,andrememberthatyour backupsystemisntnecessarilysupposedtobeadataarchivalsystem.You dontneedtokeepeverythingforever;youneedtokeepenoughtobring yourbusinessbackonlineintheeventofadisasterortorecovercorrupted oraccidentallydeletedfiles.
189
Don Jones
Figure11.9:ConvertingtoBackup2.0. AskyourBackup2.0solutionvendoriftheyoffercompressionanddeduplication,andfor somerealisticexpectationsonhowmuchspacethatwillsave.Letssayits60%.Dosome analysistoseehowmuchdiskspaceactuallychangesonyourserversonadailybasis. Decidehowmanydaysintothepastyouwanttokeeponlive,diskbasedstorage.Thendo themath: Multiplyyourdailydiskchangebythenumberofdaysyouwanttokeep Multiplythatresultbytheaveragediskreductionnumber(say,.7for70%)
Adda10%padforgrowthandmismeasurement,andyoushouldhaveanideaofhow muchdiskspaceyourbackupserverwillneed. Youalsoneedtolookatthenetworkyourbackupserverwillberunningover.Youmight havealreadybuiltadedicatednetworkinyourdatacenterforbackuptraffic;ifso,itwill probablybejustfineinthenewBackup2.0world.Ifnot,youmightwanttoconsiderdoing soandconsiderafast10GbpsEthernetnetworkforthatpurpose.Remember,thisisthe bigchangefromBackup1.0:Yourenotusingthatbackupnetworksolelyatnightduring yourmaintenancewindow;youreusingitcontinually,soitneedstobefast,andideallyit needstobededicated.
190
Don Jones
Note Yourinvestmentinafastbackupnetworkisnottoenablefastbackups, contraryasthatmightsound.Theinvestmentistoenablefasterrecovery. Remember,withcontinuousdataprotection,yourbackupsarealwaysupto date;thegoalistobeabletogetthemoffthebackupsystemandontoyour actualserversasquicklyaspossible.Thatswhya10GbpsEthernetnetwork dedicatedforbackupsisagreatidea.Itsnotalwaysanecessity,ofcourse,but itsgreatifyoucanaffordit. Addupyourcosts: YourBackup2.0solution,whichislikelygoingtobelicensedbasedonthenumber ofserversitwillbackup Newdiskstorageforyourbackupserver,ifneededThiscanbepricey;ideally,you wantastorageareanetwork(SAN)forthispurpose,althoughiSCSImakesaSAN lessexpensivethanolderfibrechannelSANtechnology Ifyouneedtoimplementadedicatedbackupnetwork,factorinthepriceofnetwork cards(ifyourserversdonthaveunusedEthernetports),cabling,switches,and otherinfrastructureequipment
Thosearenumbersyoushouldbeabletocomeupwithprettyreadily,givenaquotefroma fewvendorsandperhapsatriptoasearchenginetocheckpricesonthenetworking equipmentordiskstorage.Next,youneedtofigureoutwhatitcostsyoutodonothing. Why?Idontwantyouchangingyourwholebackupinfrastructureonmysayso(Iknow youwouldntanyway).Itshouldbeabusinessdecision,basedonwhatyoullgainfromthe investment.
CalculatingtheRealCostofYourGrandfathersBackupStrategy
Determiningthecostofyourgrandfathersbackupstrategy,unfortunately,isatough thingtoworkoutbecauseyouredealingwithrisknumbers,nothardandfastpricing.Ask anycompanyHowmuchmoneydoweloseperhourifsuchandsuchaserverisoffline andyouprobablywontgetaveryaccurateanswer;mostbusinessessimplydontknow. Butletslookattherisksassociatedwithdoingnothing. Inasimplerestorescenario,costsareactuallyabiteasiertofigureout.Letssaysomeone asksyoutorecoverafilefromaweekago,thatyouhavethenecessarytapesinhand,and thattherecoveryinformationisonlyonthosetapesitsnotsittingondisksomeplace. Howlong,onaverage,willittakesomeonetocompletelyrestorethatfile?Figurethatout, andthenfigureouthowmuchboththatpersonandthepersonwaitingonthefilearebeing paidforthatperiodoftime.Now,multiplybythenumberoftimesyouhavetodothisina year,andthatswhatBackup1.0costs.
191
Don Jones
Note Ifyoudonthaveawayoftrackinghowoftenthisisdoneinayear,you shouldgetsomekindofHelpdeskticketingsoftwaresothatyoucantrackit. Thatshouldalsoletyoucomeupwithanaveragetimetorestore.Ifyou arenttrackingthatinformation,itbecomesverydifficulttomakeacasefor improvingthesituation. Illgiveyouanexamplefrommyownpast:Ittookus,onaverage,30 minutestorecoverasinglefile.Wehadtopulltapesoutofoursafe,load them,locatethefileinthebackupsoftwaresindex,thenletitspinthe tapetotherightlocationtoretrievethedata.Frankly,30minuteson averageisprettygood.Ouradministratormadeabout$60,000ayearat thetime,andthefolksintheofficeprobablymadeasimilaramount,on average.Irecalloneyearwherewedidatleastthreeoftheseoperations aweeksoabout150ayear.Thereareabout1,920workinghoursina year,andthesebackupoperationsconsumedabout75ofthem.That meansthiskindofrecoveryoperationcostusabout$4700peryear.I knowthatsnotahugenumberasingleserverusuallycostsmorebut itaddsup.Andwewerearelativelysmallcompany;$5kayearin overheadwasahealthypercentageofourITbudget. Note FeatureslikeWindowsPreviousVersions(VolumeShadowCopy)were implementedspecificallytohelpreducethatcost.Bygivingusersaself servicemechanismandbyusingfasterdiskbasedstorageforrecent backups,youcouldprobablyreducethatcostbyalot.ButPreviousVersions hassomesignificantlimitations,includingitssnapshotbasedmethodology andthelimiteddiskspacemostcompaniescangiveit. Calculatingthecostofanentireserverdisasterorworse,adatacenterdisasteris obviouslymuchmorecomplicated.Howmanyusersareimpacted?Howmuchoftheir productivityislost?Arecustomersdirectlyimpacted?Ifso,howmuchinsaleswillyou misswhiletheserverisdown? Consideradatabaseserverthatservesasthebackendforanecommerceapplication.If thedatabaseislostthroughcorruptionorfailure,oryouneedtorestoretheentireserver forsomereason,howmuchinsalesdoesthecompanyloseperhour?Inmostcases,a singlehouroflostsalescanjustifyeventhemostexpensivebackupinfrastructureredesign. YoualsohavetoconsidersomeofthemajortechnicalandbusinessadvantagesofBackup 2.0.Thesecantalwaysbemeasureddirectlyinmonetaryterms,buttheydoofferflexibility andimprovementsthatabusinesscanappreciate.
192
Don Jones
WanttoknowwhatIliketodo?Sincethiswholehowmuchdoesafailurereallycostus questionissohardtoreallynaildown,Iliketohavealittleconferencewiththecompany executives.Howmuchareyouwillingtospend,Illask,tomakeitsothatIcanhaveyour datacenterbackonlinewithinafewhoursofitburningdown?Thatkindoftellsyouhow muchtheshareholdersarewillingtospendoncontinuousdataprotection.Thesecond questionis,Now,whenIbringyoubackup,howmuchworkinthemiddleofadisaster, mindyouareyouwillingtoredoafterIgetthesystembackonline?Howmuchmoney areyouwillingtospendtomakethatanswernone? Iveoftenbeenshockedbytheanswers.Ionceestimatedthataconsultingclientofmine wouldlose$10,000perhouriftheyweretotallyoffline.IputtogetheraBackup2.0style recoverysolutionthatwouldhavecostabout$30,000.Easymath,Ithought.Iaskedthose twoquestionsoftheCEO,andhesaid,Idspend$250kayeartogetbackonlinewithinan hour,andanother$250kayeartoensureIdidntlosemorethan10minutesofworkwhen wecamebackonline.Well,then.Iwasclearlyunderpricingmyself!Mindyou,hedidnt haveanyhardnumbersthatwasjustagutinstinctfromthepersonwhohadtoanswerto theownersandsometimes,thatsabetternumberthanyoumightthink.
TechnicalandBusinessAdvantagesofBackup2.0
AsIvewritten,sometimesitcanbedifficulttoreallyquantify,indollars,thecost associatedwithnotmovingtoaBackup2.0styleapproach.Whatcansometimeshelpisto understandsomeofthespecificbusinessandtechnologyadvantagesofthisnewapproach sothatyoucanperhapsquantifythoseadvantages.Doingsocanhelpyoujustifythecostof aredesignintermsofbenefitgainedratherthanriskavoided.
WholeServers
Backup2.0,atitsheart,offerstheabilitytofinallygrabasolidbackupofeverysingleserver inyourenvironment,enablingyoutorecoveranythingfromawholeservertoasinglefile, toaspecificpointoftime,withinseconds.Literallyseconds,especiallywithindividualfiles orgroupsoffiles;perhapsminutes,dependingontheinfrastructureyoubuild,forwhole servers. Andwhenitcomestowholeserverrecovery,Backup2.0supportsavastarrayofnew options:Recovertoavirtualdatacenter,eitheronsiteoroffsite.Recovertodissimilar hardware.Recovertothesamehardware.Whateveryouwantyoucanflextheseoptions asneededforanyparticularscenario.
193
Don Jones
Exchange
Backup2.0offersafarsuperiorapproachforExchangeServer,whichisperhapsoneofthe mostmissioncriticalpartsofanybusinessinfrastructurethesedays.Recoverindividual messageswithouthavingtorestoreanentiredatabase,andwithouthavingtostreamthose individualmessagesoutyourdatabaseoneatatimethewayBackup1.0does.Backup2.0 givesyoutheabilitytosearch,find,andrecovermessageswithoutimpactingthe productionExchangeServer,whilestillgivingyoutheabilitytorecoverentireserverswith afewbuttonclickseithertophysicalorvirtualhardware,givingyoutheflexibilityto quicklycomebackonlineafteralmostanykindoffailure.
SQLServer
SQLServerhasalwaysbeenabiteasiertodealwithintermsofrecoveringdatabases,but Backup1.0still(usually)requiresyoutorestoreanentiredatabasetoaSQLServer someplacetimeconsumingandpotentiallyimpactfulonproductionservers.Backup2.0 shouldletyouaccessbackedupdatabasesdowntoindividualtables,evenwithout actuallyrestoringthedatabasetoaproductionserver.AswithExchange,thisgranular, lowimpactabilitytorecoveranythingfromasingletabletoanentireserveroffersyou flexibility,speed,andefficiency.
SharePoint
AsSharePointgrowsmoreandmoreembeddedinourorganizationstakingarolenot unlikeExchangesintermsofitscriticalitytoourbusinessestheabilitytogranularly recoverspecificitems,orrestoreanentireSharePointserverorfarm,becomesevermore important.Backup2.0canmeetthechallenge,providedyouselectasolutionthats specificallydesignedtosupportthiskindofSharePointbackupandrecovery.
Virtualization
Finally,Backup2.0isperhapsthefirsttimewevebeenabletoproperlyaddressour growingvirtualinfrastructure.Treatvirtualmachinesasyouwouldarealone,withlow overhead,blocklevelbackupstoacentralizedserver.Youreessentiallyabletotake virtualizationoutofthemix,treatingvirtualserversexactlythesameasyouwould physicalones.
IsaRedesignThatExpensive?
IvefoundthatredesigningforBackup2.0istypicallynotthatexpensive.Letstakeaworst casescenario,whichinmyconsultingworktypicallymeanshavingto AcquireaBackup2.0softwaresolutionoftenthemostexpensivepartofthe proposal Implementabackendbackupnetworkinthedatacentertocarrythebackup solutionstraffic Perhapsupgradeoracquiretapelibrariestoprovideasecondtierofbackupstorage
194
Don Jones
Inatypicalsmalltomidsizebusiness,thesecostsareoftensmallenoughtobeconsidered incremental.Itsimpossibleformetoprovideaccuratepricequotesbecauseeverybusiness willbeslightlydifferent,butrestassuredthatthecostistypicallyinlinewith implementingamoretraditionalbackupsolution.Inotherwords,figureoutwhatyour existingBackup1.0solutioncostsyou,andyoureprobablylookingatsimilarcoststo implementBackup2.0.Oftenless,asyoullusuallybeabletoreuseyourexistingbackup server(s)andtapedrive(s),andifyoualreadyhaveabackendnetworkforbackuptraffic, youllbeabletoreusethataswell.Inabestcasescenario,infact,yourereallyjust swappingyourcurrentbackupsoftwarefornewbackupsoftware,andthatsrarelya massiveinvestment. PerhapsthebiggestconcernIrunacrossiswiththecompanysCFObecausecomputer softwareisoftendepreciatedoveralongerperiodoftimethanevencomputerhardware. Inotherwords,ifyourCFOdoesntfeelhesdepreciatedyouroldbackupsoftware,there maybesomeresistancetobuyingnewsoftware.Thatswheretheprosandconssideof theargumentcomesin:Youmightneedtoidentifysufficientbenefitthatlosingthe depreciationontheoldsoftwareisjustified.Alsobesuretofactorinanysoftware maintenancefeesthatyoupayontheoldsoftwareandthatyoullpayonthenewsoftware; becausethosemaintenancefeesarerecurring,theyretypicallynotdepreciatedatalland thosefeescansometimesconstitutethebulkofthecostofasoftwareproductoveragiven periodoftime.
TacklingtheOfficePolitics
Thefinalthingyoullprobablyhavetodealwitharetheinevitableofficepoliticsthat surroundanykindofmajorITchange.Thekeytosuccessinthisarenaistobeprepared withfactsandtoaskthateveryoneinvolvedtrytoavoidbecomingreligiousaboutthe issue. Itcanbedifficult:Wetechnologyfolksgetattachedtoourtoys,andwedontreallylike changeverymuch(althoughweenjoyinflictingitonotherswhenitssomethingwere excitedabout).Thegoalshouldbetohaveeveryoneagreetofocusonthebusinessneeds, andonhowthoseneedscanbebestachieved.
195
Don Jones
Ifindthatthereareusuallyjustafewcategoriesofoppositiontochangingacompanys backupstrategy.Executiveleveloppositionisusuallyfocusedoncost,oftenwiththeadded twistrelatedtothedepreciationofyourexistingsolutionwhichIvealreadywritten about.Executivesjustneedtounderstandthecost/benefitargumenttodecidewhetherthe purchaseisworthit.Thisbookshouldhelpyoubeabletoarticulatethebenefits,butdont argueforanewsolution.Instead,presentthebenefitsandthecostsinanunbiased, businesslikefashionyoullearnmorerespectwiththeexecutivesandhelpthemreally decideifBackup2.0isworthit. Middlemanagerialoppositionusuallycomesfromadesiretoavoidchange.Thats understandable,aschangeiswhatcausesproblemsinmostITenvironments.Thekeyhere istohaveasolidplanforimplementation,andtomakeitclearthatyoudontwantto proceedwithoutsuchaplaninplace.Thatmayinvolverunningoldandnewsolutionsin parallelinalimitedpilotforsomeperiodoftimesothatyoucanunderstandandmitigate oraddressanyproblemsthatarise.Workwithinwhateverchangemanagementframework yourcompanyusesfollowingthatprocesswillhelprelievemanagerialconcernsinmost cases. Theresoftenafeelingthatswappingbackupschemeswillbedisruptive,butitdoesntneed tobe.Backup2.0canlivesidebysidewithBackup1.0inaphased,gradualapproachthat avoidsdisruptionandofferstimeforfamiliarization. ITprofessionalleveloppositionthatis,oppositionfromthefolkswhowillimplementand operatetheBackup2.0solutionisusuallysimplefearordislikeofchange.Everyonewill havetolearnanewsetoftools,newtechniques,andnewwaysofthinking.Mitigatethat fearbyclearlycommunicatingthebenefitsofBackup2.0.Typically,Ifindthatmost administratorsarentthrilledwiththeoverheadandtediumoftheirexistingbackup solution,andthebenefitsofBackup2.0fromthatlevelonceclearlyexplainedhelps overcomeresistanceprettyquickly.
Conclusion:ThisIsNotYourFathersBackupPlan
Youknow,therealthingtorememberhereisthatBackup2.0isnew.ThatswhyIkeep callingit2.0,muchasIknowthatwillleadtothetermbeingoverusedbymarketingfolks (rememberWeb2.0?Thisisntthesame).Backup2.0isfocusedonawholenew approachtobackupandrecoveryanapproachthat,frankly,wasntevenpracticalor feasibletwodecadesagowhenourcurrentbackuppracticeswerecreated. Theproblemwitholdschoolbackupsissimplythattheyweredesignaroundthe constraintsoftechnology.Manyofthoseconstraintshavebeensolved,atthispoint,while Backup1.0hasmoreorlessfailedtokeepup.Backup2.0takesanentirelyfreshapproach, withanacknowledgementoftheproblemsofBackup1.0.Ratherthanboltingon solutionslikeopenfilemanagersandotherworkaroundstofixBackup1.0s problems,Backup2.0startsfromthegroundupwithanewwayofachievingouractual businessgoals.
196
Don Jones
Intheend,thatswhatBackup2.0isreallyallabout:meetingbusinessgoals.Ithinkyou shouldverycarefullyconsideryourexistingbackupsolutionsabilitytomeetyourreal businessgoals,andperhapsrestatethosegoalswithathisiswhatwedhaveinaperfect worldapproach.Thenstartevaluatingnewer,Backup2.0stylesolutionsanddetermine whethertheycantdoabetterjobofhelpingyoumeetthosenewer,morerealworldgoals thatyouvestated.Letthebusinessdrivethetechnology,nottheotherwayaround.
ComingUpNext
Well,thereyouhaveit:Backup2.0.Idontwanttojustleaveyouwithalistofcapabilities andasolidargumentfortheboss,though;Ialsowanttogiveyousomeconfidencethatthis stuffreallyworksintherealworld.Sointhefinalchapter,whichiscomingupnext,Im goingtosharesometalesfromthetrenchesreallifestoriesofadministratorswhoare livingwithBackup2.0rightnow.Itwontallbesunshineandrainbows,though:Youllread somegrittystoriesaboutimplementationsthatwerentwellthoughtout,andotherless thanperfectsituations.Why?Well,thoseareimportantlessonstolearn.Hopefully,youcan takethosestoriesandensurethatyourlifewithBackup2.0issmoothsailing.
197
Don Jones
Chapter12:TalesfromtheTrenches:My LifewithBackup2.0
Inthesecondchapterofthisbook,IsharedwithyousomeofthehorrorstoriesofBackup 1.0.Ididsoprimarilyasawayofhighlightinghowpoorlyourtraditionalbackup techniquesreallymeetourbusinessneeds.Inthischapter,Iwanttodotheopposite:share withyousomestoriesofBackup2.0,bothfrommyownexperienceandfromstoriesyou readershavesharedovertheyearlongproductionofthisbook.Nameshavebeenchanged toprotecttheinnocent,ofcourse,butIthinkyoullfindthesetobecompellingexamplesof howBackup2.0hasbeenapplied.Wherepossible,Illshareinformationaboutthe infrastructurethatgoeswiththesestoriessothatyoucanseesomeofthecreativeand innovativewaysBackup2.0isbeingusedinorganizationslikeyourown.
Backup1.0FiredOurAdministratorandBackup2.0PromotedMe
ImgoingtostartwithwhatIthinkmightbemyfavoritestory.Thiswassharedbyareader, althoughasImentioned,Immakingupnewnamesforeveryoneinvolved. Iworkforaprettylargeretailer,atthecorporateoffice.Wehaveabout 2000usersintheoffice,manyofwhomworkinourdistributioncenter. Wehaveabout40orsoservers,mostofwhicharefileservers.Wealso haveacoupleofExchangeservers,andofcourseActiveDirectory(AD). WealsohaveaSQLServerthathandlesourmaindatabaseofproducts, sales,andeverythingelse.Thatdatabaseisprettymuchthecenterofour universe. OurleadnetworkadministratorsnameisKevin.Kevinsetupouroriginal backupsystem,whichusessoftwareagentsoneachservertosend backupdatatoabackupserver,whichwritesthedatatotemporaryfiles ondiskandthendirectlytotape.WehavewhatIguessisapretty commontaperotationplan:Wemakeafullbackupofeverythingonthe weekendsandsendthatoffsite,thenkeepincrementalbackupsonsite fromeachnightoftheweek.
198
Don Jones
Ihadbeenpushingforacontinuousdataprotectionbackupsystem whatyouwouldcallBackup2.0,Iguess,foracoupleofmonths.Ihad finallygottenpermissiontobuyasystemforalimitedtrial,whichI decidedwouldinvolveourSQLServer(whichissoimportant)andoneof ourfileservers.KevinwasnotthrilledthatIwassteppingintohisturf withthebackupstuff,butIpushedaheadanddiditanyway.Herefused toturnoffhisoldbackupagents,whichwasfinebecauseIwasjusttrying todoasortofproofofconcept.Hewouldntevengivemespaceonhis backupserver,though,soIhadtomakedowithasomewhatolder machine.Iwasabletoputlotsofdisksintoit,though,andIwasableto setupadedicatedbackupnetworkforthefourserversinvolved.Im attachingadiagramofwhatthingslookedlikethen[seeFigure12.1].
Figure12.1:AddingapilotBackup2.0network.
199
Don Jones
Thingsseemedtogoprettywellatfirst.Iwasabletopullbackupsexactly likeyouhavedescribedinyourbook,andwedidseveraltestsof mountingthosebackupsaslivedatabaseswithouthavingtodoarestore. Kevinkeptcomplainingthatthesystemwasntasstableorasreliableas backuptapes.Itoldhimthatifhewouldhaveletmeputitonhisbackup server,wecouldhavecopiedthebackupdatatotapeeverynightor weekorwhatever,buthestilldidntwanttocooperate. Thenithappened.Asprinklerheadbrokeinthedatacenteronenight (whoinstallswatersprinklersintheirdatacenter?)andwelostbothof theExchangeservers,theSQLServer,andoneofthefileservers fortunately,oneoftheonesIwasprotectingwiththenewsetup.Weare probablyluckytherewasntanelectricalfire!TheSQLServerwasactually clusteredandwelostallthepowersuppliesinthoseservers.Somuchfor acluster!Wewereabletocobbleenoughpartstogethertogetoneof theSQLServersonline,andoneoftheExchangeservers,buttheirdisks weremessedup.KevinandIwenttothebosstotrytocomeupwitha plan. ThishappenedonaThursdaynight,sowewerealmostaweekawayfrom ourlastfullbackup.KevinwasplanningtorestoretheExchange databasestotheoneserverhehadrunning.Myplanwastoinstall VMwareontheSQLServerboxIhadsaved,asitwasprettyhefty,andto bringSQLandthemissingfileserverbackonlineasvirtualmachines.The bossapprovedbothplans,andwesetouttodoit. SoKevinwaits3hoursforlastweekendsfullbackuptocomefromoff sitebeforehecangetstarted.IwasjustinstallingESX,whichtakesabout aminute,andIhadthevirtualmachinescreatedinanother10minutes orso.IusedourBackup2.0solutiontopushthemostrecentdiskimages fortheSQLServerandthefileservers,andwithinaboutanhourtotal, theywerebothupandrunningwithabsolutelynodatalossthatwecould detect.Iwasahero! OnceKevinfinallygotstarted,herealizedthathisTuesdaynight incrementalwasbadcorruptedtapeorsomething.Thatmakesittough torestoreanyoftheotherincrementalbecausetheyallstackup.Sowe werebasicallybacktoMondaynight3daysofwork,lost.Ofcourse,it tookhimmostofthedaytorealizethis,sowewoundupcallingtheboss athomeandbreakingthebadnews.HeWasNotHappy.
200
Don Jones
Well,thiswasntthefirsttimeoneofKevinsplanshadblownupinhis face,anditwasntthefirsttimesomeonehadhadabetterideaandbeen heldbackbyKevin.Bytheendoftheweek,Kevinwaslookingforanew job,andIhadhisjobandwasbusilysettinguptherestofmybackup network,movingmybackupsolutiontothebackupserver,andputting Backup2.0intofulluseinourdatacenter. Youhavetoloveastorywherethevillaingetshisjustdesserts(wellassumetheywere just)andtheherowinsthegirl.Orrather,winscontrolofthedatacenter. Actually,thisstorypointsoutareallyimportantpointaboutBackup2.0:Itdoesnthaveto immediatelysupplantyourexistingbackupscheme.Itcanlivequitehappilybesideit, coexistingforhoweverlongyouneedtocompletethetransition.Youcouldchooseto implementBackup2.0foryourmostcriticalservers,andcontinuebackingthemupvia youroldsystem,providingabeltandsuspendersapproachuntilyourecomfortableand confidentwithBackup2.0.KeepinmindthatBackup2.0makesitaloteasiertopractice fullserverrecovery,evenenablingyoutorecoverdiskimagestoblankvirtualmachinesin aslittleasafewminutes,insomecases.Thatmeansyoucanpracticeyourrecoveryskills, tweakyourBackup2.0installation,thendecommissionyouroldbackupscheme.
TheCureforCrackberryWithdrawal
Thisonesfromareaderwhohadrespondedtomyprepublicationrequestforstories.He hadabriefsnippetincludedinChapter2ofthisbook,andhewrotebackintotellmehow itsbeengoingwiththeirnewBackup2.0solution. IwroteabouttheCrackberryWithdrawalthatyouincludedinChapter 2ofyourbookonBackup2.0.WevesinceimplementedaBackup2.0 stylecontinuousdataprotectionserverforourExchangeServers,anda coupleofmonthsago,oneofthemfailedagain.Actually,theservergot hitwithsomevirusthatcorruptedsomeoftheExchangedatabasesand wasstoppingtheBlackberryEnterpriseServersoftwarefromrunning. Theexecutiveswerereallyupsettheycanthandlebeingoff Blackberryformorethanafewminutes. Fortunately,thatsallthetimeittookus.Wefiguredoutthatthevirus hadhitduringtheeveningprior,andwewerentworriedaboutany emailsthathadbeensentorreceivedduringthattimebecausethey mightallcontaininfections.Sowejustrolledbacktheentireserverto about10minutesbeforethevirushadhit.Ittookmaybehalfanhourto dothat,verifythattheserverwasrunning,andgetupdatedvirus definitionsontheserversantivirussoftware(itgothitaboutanhour beforeitwasduetoupdatethoseautomatically).Backonline,and everyonehappy.Thankyou,Backup2.0!
201
Don Jones
ThisillustratesanothercreativeandusefulfeatureofBackup2.0:theabilitytorecovera servertoanypointintime.Youmightnotalwayswanttorecovertothemostrecent possiblediskimages,asinthiscasewhereyouwouldhavejustbeenrecoveringavirus. BecauseeachchangecapturedbytheBackup2.0systemistimestamped,youcanrestore toliterallyanysecondintimethatyoulike.Ifyoureabletodeterminewhensomething wentwrongasinthisreaderscaseyoucanrecovertojustbeforethen,andyouregood togo.
ExchangeFailureSuccess
AlsoinChapter2ofthisbook,IrepublishedastoryfromanExchangeadministratorwho hadexperiencedaserverfailureandhoursofdowntimethatevenMicrosoftcouldbarely figureout.Helost2daysofmail,andhisexecutivesfigureditcostthecompanyabout $50,000,sotheywantedtoputabettersolutioninplace.Illfinishthatstorynow: Asyoucanimagine,executivemanagementfigureditcostthecompany about$50K,sotheydefinitelywantedtoknowwhathadhappenedand howitcouldhavebeenpreventedandhowitwouldbepreventedfrom happeningagain.LetsjustsaywantedtoknowmeansthatifIdidnt haveagoodanswer,mynamewasgoingonthetopofthenextlayofflist. Iwasseriouslycommittedtofindingabetterrecoverysolution. Afterevaluatingseveralpotentialsolutionsofvaryingprice(from$199to $100K),IlikedtheBackup2.0solutionwefound.Itsablockbased imagingrecoverysolutionthatcapturestheentireExchangeServer environmentandsupportsrecoveryanythingfrombaremetaltoan individualemailmessageinjustafewclicks,simplifyingtheentire recoveryprocess.WhatwastotallycoolwastheExchangehealth checks,whichmadesurethedataIwasbackingupwasabsolutely mountable.BeforeImadetherecommendationtomyboss,Iwantedto besureIwasmakingtherightchoiceandcheckedoutafewoftheir customers,whosaidtheywerehappywiththesolution,too. InadditiontocapturingandvalidatingyourExchangedata,thetool employsauniqueinstantreplaycapabilitythatdramaticallyreduces volumerecoverytimesfromhourstominutesregardlessofthedataset sizebeingrecovered.Afterarollbackisinitiated,thevolumeandstorage groupsareautomaticallyandimmediatelymountedfromthebackup server,providinguserswithaccesstoemailduringtherecoveryprocess. ApparentlytheycallthefeatureLiveReplay;allIknowisitsavedmefrom beingDeadAdmin. Youcanreadthewholestoryathttp://appassure.ning.com/profiles/blogs/whatilearned onmyworst day?mkt_tok=3RkMMJWWfF9wsRow5%2FmYJoDpwmWGd5mht7VzDtPj1OY6hBssJJKtUg %3D%3D,ifyoureinterested. 202
Don Jones
ThishighlightsanotheradvantageoftheBackup2.0concept,whichisthatindividual solutionvendorsarefreetoaddreallycreativefeatures.TheLiveReplayfeaturethis readerwroteaboutisagoodexample:Ratherthanhavingtocopytheserversentiredisk image,thissolutioncanmounttheExchangedatabasesdirectlyfromthebackupserver, whicheliminatesalotoffilecopyingtimeandgetsyoubackonlinefaster.Performance mightbesomewhatdegraded,butyoureonline,andyoucanalwayshandlethemoretime consumingdiskcopyduringamaintenancewindowwhenyourusersarentsoanxiousto getintotheiremail.
VirtuallyOnline
ThisstoryillustratesagreatuseofnotonlyBackup2.0butalsothegrowingpopularityof cloudstorageandtheuseofthecloudasarecoverymechanism.Thisisdefinitely somethingthatrequiresaclevervendortobeyourbusinesspartner,butIthinkitsagreat tale. Ourcompanyhasonlyabout30servers,includingseveralSQLServers andacoupleofExchangeservers,butweusethemtoprocessalmost$30 millioninsaleseveryyear.Soasyoumightimagine,theseserversare veryimportanttouswithoutthem,welose$80,000adayor$8000an hour.Wereasmallcompany,soitsdifficultforustojustifythecostofa completeoffsiterecoveryfacility.Evenwith$80,000adayatrisk,those recoveryfacilitiesareexpensiveandtheycantakealongtimetobring onlineafterafailure. WestartedlookingatBackup2.0afterIreadthefirstfewchaptersof yourbook,andweworkedwithacoupleofValueAddedResellers(VARs) whohaveimplementedtheirownservicestosupplementtheBackup2.0 solutionthattheyresell.Oneofthoseservicesisanetworkappliance thatsitsonournetwork.Basically,wereplicateourbackupdatatoit,and itreplicatesthatdataoffsite;itusesbandwidththrottlinganddoesalot ofthereplicationatnight,soitdoesntkillourInternetconnection.Ill includeadiagramofhowwehavethissetup[seeFigure12.2].
203
Don Jones
Figure12.2:Offsitebackupdatareplication. Well,wefinallyputitalltothetestlastweek.Wedeliberatelyshutdown ourentiredatacenterhitthebigredSTOPbuttonthatsjustnextto thedoor,somethingIvealwayswantedtodoanddidalivetestofour disasterrecoveryplan. Whathappensiswecallthevendor,andtheyinitiateawarmrecovery ontheirend.Theyspinupabunchofvirtualmachinesintheirown virtualizationinfrastructure,thenrestoreourmostcriticalserversto thosevirtualmachines.Thentheybeginspinningupothervirtual machinesforwhatwecalloursecondtierservers.TheysetupaVirtual PrivateNetwork(VPN)fromtheirdatacentertoours,andourusers accessourserversfromacrossthatVPN.Becauseofthereplicationlag time,weweremissingafewhoursworthofdata,butitwasntmuch.In arealsituation,wemightevenbeabletogetmorerecentfilesandso forthfromthelocalbackupcopies,unlesssomethingreallybadhad happenedtothedatacenter.
204
Don Jones
Italltookjustacoupleofhoursbeforewewerebackonline.Payingfor virtualspaceisalotcheaperthantraditionaloffsiterecoveryfacilities, andourplanworkedperfectlyforus.Managementwasveryhappywe weregladtheVARwasabletohelpusputtogetheraplanlikethis. Itsalsoproventhemodeltous.Managementhasbeenconsidering openingasecondofficeelsewhereinthecountry,andwenowknowthat eachofourofficescouldactasawarmrecoverysitefortheother, simplybyreplicatingbackuptraffic.OtherthantheWANcosts,we wouldnthavetoinvestmuchinfrastructureinthat,whichisperfectfora smallercompanylikeours. ThisisreallyatestamenttothevalueofVARs,whocantakeagreatpieceofsoftwarelike aBackup2.0toolandsupplementitwithserviceslikethis,toreallycreateacomplete solutionthatsolvesallofacompanysbusinessrequirementseffectivelyandeconomically. And,asthisreaderpointsout,youcoulddothesamethingonyourownifyoualreadyhave morethanonesite.ConsiderFigure12.3.
Figure12.3:Creatingyourownoffsiterecovery.
205
Don Jones
ByreplicatingbackuptrafficacrossyourWAN(ortheInternet),youcankeepcopiesof yourdatasafelyoffsitewithoutthehassleofshufflingtapes.Ifatimecomeswhenyou needtorecoveryourcompletedatacenter,yourdataissafelyinanotherphysicallocation. UsingBackup2.0,youcanrestorethosediskimagestophysicalorvirtualmachines, bringingcriticalcomputersonlinequickly.Backup2.0scontinuousdataprotection techniquesmakeitideallysuitedforthistypeofenvironment.
Wait,WeGotThat?
Thisisafunstoryaboutthatfirst,franticweekinanewenvironmentandwhathappens whenthefolkswhoprecededyouwerethinkingclearly. SoIstartmynewjob,andIvebeenintheplacemaybeaweekwhen bothofourdomaincontrollersfail.Ofcourse,serverrecoveryisthemain partofmyjobdescription,soImsupposedtobethesuperhero.Thisisa smallenvironment,andthedomaincontrollerswereactuallyrunningin virtualmachinesanditwasthevirtualmachinehostthatfailed.Well,we getthethingbackonlinewith500usersyellingabouthowtheycouldnt doanywork,ofcourseandwefindthatwestillcantlogoncorrectly. Whatevertooktheserverofflinehadcorruptedsomethinginmemory, probablybecausethedomainwastoast. SoIgotothebossandtellhimthatweregoingtohavetorebuildthe domainfromscratch.Whataboutthebackups?hesays.Iexplainthat recoveringanentireforesttakesspecialtools,youhavetocallMicrosoft ConsultingServices,allthat.Thatswhenoneofmynewcoworkers fortunately,abitmoreinformedthanIwaspulledmeaside.Thisisnt reallyafullforestrecovery,hesaid.Wejustneedtobringthosetwo serversbackfromthepointintimetheywereat.Wedontuseanormal backupsolution,itcandothis. Well,whoknew.Iwasthinkingbackuptapes,butapparentlytheguywho hadmyjobbeforemeimplementedanewserverbasedsolution. Basically,Ihitadozenbuttonsorsoandbothdomaincontrollerswere backonline.Inlikehalfanhour.ThatswhenIstartedlookingintoBackup 2.0andfoundyourbook.Yourerightthisissomuchbetterthan backuptapesIcanteventellyou.
206
Don Jones
Itsanicefeelingwhenyoureabletomakeeverythingwork,especiallywhenyour environmenthastherighttoolsforthejob.Incorrespondingwiththisreader,Ifoundthat hisbiggestamazementcamefromthespeedwithwhichtherecoveryhappened.The companyhadinvestedinprettyfast,dedicatedEthernetconnectionsbetweentheirservers andthebackupserver,andthatbandwidthmadeallthedifference.Hetellsmenowthathe thinkshecouldactuallypulloffthesamerecoveryalotfasternowthatheknowswhat toolshehasavailableandhowtooperatethem.Interestingly,hisfirstexperiencewasdone almostentirelywithouttheproductsdocumentationhejustgotintotheUIandstarted working.Thatsasignofagreatpieceofsoftware,isntit?
StoptheSpindles!OrWhatever
WorkingintechnologyforasmanyyearsasIhave(andyouprobablyfeelthesameway), itstoughtoimaginehowusersenvisionitallworking.Storieslikethisareagreat reminderandagoodexampleofhowBackup2.0canhelpwiththelittledisasters,too. Heresafunnyoneyoucanuseinyourbookifyouwant.Igetacalllast weekfromoneofourusers,whosyellingintothephone.Ijustdeleted afilebyaccidentanditsgonecanyoustopwhateverisdeletingit beforeitstoolate?Hethoughttherewassomekindofprocessor somethingthatdeletedfilesonadelay.Sadly,weallknowthisisnot true.ButIdidnttellhimthat. WehaveoneofthoseBackup2.0solutionsyoutalkaboutinyourbook, anditgrabschangesastheyhappen.SoIjustpoppeditopen,mounted themostrecentpointintimeasadiskimage,andcopiedtheusersfile tomyworkstation.IemailedittohimwithanotethatIwasabletopull itoutbeforethefilewaspermanentlydestroyed.HethinksImahero. Thisisprobablyrepresentativeofwhatmostdisasterrecoverysituationslooklike.To youandme,notadisaster;totheuserwhodeletedthefile,itsatragedyinthemaking.We havetoplanforthewholeserverandentiredatacenterscenarios,butthefactisthatmost ofusspendmostofourrecoverytimegettingbackindividualfileshereandthere.
EnoughIsNeverEnoughUntilItIs
IrememberwhenIwasanAS/400operator,andweboughtanewtapedrivethatused massivelyhighercapacitytapes.Iwassoexcitedthatwecouldnotonlydothenightly incrementalbackupsusingasinglerackoftapes,inmostcaseswecoulddothemtojust oneortwotapes.Finally,Icouldkickbackandletthebackupsrunwithouthavingto remembertogochangetaperackshalfwaythrough.Backupswerealotfaster,too,which meantournightshiftusershadtodowithouttheirAS/400foralotlesstime.Ah,thegood oldBackup1.0days.Sometimes,though,buyingeverbettertapedrivesjustisnttheright answer.
207
Don Jones
WhenIfirstgottomycompany,wewereusingprettydecenttapedrives. Fullbackupsontheweekendsrequiredanoperatoronsite,though, becauseyoudhavetochangerackstwoorthreetimestogeteverything. Aboutayearago,weupgradedthedrives,andyoucoulddothewhole runonasinglerack.WedloadtherackonFriday,setthebackupsfor automatic,andenjoytheweekend.Nomoreweekendunpaid overtime! Well,wejustaddedabunchofserversandthatrackoftapesisnt enough.Ofcourse,wevedownsizedabitsinceayearago,andnowIm theonlyonewhosresponsibleforbackups.SoImexpectedtofloat halfofmyweekendIcantakeoffearlyonFriday,butIhavetocomein onSaturdayafternoonstoswaptaperacks. Toheckwiththat,Itoldmyboss.Iaskedifwecouldupgradetonewer drivesagain,andhesaidnowaytheseonesareonlyayearold.ThenI startedlookingatBackup2.0solutions,anddiggingintowhatsomeof themcost.IpresentedaplantouseaBackup2.0systeminstead,andto juststreamthebackupdatatotapeduringtheweek.Youknow,when wereallhereanyway.Well,itwasntthatexpensive,sohegotapproval. Nowwecanrecoverrightfromdiskbasedstorage.Tapesarejusta backupofthebackup,reallywehavetosendsomethingoffsitejustin case,sothatswhatthetapesarefor.AndIgetmywholeweekend. Again,agreatillustrationofrethinkingthings.Ofcourse,youcouldaccomplishmuchthe samethingwithadiskbasedBackup1.0systembutwithBackup2.0,ofcourse,youre gettingcontinuousdataprotection,pointintimerecovery,andsoforth,sowhynotgothat routewhilesavingyourweekend?
SQLServerSolution
HeresanotherfollowupfromChapter2.Imactuallygoingtoreprinttheoriginalstory, becauseitsshortandcompelling: IhavebackupsinthreeversionsofSQLServer,indevelopment, production,andtestingenvironments,onmultipleservers.Lotsof backups,inotherwordsprobably45to50instancestotal,withasmany as45databasesperinstance.
208
Don Jones
Wegrabfullbackupseverynight,andtransactionlogbackupsevery15 minutes.Soourmaximumdataloss,ifeverythinggoesright,is15 minutes.Wetestourbackupsbyrestoringthefullbackupstoourtest environment.Thathelpskeepthetestenvironmentcloselymatchedto theproductionenvironment,whichisvaluableforthesoftware developers,andhelpsusmakesurethosebackupsareworking.Weroll lastweeksbackupsintoadifferentfolderandgrabthosetotape. WegetfailurenotificationsthroughemailandSystemCenterOperations Manager,andwerelyonmanualinspectionofbackupjobsandfoldersto makesureeverythingworks. Rightnow,weretryingtoplaywithdifferentbackupschedulesandlook atusingdifferentialbackups,alltolessenthenetworktrafficandthe drivespaceoccupiedbythemanyfullbackups.However,weneedto keepourrecoverytimesshort,soweretestingtoseehowmuch overheadthedifferentialsaddtorecoverytimes. Crazy,right?Itsalotofwork,butitsalsoanincrediblycommonscenarioforSQLServer, especiallyinsmallerandmidsizeenvironments.IheardbackfromthisreaderjustasIwas startingtoputthischaptertogether.TheyvegotBackup2.0,andthenewsisgood: IwasabletosellthebossonaBackup2.0solutionbasedonwhatyou hadwrittenaboutSQLServer,butevenIwasntpreparedforhowmuch easierthingsare.WesimplyshutoffallthenativeSQLServerbackups, includingthetransactionlogs.Westillmakeanativefullbackupevery week,justtokeepthelogstruncatedandtohaveastaticcopytosend offsite,butwerebackingupabout2000instancesofSQLServermost ofwhicharesmall,fromdevelopmentenvironmentscontinuously.We neverhavemorethanaminuteortwoofdataatrisk.Wedonthaveto monitorbackupjobs,wedonthavetocopybackupfiles,andwedont evenhavetomanuallyrestorestuff.Whenitcomestimetoupdatethe testenvironment,wejustpointthebackupserveratadestinationandhit go.Wecanprovisionthetestenvironmenttomatchanydatabase,at anypointintime. AnothercreativeuseofBackup2.0.Obviously,movingfrom15minutesatrisktoaminute ortwoisabenefit,nottomentionalotlessmanualmonitoringandeffort.Butbeingableto quicklyrestoreadatabasetoatestenvironmentisabigbonus,andbeingabletodosoto agivenpointintimemustopenupalotofreallyvaluabletestscenarios,nottomention lessentheoverheadofmanagingallthatbackupdata.
209
Don Jones
Hotfixes,GoneCold
Patchesandhotfixescanbejustasmuchacauseforproblemsasusererror,serverfailure, andotherdisasters.Heresastorythatsomeofyoumayrecognize,andacreativeuseof Backup2.0tosolvetheproblem.Ivechangedtheantivirusvendornamebecausethe actualcompanyisntimportantthiscouldhappentoanyone. Werunabout100servers,allofwhichrunXYZantivirussoftware.That companyissuedapatchorupdaterecently,whichwedulyinstalled. Unfortunately,ourserversstartedworkingnotsowellafterthat.A coupleofthemshutdownandwouldntrestart,even.Notgood. Whileoneofmycoworkerswasonthephonewiththecompanystech support,andanotheronewithMicrosoft,Iheadedforthebackupserver. WehaveabackupsolutionthatworksliketheBackup2.0kindsthatyou describeitcopiesdiskblocksandtimestampsthem.Theproblemis,I didntwanttorestoretheentireserver;Ijustwantedtorestoreafew files.Ilookedatthepatchwehadappliedanditonlyupdatedfourfiles, soIwentintothebackupsystemandgotthefouroldfilesout.Theywere slightlydifferentfromservertoserverprobablydatafilesofsome kindsoIjustgotthemfromeachserversbackups.Theneatthingis thatwecandothisminutebyminute,soIwasabletogetthefiles exactlyastheywereJUSTbeforethepatchwasintroduced. Putthefilesontheirrespectiveservers,rebooted,andallwaswell.Oh, andwerestillwiththatsamecompanyforantivirus.Hopefully,itwont happenagain,butatleastnowwedonthavetoexplicitlymakeabackup beforeinstallingpatches. ThisisactuallyascenarioIdontrecallwritingaboutinthisbook.Youknow,theAlways BackupYourServerBeforeInstallingPatchesnoticethateveryhotfixorpatchorservice packcomeswith.Whodoesthis?Somecompaniesdo,Iknow,buteasilyjustasmanydont. Andyes,weallshould;itsjustthatbackupstakesolong.Ortheydid,untilBackup2.0. Now,yourserversarealreadybackedup,sotheresnoneedtoperformaspecificprepatch backup.Justdropthepatchesinplace,andifyoudontlikewhathappens,rollemback fromyourbackupconsole.
210
Don Jones
DeDuplicationDiscovery
AnotherhorrorstoryfromChapter2involvedacompanythatwasbackingupalotof duplicatedfiles,dueinparttotheiruseofDFSreplicas.Thatstoryconcluded: Istartedlookingintoit,andrealizedthatthecompanyhasbeendoinga lotofDFSreplicasabout50to60GBworthrightnow,andtheyre addingmoreallthetime.Thisisthesamedatabeingbackedupoverand overagainindifferentlocations.Allthatduplicateddataiswhatscausing theproblem.Istartedtryingtofigureoutexactlyhowmuchduplicated datawehavesothatIcouldmakeabusinesscasetomanagement.In doingso,Ifoundthatmanyofourfileservershavemultiplecopiesofthe samefiles,allonthesameserver.Alotofthetimeitcomesfromusers homedirectories:Thetwoserversthathostthe\\Company\UsersDFS nodehavetensofthousandsofduplicatedfilesbecauseusersdrag commonlyusedfilesintotheirownhomefolders. Wewerebackingupatotalof13.2TBofdata,andcloseto30%ofitwas duplicateddata.Onethird!Werecurrentlytryingtofigureouthowto excludetheduplicates,butitsactuallyverytrickywecantjustnot backupusershomefolders! Ifoundacolleaguewhohadbeeninasimilarsituation.HiscompanyhasbegunaBackup 2.0implementation,andtheyselectedasolutionspecificallyforitsabilitytocompressand deduplicatethebackedupdata.Iinterviewedhimtolearnmore,andhereswhathesaid: Wehavealotofduplicateddataonournetwork,too,muchofwhichisin userhomefolders.Butyouknow,evenifyoudonthaveexactly duplicatedfiles,everyoneprobablyhasalotofduplicatedataatthedisk blocklevel.Continuousdataprotectionsolutionscanoperateatthedisk blocklevel,sowelookedforonethatdeduplicatedatthatlevel,too. Illgiveyouaspecificexample:Justacrossourdozenorsofileservers,we havecloseto24TBofdata.Withouroldbackupsystem,wecould compressthat,andwegotabouta2:1compressionratio,orabouta50% reductionindatasize.Thatsstill12TBofdataanawfullottobackup. WehaveprettynewDLTtapedrives,andeachtapeholdsabout800GB, soIthinkwereusingacoupleoftapestopullafullbackupofallthat.
211
Don Jones
Wehadoneofourdeveloperswritealittletoolthatanalyzedduplication atthediskblocklevel,andthatswhatreallyjustifiedtheBackup2.0 expenditure:Histool(sorry,Icantshareit)suggestedthatwecould reduceourbackupsizebyatotalof80%byusingbothcompressionand deduplication.Thats24TBto480GB.Ithinkeveryonesjawshitthe table.Wecouldeasilyfitanentirefullbackupononetape,anditalso madeitalotmorelikelythatwedbeabletoaffordthediskbased storagethatBackup2.0solutionsrelyon. Acoupleofmonthslater,andthatsprettymuchthelevelofreduction wereseeing.Rightnow,wevegotfiveofthefileserversonthenew system,withatotalofabout15TBofdata,andwereseeingfullbackups occupyabout300GBofspace.Ofcourse,wereallyonlydothatonce, thenthesolutionjustgrabschangesfromthatpointon. Deduplicationisavailableinalotofways,includinginsomeBackup1.0stylesolutions. ButhavingitbuiltintoaBackup2.0solutionnotonlygivesyouthebenefitsofBackup2.0 butalsomakesitalotmorepracticaltoimplementbecauseyouremakingmuchmore efficientuseofthebackupsolutionsstorage. Iwanttopointoutthatyoushouldspecificallylookfordiskblockleveldeduplicationfor maximumsavings,ratherthanhigherlevel,filebaseddeduplication.Thelattercanonly saveyouspacewhenentirefilesmatchup,whereasdiskblockleveldeduplicationisalot morelikelytofindagreaternumberofduplicates.Figure12.4illustrateswhatthiskindof deduplicationwilllooklike.
Figure12.4:Replacingduplicateblockswithpointerstothefirstduplicate.
212
Don Jones
ItsaBackupandanArchive!
Inapreviouschapter,Icautionedyouagainstusingyourbackupsystemasapermanent dataarchive.Backupsshouldbeusedtostoreonlyasmuchdataasyoudwanttobring backintoproductionuseafteraloss;theyrenotnecessarilyintendedtostoredatafor yearsandyears.ButIdidhaveonereaderwhodisagreed: Ihavetotellyouthatwevebeenveryhappywithournewcontinuous dataprotectionsolution,especiallyasitappliestoourExchangeServer. Infact,despiteyourcautiontonotusethesystemasapermanent archive,wehavefoundittobeverywellsuitedforjustthat. Ourcompanypolicyrequiresustokeepabout3yearsworthofemails, andweusedtoleavethatinformationinExchange.Needlesstosay,our Exchangedatabasesbecameprettyunmanageable.Wevedecided insteadtojustspendthemoneyondiskspaceforournewbackup solution,toletitkeepalotofhistoricaldata,andtotrimdownour productionExchangedatabases. Ourbackupsolutionsuserinterfaceactuallyhasexcellentsearch features,anditunderstandstheformatoftheExchangedatabases.So wereabletosearcharchivedmessagesrightfromwithinthebackup solution,andwevealreadyhadacoupleoftimeswhenwevehadto accessindividualmessagesfromacoupleofyearsinthepast.Because doingthatdoesnttouchtheExchangesystematall,ourproductionmail servershavebecomemuchmoreefficient.Werelookingtocreatesome kindoftieredstoragehierarchysothatwecankeepevenlongerperiods ofhistoricaldata,ifneedbe,andwethinkourbackupsolutionwill accommodateusinthat. Diskspaceisprettycheap,evenredundantserverbasedstorage.Alot cheaperthanitusedtobe,atleast.Soforusitsaneasyinvestmentto make. Whichjustgoestoshowthatyoucantanticipateeveryonesbusinesspriorities.Illadmit thataBackup2.0stylesolutiondoesofferacompellingargumentforsecondaryuseasan archive,ifyourewillingtodedicatethediskspacetoit.
213
Don Jones
VirtualNirvana
Finally,mylaststoryisonethatsprobablyalltoofamiliartothoseofyouwhohaveto answercallsfromyourusers.IthinkitsagreatexampleofhowtoreallyputBackup2.0to cleveruse,too. WeuseaBackup2.0stylesolutiontobackupallourvirtualservers.Now, wedosomethingabitdifferentthanwhatsomeothersmightdo.Weput ourbackupagentinsidethevirtualmachine,andwebackupeverything thatway,butwealsobackupthevirtualmachinediskfilesthemselves.I know,itsredundant,buttherearecaseswhereitcomesinveryhandy. Perfectexample:Afewmonthsback,wehadausercallandtellusthat thepayrollsystemwasoffline.Wefiguredoutprettyquicklythatitwas duetosomecorrupteddatabasesthishashappenedbeforebutwe werealittleunsureofhowfarbackwecouldrollthedatabase,andwe franklywerentevensurewhichfilesweneededtorestore.Thesystem usesaproprietarydatabaseengineandthedataisspreadacrosswhat seemslikeamillionfiles. Anotheradminwasworkingthisproblem,andhewastryingtofigureout exactlywhichfilestopullfromthebackupsystem.Ifoundhimdigging throughthefileslist,andaskedwhatwasup.Heexplainedthesituation, andIsaid,Youknowwhat?Letsjustblowawaythewholevirtual machine.Helookedblank,thenrememberedthatBackup2.0madethat prettyeasy. Nowhereswhywebackupboththeinsideofthevirtualmachineaswell asitsdiskimages.ChoiceAistorestoretheinsideofthevirtual machine,basicallyrollingitbackasifitwasaphysicalmachine.Thatcan takeawhilebecauseyouregettingtheslowerdiskI/Oofthevirtual machine.Inthiscase,timewasafactor,sowerolledbacktheoutside ofthevirtualmachinethatis,itsdiskimages.Muchfasteroperation lessthan20minutes,ifIremembercorrectly. Agoodexampleofhowdifferentformsofbackupscanmeetdifferentneeds!
214
Don Jones
Conclusion
Well,thereyouhaveit.Hopefullythesestorieshavegivenyousomeideasabouthow Backup2.0cansaveyoutime,money,andanguish,andhopefullythe11preceding chaptershavehelpedyouunderstandhowBackup2.0canfitintospecificnichesofyour organization. Inafewyearstime,ImbettingwellseemoreandmorecompaniesadoptingaBackup2.0 stylecontinuousdataprotectionsolution.Infact,itwontbelongbeforeBackup2.0is simplythestandardthateveryoneuses.Werecertainlyseeingmoreandmorevendorsin thisspace,providingavarietyofsolutions,whichindicatestheyrepayingcloseattentionto businessesneeds. Ifnothingelse,thisbookshouldhavepreparedyoutobeasmartershoppersothatwhen youstartevaluatingBackup2.0stylesolutions,youknowwhatfeaturestolookfor,whatto prioritize,andwhatyourbusinessfullrangeofneedsislikelytobe. Aboveall,IhopeyouveenjoyedreadingthisbookasmuchasIveenjoyedwritingit.Thisis thelongestbookIvewrittenforRealtimePublishers,andIwanttothankthemfor publishingit,andIwanttothankthebookssponsorfornotonlymakingthisproject financiallyviablebutalsoforgivingmethefreedomtoreallytelltheBackup2.0storythe wayIwanted. Goodluckwithyourbackups!
215

The Definitive Guide To

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

The Definitive Guide To

Hochgeladen von

Copyright:

Verfügbare Formate

The Definitive Guide To

Windows Application and Server Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

Full database backup

Transaction log backup

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

Thisispowerfulmagic,butitsareality:Todaysmarketincludessolutionsthatfollowthis technique.ItdeliversBackup2.0:continuousbackupsthataredesignedforrestores,notfor archiving.

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

Remember:Backupsshouldpreventusfromlosinganydataorlosinganywork,andensure thatwealwayshaveaccesstoourdatawithaslittledowntimeaspossible.TheBackup1.0 mentalityoriginallyemployedinthishorrorstorycertainlydidntmeetanyofthose criteria.

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

ItjustseemsasifBackup2.0,asasetofpracticesandcapabilities,ismuchmorewellsuited toSQLServerthantheoldstyleBackup1.0mentalityourauthorisrelyingon.Welllookat SQLServerinmoredetailinChapter5.

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0

The Definitive Guide to Windows Application Server and Backup 2.0