Beruflich Dokumente
Kultur Dokumente
JamesDixon ChiefGeek,Pentaho
November2010
Copyright20072010PentahoCorporation.Redistributionpermitted.Alltrademarksarethepropertyoftheirrespectiveowners.Forthe latestinformation,pleasevisitourwebsiteatwww.pentaho.com.
PentahoAgileBI:Aniterativemethodologyforflexible,fastandcosteffectiveBIprojects
Contents
Contents.................................................................................................................................. 2 Introduction ............................................................................................................................ 3 TheChallengesofTraditionalBusinessIntelligence .................................................................. 3 MooresLaw.............................................................................................................................. 4 CloudComputing....................................................................................................................... 4 FuzzyReturn.............................................................................................................................. 4 LackofSharedVision ................................................................................................................ 4 DevelopmentLatency ............................................................................................................... 4 TopDownDeficiencies.............................................................................................................. 5 BottomUpDeficiencies ............................................................................................................ 5 NoSmallBIProjects ................................................................................................................ 5 ThePrototypingCosts ............................................................................................................... 5 Abandonment ........................................................................................................................... 6 SummaryoftheProblems......................................................................................................... 6 TheAgileApproachtoBusinessIntelligence............................................................................. 7 WhatDoWeMeanbyAgileBI? ................................................................................................ 7 AgileandLeanPrinciples........................................................................................................... 7 LeanDelivery............................................................................................................................. 7 AgileTeams ............................................................................................................................... 8 AgileHardware.......................................................................................................................... 8 AgileSoftware ........................................................................................................................... 9 PentahosAgileBIInitiative ..................................................................................................... 9 Tools ........................................................................................................................................ 10 DeploymentOptions ............................................................................................................... 10 AgileBehavior ......................................................................................................................... 10 AgileBIUseCases.................................................................................................................... 11 TheBoundariesofAgileBI ..................................................................................................... 11 Summary ................................................................................................................................. 12 DownloadandContactInformation........................................................................................ 12 References............................................................................................................................... 12
Pentaho
PentahoAgileBI:Aniterativemethodologyforflexible,fastandcosteffectiveBIprojects
Introduction
AtPentahowebelievethattheoldtechnologies,theoldpricing,andtheoldapproachesusedforBusiness Intelligenceproductsarenotwellsuitedtotodaysenvironment. ThiswhitepaperintroducesPentahosAgileBIinitiativewhichencompasses:
Technology:Providesintegrateddesign,modeling,andvisualizationtools. Participants:ExpandstheBIdeveloperbase. Processes:EnablesnewbehaviorsandnewBIusecases. Deployment:Enablesmigrationbetweendesktop,public/privateclouds,andonpremise
environments.
Economics:Reducestheoverallcostsandallowsincrementalspendingasvalueisrealized.
TheChallengesofTraditionalBusinessIntelligence
TheBusinessIntelligence(BI)marketisfacedwithmanyfactorsthatareboundtochangeit. SpreadsheetsarewidelyrecognizedasthemostcommonlyusedBItool.TheseBIspreadsheetsareknown asspreadmarts.Howeverthesespreadmartsolutionshavemanyissuesoftheirown,includingsecurity, dataquality,consistency,scalability,maintenancecostsandlackofmanyimportantBIfeatures.Despite thesedownsidestheyexistbecausemanyofthetoolsandtechniquesthataredesignedspecificallyforBI donotprovideabetteralternative:eachcomeswithitsownproblems.TheresultisthatBIprojectsoften fail.Theyareabandonedbeforetheyarestarted,abandonedduringdevelopment,orneverusedbecause theydonotdeliverthefeaturesorvaluethatusersexpect. ThesectionsbelowdescribetheproblemsfacedbyBIprojectsandtools.
PentahoAgileBI:Aniterativemethodologyforflexible,fastandcosteffectiveBIprojects
MooresLaw
MooresLawstatesthatcomputerchipperformancedoublesevery20months.Datawarehouseswere firstinventedinthemid1980s,andonlythebiggestcompaniescouldaffordthem.Duringthaterathe commoditychip,theIntel386chip,had275ktransistors.Todaytheequivalentcommoditychip,theCore 2Duo,has291milliontransistors.Inthe15yearssincetheinventionofdatawarehouses,computing powerhasincreasedbyafactorof1000.Itisonlynaturalthat,ascomputingpowerincreases,systems thatwerepreviouslyexpensivebecomecheaper,andeventuallyacommodity.TheBusinessIntelligence marketwill,naturally,beaffectedbythistrend:companiescancreateBIsolutionsthattheycouldnot affordbefore,andindividualusershaveequipmentcapableofrunningbasicBIsolutions.
CloudComputing
Theemergenceofpubliccomputingclouds,suchastheAmazonECcloud,andonpremiseclouds,suchas Eucalyptus,havethepotentialtoaffecttheBIindustry.Utilitypricingandtheabilitytocreateaninstance ofanBIserverquicklyandcheaplyisverypowerful.
FuzzyReturn
Whilethebenefit(orreturn)ofacompletedBIprojectisoftendifficultorimpossibletoquantify,itis relativelyeasytomakequalitativestatementssuchas:Wewillbeabletomakequickerdecisionsthat willhelpreduceprojectcosts,or:Wewillbeabletomakebettermarketingdecisionsthatwillincrease sales.Makingquantitativestatementslike:Wewillcutprojectcostsby15%bymakingquicker decisionsaremuchmoredifficultbecausethereturnoninvestment(ROI)isdependentontheasyet unknownreturnontheBIproject.WithanuncertainROI,anappropriatelevelofinvestmentalso becomesdifficulttoestimate.WithBItoolsthathavelargeupfrontcosts,thisproblembecomeseven worsebecausetherewillbenosignificantinvestmentunlessthereissomeexpectationofalargereturn. Asaconsequence,manyBIprojectsareneverstarted.
LackofSharedVision
Manyuserscannotcompletelyenvisiontheendresultthatisbeingdeveloped.Frequently,whenBIusers firstgetaccesstoanewsystem,theywillimmediatelyperceiveawholenewsetofrequirementsthat theyhadnotrealizedbefore.Unfortunatelytheyoftencannotprovidefeedbackontherequirements, design,orvalueofaBIsystemuntiltheyseeactualresultswithrealdata.AsaresultBIdevelopersoften workwithinitialrequirementsthatareeitheraccurateorincomplete. Inaddition,mostusersdontunderstandtheterminologyusedtodescribetheplannedBIsystem.This makesitevenhardertocreateasharedvision.
DevelopmentLatency
DuringtheexecutionofaBIproject,thereshouldbecheckpointstogatheruserfeedback.Thisfeedback shouldbeusedtovalidatethatthesystembeingdevelopedmeetsthebusinessexpectations.InmanyBI projects,thetimebetweencheckpointsistoolong,increasingtheriskandlikelihoodoffailure.The problemdeepenswhentimeorresourcesconstraintspreventthefeedbackfrombeingincorporatedinto thesolution.
PentahoAgileBI:Aniterativemethodologyforflexible,fastandcosteffectiveBIprojects
TopDownDeficiencies
InatopdownapproachtoaBIproject,youstartbygatheringrequirements,thenyoudesignasystem impliedbythoserequirements,andthenyouimplementthatsystem.Theproblemwiththisapproachis that,duetocommunicationandvisiongaps,itislikelythatyourinitialrequirementsareincomplete, resultinginaprojectthatfallsshortofexpectations.Additionally,youllspendconsiderabletimeand moneybeforethisfactbecomesevident.Ifyoucantbeginwithasetofrequirementsthatisclear, accurate,complete,relevant,timely,understoodandtrusted,atopdownapproachisveryrisky.
BottomUpDeficiencies
InabottomupapproachtoaBIproject,youstartbyprovidingaBIsolutionforasourcesystem(ERP, CRM,etc.)oradatasourcewithoutmuchregardforuserrequirements.Byprovidingreports,dashboards, trending,summaries,andsliceanddicefunctionalityforasourcesystem,youarelikelytomeetatleast someofyourusersrequirements.Theproblemwiththisapproachisthattheanswerstousersbiggest problemsmightbeoutsideoftheavailabledata.Alittleenrichmentofthedatamightaddsignificant value.Aswiththetopdownapproach,youwillspendconsiderabletimeandmoneybeforeyoudiscover thisshortcoming.
NoSmallBIProjects
ManyBIexpertsrecommendthatBIteamsStartsmall,butthinkbig.Theyrecommendstartingwitha smallprojecttogetsomesuccessandmomentum,andthencontinuetobiggerandbiggerthings. However,evenstartingsmallcanbehardwhen,tomakeanyprogress,youneedthetimeandskillsof sponsors,endusers,ITdevelopers,consultants,businessanalysts,andDBAs.Inmanycasesittakesa strategicinitiativeoramandatefrommanagementtogetacrossfunctionalgrouplikethistoworkona projecttogether.Undertypicalworkloadsandbusinesspressuresitishardtogetparticipationfromall thenecessarygroups.TheymayalsobelievethatdurationofthepotentialBIprojectistoollong,thereby reducingthebenefitofcompletingit. Inadditiontotherequisitehumancapital,thehardwareandsoftwarecostsincreasethesizeoftheBI projectsinitialinvestment.IdeallyyoushouldcollectfeedbackfromalargeuserpopulationduringaBI project,butsoftwarethatislicensedonaperuserbasismayprohibitthis.
ThePrototypingCosts
Giventheproblemsabove,itseemssensibletoperformaprototype,pilot,orfeasibilitystudybefore startingaBIproject.Thisway,userswillhavetheopportunitytoprovideconcretefeedbackaboutthe solutionanditsbenefits.Indeed,manyBIexpertsrecommendusing510%oftheprojectsbudgetto createaprototype.Prototypingisvaluablebecauseitprovidestheopportunitytoperformasecond iterationoftherequirementsanddesignofthesystembeforebuildingitforproduction. Prototypingworkswellwhenyouhavealargebudget,butwhenthebudgetforaBIprojectissmall,there isaproblem5%ofasmallbudgetisaverysmallbudget. Unfortunately,manyoftheBItoolsavailabletodayareexpensiveandlicensedconservatively,making themtoocostlytouseforprototypingwithoutviolatingtheirlicenseagreements.Tohelpalleviatethis problem,someBIvendorsprovidepresalessupporttojumpstarttheproject.However,involvinga
PentahoAgileBI:Aniterativemethodologyforflexible,fastandcosteffectiveBIprojects
Abandonment
InrealitysomeBIsolutionsfallintodisuseovertime.Sometimesthishappensquickly,othertimesslowly. Therearenumerousreasonsforthis,notallofwhicharenecessarilybad,andinclude:
Theexpectedbenefitwasntdelivered. TheinsightsprovidedbytheBIsolutionshiftfocusfromdiscoveringissuestosolvingthem. Itbecomesquicklyapparentthatoperationalchangesareneededtofixdataqualityissues(e.g.
incompletedataincriticalelements,forexampleReasonAccountClosed)
Changeincorporatepriorities,ordepartmentalgoals.
InvestingtimeandmoneybuildingBIsolutionsthathaveanuncertainlongevityisobviouslyrisky.
SummaryoftheProblems
TherearemultipleproblemsencounteredbythetraditionalapproachtoBIprojects.Theseproblemscan begroupedintocategories:
Peopleandskillsrequired:Manyprojectsneverstartbecausethenumberanddiversityofthepeople
requiredistoogreat.
Lackofiterations:Manyprojectsfailbecausetheinitialprototype,ifdoneatall,istheonlyiterationof
requirementsanddesign.
Suitabilityofthetools:TheusabilityandproductivityoftheexistingBItoolsareimpedimentsformany
BIprojects,asarethehardwarerequirementsforthecombinedtoolset.
Costs:ThepricingandlicensingofBIsoftwareandthecostofthenecessaryhardwareincreasestherisk
ofundertakingaBIproject.
PentahoAgileBI:Aniterativemethodologyforflexible,fastandcosteffectiveBIprojects
TheAgileApproachtoBusinessIntelligence
AsdiscussedabovethereareproblemsinBIprojectsrelatedtopeople,processes,software/hardware, andcosts.Anysolutiontotheseproblemsshouldaddressalloftheseareas.WeatPentahobelievethat AgileBIachievesthis.
WhatDoWeMeanbyAgileBI?
Thewordagileisusedasabuzzwordinmanycontextsandindifferentways.Weareusingthewordin itstraditionaldefinition:theabilitytomovequicklyandeasily,inanimbleandwellcoordinatedway. So,byAgileBI,wemeantheabilitytocreateBIsolutionsquicklyandeasily,inanimbleandwell coordinatedway. UsinganagileapproachimprovesthesuccessofBIprojects,andenablesyoutostartmoreprojects.It doesthisbychangingtheeconomics,thetechnicalsolution,andtheexecutionoftheprojects.
AgileandLeanPrinciples
Inrecentyearsorganizationshavebeenincreasinglyusingagileandleansoftwaredevelopment methodologiesandtools.Thisriseinpopularityisspurringtheadoptionofagilephilosophiesinother domains.
AdaptingtheprinciplesoftheAgileManifestotoworkwithBIleadstothese: Satisfythecustomerthroughearlyandcontinuousdeliveryofvaluabledataandfeatures. Welcomechangingrequirements,evenlateindevelopment. Deliveraworkingsolutionfrequentlyandmeasureprogressbythis. Fosteracloserworkingrelationshipbetweenbusinesspeopleanddevelopersthroughouttheproject. Buildprojectsaroundmotivatedandknowledgeableindividuals. Decidelate,deliverfast. ThefrequentdeliveryofaworkingsolutionwillobviouslysolvesomeoftheproblemsBIprojectsface: Communicationandvisiongapswillbereducedineachiterationasendusersseetheworkingresults. Developmentlatencywillbesignificantlyreduced. Shortcomingsofthetopdownorbottomupapproachwillbealleviatedasrapiditerationsallowa
hybridapproachthatcombinesoralternatesthem.
LeanDelivery
Youcanreducedevelopmenttasksandcostsbyusingthedecidelateprinciple.Bytreatingthefirst deliveryofaBIsolutionastemporaryuntilprovenotherwise,youavoidextraworkandcost.Some examplesofsavingsare:
Usemanualflatfileextractsfromsourcesystemsinsteadoffullyautomateddataflows.
PentahoAgileBI:Aniterativemethodologyforflexible,fastandcosteffectiveBIprojects
Extractapartial(butstilluseful)setofdata.Thedatacanbelimitedbyatimerangeorcanbe
restrictedtoasubsetofageographical,organizationalorotherdimension.Makesuretheextracted dataisfullyusefultoasubsetofusers,notpartiallyusefultoallofthem.
Transformthedataintosimplefacttablesinsteadofstar,snowflakeorothercomplexdataschemas. Installthesolutiononexistinghardware,orcloudbasedhardware. Useopensourcedatabases,middleware,andfrontendsoftwareinsteadofproprietarysoftware. Dontbotherwithautomation,auditing,productioncontrols,etc.
AgileTeams
AnagileBIteamistypicallymadeupof45people,eachtypicallyhavingoneoftheseroles:ITDeveloper, ProjectManager,BIConsultant,EndUser,BusinessAnalyst,and/oraDatabaseAdministrator.Anyof thesepeopleiscapableofstartingaprojectontheirown. Manyspreadmartsinexistencetodayarecomplicatedandintricate.Mosthavebeenconstructedbyend usersbecauseanofficiallysanctionedBIsolutionisneitheravailablenorplanned.Thisshowsthatthereis apopulationoftechnicallyorientedenduserswhoarewillingandabletocreateBIsolutions.Having theseindividualsontheteamandgivingthemtoolsthatenablethemtoexperimentwillhelpBIprojects significantly. Ideallytheteamshouldbebasedinthesamelocation,andiftheycanworkinthesameroommostofthe time,thatsevenbetter.Regardlessoflocation,theteamshouldbeprovidedwithtoolstohelpthem collaborate,suchasforums,mailinglists,wikis,andadocument/contentmanagementsystem.
AgileHardware
IfyouneedtoacquirecomputinghardwarebeforeaBIprojectcanbegin,youcanrunintotrouble.In somecasesitdelaysthestartofaproject,inothercasesitisacontributingfactorinaprojects cancellation. Togetaprojectgoingquickly,orenableaprototypetobeconductedcheaply,youllfinditadvantageous touseoneofthefollowing:
Userhardware:Usingexistingdesktops,workstations,orlaptopsmeansnoprocurementdelaysor
PentahoAgileBI:Aniterativemethodologyforflexible,fastandcosteffectiveBIprojects
budgetspending.Adesktopenvironmentisgreatforabusinessanalystortechnicallyorientedend usertogetstartedonaproject.
Cloudcomputing:CloudcomputingquicklyandcheaplymakeaBIsolutionavailabletoadistributed
groupofpeople.ThisincludesbothpubliccloudslikeAmazonEC2andprivatecloudslikeEucalyptus. Insomecasesusershardwareislockeddownandonlycertainapplicationsareavailabletothemsuchas officeproductivity,email,web,andcorporateapplications.Inthesecasecloudcomputinggives technicallyorientedendusersanewoption InmostcasesaBIsolutionwillgointoproductionondedicated,onpremisehardware.Butprototyping anddevelopmentcanbedoneondesktopmachinesandcloudenvironments.Theabilitytomigrateeasily fromuserhardwaretocloudenvironments,andcloudenvironmentstostaticdeploymentsfurther increasestheproductivityoftheteamandtheflexibilityoftheproject. Forthesehardwareoptionstobeviable,theBIsoftwaremustbesuitable(intermsoflicensingand hardwarerequirements)forallthoseenvironments.Thesoftwaremustscaleuptomeetthedemandsof theproductiondeployment,butitmustalsoscaledownontolaptopsandutilityhardware.
AgileSoftware
AnagileapproachworksbestwheniterationsoftheBIsolutionarefrequentlydeliveredtoagroupof endusers,whoprovidevaluablefeedbackandchangingrequirementsbasedontheprogresssofar. Thisimpliessomerequirementsonthesoftwareused.TheBIsoftwareusedshould:
Supportquickiterations:Iterationswilltakelongerifthetoolsarecumbersome,hardtouse,ordonot
workwelltogether.
OfferfullBIcapabilities:Eventhequickestprototypeoriterationislikelytoinvolvedata
transformation,dataquality,modeling,visualization,andcontentcreation.
Makebasicfeatureseasytouse:Thesoftwareshouldenabletechnicallyorientedendusersto
participateinorinitiatedevelopmentofaBIsolution.
Allowdeliverytoalargeaudience:Valuablefeedbackwillbelostifthelicensingofthesoftware
restrictsthepotentialpoolofendusersprovidingfeedback.Forthisreasonyoushouldavoidsoftware thatislicensedperuser.
Allowprototyping:Theabilitytoperformprototypesorpilotprojectsatwill,withoutthehindranceof
softwarelicensingissues,enablesmanymoreBIprojectstobeconsideredfordevelopment.
PentahosAgileBIInitiative
In2009PentahostartedanAgileBIinitiative:http://www.pentaho.com/agile_bi/ AlongwiththereleaseofthispaperPentahoislaunchingthefirsttoolsetwithallthedeploymentand pricingoptionsneededforAgileBI.Thisisthefirstversionofthesetools,completewithintegrateddesign toolsandutilitypricing.
PentahoAgileBI:Aniterativemethodologyforflexible,fastandcosteffectiveBIprojects
Tools
AnintegratedETL,modelinganddesignenvironment,andaBIserver: Unlimited,perpetual,freeuseoptions:Freedesktopdesignenvironment.OpensourceBIserver. Enterpriseoptions:Enterpriserepositoryforsecurity,collaboration,andversioning.EnterpriseETL
sourcesintodatamartsanddatawarehouses
Reporting:pixelperfectoradhocreportingeitherdirectlyagainstsourcesystemsorusingacentralized
BImetadatalayer
Analysis:interactivedataanalysisusingarelationalOLAP(ROLAP)architecturethatdelivershigh
performanceforbusinessuserseveninlargedataenvironments
Dashboards:integratedviewsofkeybusinessmetricsusingreports,charts,dials,maps,orothervisual
displaytechniques
PredictiveAnalytics:advanceddataanalysisdesignedtouncoverhiddenpatternsindataandto
supportpredictiveanalytics
BIServer:thesupportinginfrastructureforPentahosenduserBIcapabilitieswhichincludesservices
forscheduling,distribution,metadata,security,portalintegration,andmore
DeploymentOptions
DesigntoolsandserversarecrossplatformWindows,Linux,OSX,Solaris.Alltoolsandserverscanbe runoncommoditylaptops. Enterpriseoptionsareavailableonpremise,hosted,orcloudbasedwithutilitypricing.
AgileBehavior
Specifically,thesetools,deploymentoptions,andpricingoptionsallowBIpractitionerstobehaveinnew ways:
ABIprojectcanbestartedbyasingleenduser,businessanalyst,ITdeveloper,databaseadministrator,
orconsultant.
Differentparticipantscanbeengagedsequentially,notsimultaneously.Anenduser,businessanalyst,
orconsultantcancreateaBIproject,thentheITgroupcaninstitutionalizeitovertime,asitsusage dictates.
ABIprojectcanbedevelopedonalaptop,onahostedservice,inthecloud,orinadatacenter.The
projectcanbeeasilymovedamongtheseenvironments.
Aprototypecanbecompletedforlessthanafewhundreddollarscashoutlay,ornocashoutlay. SpreadmartdevelopersbecomeBIdevelopers,andhavetheadvantagesofboth:control,flexibility,
10
PentahoAgileBI:Aniterativemethodologyforflexible,fastandcosteffectiveBIprojects
selfsufficiency,scalability,security,andreliability.
AgileBIUseCases
AgileBIcanbeusedindifferentscenarios.ThesearesomeexamplesofusingAgileBIforprojectsthatare drivenbyIT.
FastTrack:TakeyourmostimportantBIproject,andbeagilewithit.Createaprototypeusingexisting
whereinterestingdatafieldsarenotconsistentlypopulated.Altertheapplicationlogicoroperational proceduressothatthosefieldsbecomesuitableforfutureanalysis.
ThesearesomeexamplesofusingAgileBIforprojectsthataredrivenbyendusers. SpreadmartConversion:Findyourspreadmartauthors,providethemthetoolstoturntheir
themcreatetheirownprototypesandsolutions.Monitorthemtoseewhichareusedfrequently.Turn theseintosupportedsolutions.
TheBoundariesofAgileBI
SowherearetheboundariesofAgileBI?WhatisnotAgileBI? AgileBIisnotaproductitiscombinationoftechnology,economics,andexecutionthatenablesnew behaviors. AgileBIisnotanalternativetotheKimballDataWarehousemethodology.AgileBIprovidesnewwaysto approachBIprojects.YoucanuseAgileBItocreatedatamartsoneatatimeorinparallel,andthenuse theKimballDWmethodologytoapproachthecreationofadatawarehouse. AgileBI,becauseofitsiterativenature,itisnotidealforfixedprice,waterfallstyleprojects.Asan alternativeapproach,someconsultingcompaniesoffertheirtechnicalexpertiseonapayperiteration
11
PentahoAgileBI:Aniterativemethodologyforflexible,fastandcosteffectiveBIprojects
Summary
AgileBIchangesourperceptionofBIprojectsbydramaticallychangingtheireconomicsandexecution. Insteadofregardingthemassomethingthattheorganizationmightstartnextquarteriftheycanlineup theresources,theycanbeviewedassomethingthatIcanstartthisafternoon. ThetraditionalBIvendorshavetalkedaboutBIforthemasses,BIeverywhere,andBIforeveryonefor years.Whatnoneofthemhavedoneisdeliveratoolsetthatenablesthistoactuallyhappen.Pentahos AgileBI,bychangingthetechnical,operational,andeconomicfactorsofBI,enablesnewbehaviorsbyall participantsinBIprojects.ThesenewbehaviorsenableBItocrossthechasmfrombeingmanagement mandated,tobeinguserdriven.
DownloadandContactInformation
PentahoAgileBIhttp://www.pentaho.com/agile_bi
References
Pentaho:http://www.pentaho.com AgileManifesto:http://www.agilemanifesto.org AgileSoftwareDevelopment:http://en.wikipedia.org/wiki/Agile_software_development LeanSoftwareDevelopment:http://en.wikipedia.org/wiki/Lean_software_development LeanDelivery:http://blogs.forrester.com/boris_evelson/10-03-03-333_rule_keep_your_bi_apps_check MooresLaw:http://en.wikipedia.org/wiki/Moore%27s_law TransistorCounts:http://en.wikipedia.org/wiki/Transistor_count
12