You are on page 1of 35



Stefan Zahrer


 Why introduce Run Book Automation
 Planning
 Tool selection and appraisal
 Experience so far
 RBA Design Framework
 Summary

Titel der Präsentation und Name des Redners




2011 4 . Main objective: Improve RUN efficiency in Infrastructure Operations – reduce cost 2.09.RBA implementation project objective 1. Develop a RBA platform for all global data centers 3. Establish a RBA service Quality criteria  Improve provisioning speed and quality  Make automation available to all services  Ensure reusability of flows and low maintenance effort  Quickly deliver financial benefits Oh. almost forgot: Make this happen in a infrastructure that was build over the last 10 years without automation in mind… Titel der Präsentation und Name des Redners 30.

09.Vision: Fully automated provisioning – no hands-on tasks Manual Provisioning A u t o m a t i o n  Rapid Provisioning  High Standardization  Up-to-date Documentation  Reliable Quality – No Human Errors  Low Process Costs  No Workload Buildup  Less Operational Management Industrialized Provisioning  Fixed Service Offering  Self-service Portal with service request dialog  Detailed execution logs  Reliable lead time and quality Titel der Präsentation und Name des Redners 30.2011 5 .


2011 7 . etc.09.) Review RBA Design Framework Iterations More Automations in Munich Automations for global Datacenters Automations in the making Installation of Managed virtual Linux Server Installation of Oracle Databases Promote AD Member Server to Domain Controller Automation for HVD data store administration (later reuse for Netapp Storage London) Titel der Präsentation und Name des Redners 30.Planning 2010 2011 2012 Establish RBA Service POC for tool selection Pilot Virtual Windows Server Management Decision Other Flows and Automations (HP SM Interface.


HP OO wins in proof of concept  3 tools in POC  5 automation tasks to demonstrate capabilities  Only HP OO can complete all 5 tasks within a set time Category HP OO B C Accomplishment of mandatory workflows ++ ++ + Accomplishment of optional workflow ++ o o Skills of personnel (Number of personnel) ++ ++ + +++ + o ++ ++ + + + - Number of pre-assembled connectors and potential workflow development effort Overall quality of results Pricing 9 .

 Fix one or two bugs OO Central HP OO Studio features to improve  Flow synchronization and staging is complex  Replace file system based repositories with database repository with staging functionality  Coordination of shared flow customizing is difficult  Improve check-in/check-out functionality  Feature to synchronize system accounts between repositories  More transparency on flow permissions.09.2011 10 .HP OO tool appraisal after nine month of flow customizing In general very happy with HP OO tool!!!  Easy to use  Flow customizing more efficient than classical scripting  Program control in graphical flows is easy to understand HP OO Central Web Portal  More functionality for OO Central user input dialog Calendar for date selection. etc. better handling Non-functional  Further development internet community and knowledge base  Optimize support reaction times  Experienced HP OO experts in short supply  Accelerator Packs and Integrations are like coding no des simple Titelexamples der Präsentationbut und Name Redners drag&drop 30. set default values. radio button.


People  People like their jobs – sometimes not readily offering information/support  For automation we need to now how things are done but also why they are done this way – understanding both takes time  Automation requires cross-service collaboration  Initial concern about automation going berserk  In the beginning people were critical – my job is to difficult for automation  Automation without peoples support may be possible but effort explodes Stefan Zahrer 30.2011 12 .09.

2011 13 . very complex indeed  Ask for picture of the architecture – discover it’s in people’s heads  Work on production systems as early as possible – lab systems are not the same  Architecture must be designed with automation in mind  Automation can be driver for standardization Stefan Zahrer 30.09.Technology  Infrastructure is complex – very.

Server.2011 14 . DB. etc.) Offshoring  More advanced administration tasks  Better understanding of infrastructure required  Participation in automation design Documentation  Documentation in automation tool  Consistent and reliable  Focus on technical administration processes Titel der Präsentation und Name des Redners 30. …)  Administration designed for automation (tool-based decisions.09. OS. not in run operations  Share responsibility for automated service delivery Develop Business Model Outsourcing Provider  Collaborate in automation design.Run Book Automation is organizational change for all IT functions Automation Benefits  High Standardization  Fixed Service Offering  Fast and reliable provisioning Develop People Develop Standards Application Development  Identical processes for all projects  Develop Self-Service provisioning know-how  Standardize architecture to a higher level Infrastructure Operations  Design overarching processes and consistent infrastructure  Cross-service collaboration  Work on architecture. build and release  Include automation in continuous service improvement process  Support operators to become experts with crossservice knowledge Develop Qualification Technical Infrastructure  Standardization on all levels (Storage.


new CMDB attributes. … Titel der Präsentation und Name des Redners 30. feature changes.2011 16 . …  In the organization: Changes to processes.The Gordian Knot – how to ensure reusability and maintainability HP OO flows execute functions in infrastructure systems based on organizational processes Automation Flow Automation Flow Automation Flow Automation Flow Infrastructure Process sequence and technical operations are closely interwoven  What happens if things were to change?  In the infrastructure: new software version.09. new systems.

2011 17 .Scheduling of Controller Flow for every CI at defined Downtime or execute immediately Private Database Flow Staging and Deployment  Advanced logging / reporting  Documentation of user input  Infrastructure Topology Model  Flow Repositories  Flow Release Process Transactional Data Persistence Table Server Name Table Logical Logging Master Data Key Value Store Infrastructure Topology Model Titel der Präsentation und Name des Redners 30.09.Searching for Downtime of every CI in I3S .Major topics of RBA Design Framework Flow Authoring ITIL Integration Start Start of scheduled Controller Flow Select and Start UserInterface-Flow in HP OO Central Check if respective Change is canceled Input in User-InterfaceFlow of HP OO Canceled? System Call to I3S (Create/Save Change) User Interface (UI) Controller (CO) Utility (UT)  Flow Architecture  Flow Call Relationships  Naming Conventions  Authorization Model Optional: Subcontroller (SC) Action Plan (AP)     Yes Abort Report in Private Database No No CI still exists in I3S Change? Change Management Incident Management CMDB Maintenance Downtime Management Freeze times.Acceptance of I3S Task . … System Call to I3S to verify Change Relevance No Yes Check if running in Downtime Window Mode (Flow Input) Change Relevant Flag set? Yes Yes Check if Downtime is still valid Input Risk.and Impact-Parameters No Yes Read Input Values from Private Database Time is agreed Execution Time? Do Work on CI Downtime Window Input Parameters for Time is agreed (Start Time/Finish Time) Input Parameters for Downtime Request Update Data in Private DB Document in I3S Fetch Downtimes via I3S for first task Start immediately after aproval Exited with Error? Create Incident yes no Check for last CI Possibility that downtime window is not available yes Close Task no Submit Change Exit Writing of Variables in Private Database End of User Interface Flow Aproval Process First Automated Task (Status assigned) I3S creates XML-File Infrastructure Interface (II) Scheduled File Check picking up XML-File Start of Scheduling Flow . veto.


Summary  Tools are (finally) ready for large style automation  Vendors starting to include interfaces for automation in their products  Automation of existing infrastructure is possible to large extend  Long time risk in using tools without architecture model in place  Implementation effort is significant  Financial benefit/ROI depends on number of runs Stefan Zahrer 30.09.2011 19 .

2011 20 .Questions Stefan Zahrer 30.09.


2011 22 .Flow Architecture – separates process control and system control User Interface (UI) Utility (UT) Controller (CO) Creates GUI for users • Populates selection lists • Verifies user input • Checks user permissions Controls processing order of tasks Optional: Subcontroller (SC) Combines tasks on different systems to reduce complexity of Controllers Action Plan (AP) Combines functions to execute a complex task on one system Infrastructure Interface (II) Executes a single function on one infrastructure system Titel der Präsentation und Name des Redners 30.09.

09.Organizational Process Can be tested without affecting infrastructure Controller High Very Low Organizational Process High Subcontroller Medium Low Technical Process High Action Plan Medium High Infrastructure systems and applications Low Infrastructure Interface Low High Infrastructure systems and applications Low Utility High New HP OO versions only Low Low Titel der Präsentation und Name des Redners Modify/Test Effort 30.Characteristics of flow types Flow Type Complexity Reusability Modify and Test for change to User Interface Very High Very Low .User input Medium .Infrastructure topology .2011 23 .

Rules for flow calls limit complexity User Interface (UI) Scheduler Persistence Table Utility (UT) Controller (CO) Optional: Subcontroller (SC) Action Plan (AP) Execute Configuration Changes Action Plan (AP) Infrastructure Interface (II) Infrastructure Interface (II) Execute Configuration Changes • Gather information • Create i3S Changes • Logging Private DB Logging.09. Infrastructure Systems Titel der Präsentation und Name des Redners 30.2011 24 . etc.

more to come… Flow run specific variables Logical Logging  Reporting of success/failure  Detailed reporting for services  Documentation of user input  Handover variables for scheduled runs Basis for result reporting Master Data Key Value Store Global Constants Infrastructure Topology Model Infrastructure Topology Model  Dependencies in infrastructure (Locations. AD Entities.2011 25 . preferences.09. order.)  Supports guided user input  Reduces flow complexity Titel der Präsentation und Name des Redners 30. Networks.Private Database enhances HP OO standard features Transactional Data IP-NameTable Persistence Table Server Name Mgmt. etc. SW Packages.

2011 26 . default selections  Limit selections to valid combinations RBA_Entities_RBA_Entities_RBA_Entities ID int <pk> Entity_ID int <fk1> Entity_Entity_ID int <fk2> Description varchar(512) RBA_Types ID int <pk> Type varchar(512) Description varchar(512) RBA_Entities ID int <pk> Type_ID int <fk> Name varchar(512) Description varchar(512) RBA_Entities_Extended_Attributes ID int <pk> Entity_ID int <fk> Name varchar(512) Value varchar(512) Description varchar(512)  Dynamic selections in Controller flows based on  System architecture. …  Preferences (e.09.g. usage. location. prefer one ESX Cluster whenever possible)  Control software installation flows  Software versions/editions  Installation order  Required and optional packages  Staging data from DEV to PRD RBA_Entities_RBA_Entities ID int <pk> Parent_Entity_ID int <fk2> Child_Entity_IDint <fk1> Description varchar(512) RBA_Entities_RBA_Entities_Extended_Attributes ID int <pk> Entitiy_Entity_ID int <fk> Name varchar(512) Value varchar(512) Description varchar(512)  No “hard encoded” values in flows  Reduce flow complexity – data sets defined by SQL queries  Less need to change to flows – edit database or SQL statement only  Database changes are immediately available in all affected flows Titel der Präsentation und Name des Redners 30.Infrastructure Topology Model stores information in a structured easy to maintain form RBA_Entities_RBA_Entities_RBA_Entities_Extended_Attributes ID int <pk> Entitiy_Entitiy_Entity_ID int <fk> Name varchar(512) Value varchar(512) Description varchar(512) Main purposes  Enhance User Input flows  Content of list boxes.

2011 27 .Interaction between flows and Private DB User Interface Flows Dialog Step 1 SQL SQL Dialog Step 2 Execution Flows Summary Step SQL Controller Flow HP OO Scheduler SQL Private DB Logical Logging Table Persistence Table Infrastructure Topology Model Titel der Präsentation und Name des Redners 30.09.

2011 28 . Groups Owner Group Titel der Präsentation und Name des Redners 30.Dependencies virt. Windows Installation – simple view Vmware Network Label AD Domains Network DNS Domains Vmware Datastore Suffix Network Segments Vmware Cluster Vmware Datastore Exclude Sites Vmware Datacenter Vmware vCenter Location Name Hosting Site Windows SW Depot Windows Server Packages Windows Server Role Usage Cost Center Backup Environment Type Vmware Resource Pool Windows Server OS Edition Job Mgmt Server Windows Server Image Hostname Prefix Windows Server OS Version Windows Server Add.09.


09.Input dialog with list boxes – supports the user input and avoids typos Step 1 Step 2 Step 5: Input Summary Step by Step Input  Most input is selected from list boxes  Comments support user Titel der Präsentation und Name des Redners 30.2011 30 .

Only valid combinations can be selected – less room for errors Step 1 Step 2 Limited selections Infrastructure Topology Model Cost Center and description from database Titel der Präsentation und Name des Redners 30.2011 31 .09.

getIpByName AP.vm.deployHost AP.datastore.vm.datastore.vmware4.getLargest 7/21/2011 2:27:54 PM -30 7/21/2011 2:27:55 PM -30 C1234567 C1234567 n059307 n059307 mucs801126 mucs801126 Start Success 7/21/2011 2:46:34 PM -30 C1234567 n059307 mucs801126 Start AP.dtm.getLargest 7/21/2011 2:27:54 PM -30 C1234567 n059307 mucs801126 Success AP.getDetails AP.getNewHostname AP.09.getNewHostname AP.customizationspec.deploy 7/21/2011 2:27:55 PM -30 C1234567 n059307 mucs801126 Start AP.2011 32 .vmware4.infoblox.vmware4.vm.vm.create Comment Start of VM Create Start of VM Deploy Windows Deployment failed! ServerDeploy failed! Titel der Präsentation und Name des Redners 30.vm.vmware4.setPowerState SC.vm.infoblox.task.deploy AP.vm.customizationspec.waitUntilFinished 7/21/2011 2:46:34 PM -30 C1234567 n059307 mucs801126 Success AP.getIpByName AP.infoblox.vm.deployHost CO.vmware4.Detailed logical reporting of all changes to infrastructure UUID = 7a18cc03-5cba-4d2b-9eab-1b594f3b38bc Timestamp RunID 7/21/2011 2:27:50 PM -30 7/21/2011 2:27:50 PM -30 7/21/2011 2:27:51 PM -30 7/21/2011 2:27:51 PM -30 7/21/2011 2:27:51 PM -30 7/21/2011 2:27:51 PM -30 7/21/2011 2:27:51 PM -30 7/21/2011 2:27:51 PM -30 7/21/2011 2:27:51 PM -30 ChangeID C1234567 C1234567 C1234567 C1234567 C1234567 C1234567 C1234567 C1234567 C1234567 Requester n059307 n059307 n059307 n059307 n059307 n059307 n059307 n059307 n059307 CI mucs801126 mucs801126 mucs801126 mucs801126 mucs801126 mucs801126 mucs801126 Status Start Start Success Start Success Success Start Start VM not found Flow CO.vmware4.vm.vmware4.task.vmware4.vm.setCpuAndMemorySize 7/21/2011 2:46:49 PM -30 C1234567 n059307 mucs801126 Success AP.vm.getDetails 7/21/2011 2:27:51 PM -30 C1234567 n059307 mucs801126 Start 7/21/2011 2:27:52 PM -30 C1234567 n059307 mucs801126 Success AP.vmware4.vm.modify 7/21/2011 2:27:52 PM -30 C1234567 n059307 mucs801126 Start AP.vmware4.addDownTimeWindow SC.vmware4.create AP.infoblox.vmware4.setCpuAndMemorySize 7/21/2011 2:46:49 PM -30 7/21/2011 2:46:59 PM -30 7/21/2011 3:05:18 PM -30 7/21/2011 3:05:18 PM -30 C1234567 C1234567 C1234567 C1234567 n059307 n059307 n059307 n059307 mucs801126 mucs801126 mucs801126 mucs801126 Start Success Failure Failure AP.vmware4.setPowerState AP.vmware4.vmware4.

2011 33 .HP OO executes functions on infrastructure components HP OO flows Infoblox Network DTM i3S Change OVO Monitoring planned SCCM HP OO i3S CMDB $Univers Job Mgmt VMware vSphere i3S Incident SCOM Monitoring Netapp Storage & Fabric AD Directory Service HP SA Networker Backup Windows / Linux OS DPPM  Reuse for other datacenters requires standards  Similar infrastructure systems and architecture  Management and administration of systems designed to support automation  Common ITIL management tools and processes Titel der Präsentation und Name des Redners 30.09.

make available to DIPRO/AO.and mums-servers in most common networks Not covered: Add new server to DTM Tool Add new server to Networker DB Reserve IP Address and IP Name for mucs/-mums-Servers Users: dedicated Team members Permissions currently limited to four networks/VLANs Create/revert/delete VMware Windows snapshots Users: BA Service Limited usage.2011 34 . W2S Servers Users: DMG. SAP Basis Installation of Oracle Databases Users: Oracle Service Promote AD Member Server to Domain Controller Users: AD Service Automation for HVD data store administration (later reuse for Netapp Storage London) Users: Storage Service.NET? Planned Installation of virt. SAP Basis Titel der Präsentation und Name des Redners 30.Status flow customizing Topic Scope Released for Munich DC Installation of managed virtual Windows servers Users: IT Operations mucs. AI Installation of Managed virtual Linux Server For SAP virtualization 50 – 70 Linux installations required from November Users: Linux Service.09. Windows Service Automation of simple changes to $Universe backup jobs Users: Job Mgmt. Service.