7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011

DATA CENTER O&M BEST PRACTICES
Presented by: Terry Rodgers, CPE, CPMP

Vice President, Sustainable Operations Primary Integration Solutions, Inc. trodgers@primaryintegration.com
Agenda
Data Center O&M Best Practices Perspectives Designing for Maintainability Commissioning, Construction, Startup & Testing O&M Staffing and Organization Operations & Maintenance Processes Existing Facilities
www.primaryintegration.com
Data Center O&M Best Practices

As the robustness & complexity of critical facilities has increased for improved fault-tolerance and concurrent maintenance, the importance of equally robust O&M practices has become apparent. As much as 75% or more impact events where critical missions have been compromised are associated with human Activity.
Perspectives
A&E GC & Contractors Mfgrs & Vendors CxA Owner Programming, engineer & design, submittals & RFIs, CA 6 to 9 months of intense effort then some minor CA support Propose, purchase, construct, punch/commission 9 to 15 months with minor follow-up effort Submittals/shop dwgs, assemble, startup & commission 3 to 6 months and a 12 month warranty + SLAs Design, construction, startup, acceptance, training & documentation 9 to 15 months plus 10 month pre-warranty return effort Design, construct, commission, transition, sustained operations 15 months design/construction, 12 months transition, 24 years operations
Perspectives
The A&Es, GC & Contractors, Manufacturers and Vendors, and Commissioning Agents completion date (i.e., project close-out) is the Owners start date (i.e., Day-1 of mission critical operations) Dont let Sustainable Operations be an after-thought!
Designing for Maintainability

Include facility O&M considerations in the programming phase Define facility O&M needs and expectations in the Basis-of-Design Specify the associated O&M deliverables in the Construction Documents

Include O&M considerations in the programming phase via the Owners Project Requirements (OPR) document
O&M Organization, Reporting Structure, & Budget
O&M Staffing (24x7 Coverage, Trades, Quals, etc.) In-House, Outsourced, SLA, or a combination Training Program (Day-1, New Hire, & Recurring) O&M Programs & Processes CMMS (new or expand existing) Asset Management System (inventory control) Configuration Management System Comprehensive As-Built Documentation & Doc Control
Facilities Management space-planning (offices, command center, shops, etc.)

The Basis-of-Design (BOD) document should capture O&M requirements Standardized naming conventions across CMMS, AMS, and CMS Standardized color-coding and labeling conventions Equipment submittals and start-up checklists that capture input data for CMMS and AMS (consumables, spare parts, PM structure, etc.) O&M Staffing & Facilities Management Facilities Command Center & event notification & escalation Shops & work-areas (mech, elect, controls, carpenter, landscape, etc.) General storage (outdoor, inside, environmental, secure, etc.) Specialized storage (Hazmat, flammable, & other special storage)

CDs (Drwgs & Specs) should define O&M deliverables as part of the overall construction project Equipment, System, & Space naming and color conventions Signage, Bar-coding, RFID, etc. A vs. B plants (mechanical & electrical) should be easily differentiated Color code systems (mechanical, electrical, controls, critical vs. non-critical, etc.) Valve and switch tagging Drawings should include Valve and Switch schedules (tables) Specifications should dictate size, material, location, ease-of-view, etc. Maintenance and access requirements Shown on drawings, permanently identified in field Access panels with labels where covered/closed-in Heavy equipment transport including truck/forklift routes, overhead rigging, pick-points Emergency lighting (response vs. egress) Storage and spare parts requirements
10

Most (if not all) Data Centers are designed with the expectation they will be expanded or modified during their lifespan The design must accommodate this expectation Appropriate means to isolate infrastructure between production from expansion Avoid loss of redundancy schemes during construction These situations increase the risk of impact events occurring and are when quality O&M processes are most important O&M staff well trained and practiced at reconfiguring systems and isolating equipment and distribution paths Updated and accurate as-built docs and operating procedures Critical work authorization review and approval processes Routine means to test/verify isolation points (valve leak tests, breaker tests, etc.)
11

Specifications should include defining close-out documentation As-Built (vs. redlines) and exact content and formats Redlines maintained throughout construction Formal Record Drawings (As-Built) in hardcopy & electronic Field verified for accuracy prior to acceptance
O&M Training details, deliverables, content, effort, and quality Systems Operations & Maintenance Manuals (vs. Manufacturers O&M manuals)
12

Systems Operations & Maintenance Manuals (SOMMs) should be organized by category (HVAC, FP, Security, BMS, etc.), system, & sub-system Include for each system: Descriptive narrative Single-line diagrams (one for each operating mode and/or configuration) Sequences-of-Operations, setpoints, alarm thresholds, safeties, etc. BMS points lists (DIs, DOs, AIs, AOs, etc.) Standard Operating Procedures (Normal, Maint., Emergency, Recovery, etc.) Approved submittals Commissioning scripts including baseline data Spare Part Lists (including on-site inventory) Warranties Service Level Agreements
13

Specifications should define O&M Training Curriculum, Scope, & Content Venues (classroom, shop, on-the-job) Instructor qualifications Duration and times (i.e., shift coverage) Testing and/or certifications Safety considerations (OSHA, Arc Flash, PPE, etc.) Operating procedures (normal, maintenance, emergency, & recovery modes) Equipment level, Systems level, and Integrated Systems level scenarios Maintenance procedures (inspections, preventive, predictive, corrective, replacement, etc.) NOTE: Commissioning affords excellent training opportunities
14
Commissioning, Construction, Startup & Testing

The overarching goal is to deliver the site, staff, and processes totally constructed and provisioned on Day-1 to support 24x7 mission uptime for the life of the facility. Commissioning Process Design Phase Commissioning Process Strategy Use the Commissioning Process to Provision for O&M Transition from Construction to Production ASHRAE Guideline 0-2005 The Commissioning Process is an excellent reference and user-friendly guide geared towards new construction. A sister guideline is currently being developed for commissioning existing facilities.
16

Commissioning process (Cx is not a procedure) Design Phase Ensure the OPR captures construction QC, performance validation, and facility O&M considerations Ensure the BOD complies with the OPR and defines high-level requirements Ensure the CDs comply with the BOD and include clear and concise specifications Ensure the CDs capture the requirements and effort associated with Commissioning (including documentation & training) Develop the Commissioning Plan to ensure the built product can be operated, maintained, and tested to validate performance
17

Commissioning Process Strategy Identify issues and discrepancies as early as possible so they can be resolved with least impact on schedule, resources, and costs Factory Witness Tests identify issues prior to shipment Field Component Verification identifies issues upon receipt and ensure proper storage & care prior to installation Systems Construction Verification (progress inspections) identify issues prior to start-up and functional testing Site Acceptance Testing identifies issues at the component, equipment, and system level before testing interfaces Integrated Systems Testing identifies issues at system interfaces and facility-wide level prior to going live.
18

Use the Commissioning Process to provision for O&M Use the Commissioning design review process to ensure the OPR, BOD, and CDs capture O&M needs O&M staff participation in commissioning (FWTs, inspections, and testing) affords in-depth ops training and establishes contacts with the EOR, mfgrs/vendors, and contractors Submittals, Field Component Verification, equipment startup forms, and completed test scripts can capture and document data for input to the CMMS, AMS, & CMS and spare parts collection & inventory Functional Tests and Integrated Systems Tests set the original baseline for performance
19

Use the Commissioning Process to provision for O&M The Commissioning Process includes ensuring the built facility is thoroughly documented System Operations & Maintenance Manuals should be developed concurrently with the construction phase
SOMMs are a compilation of data and documents from:

OPR, BOD, and CDs Record Drawings (As-Builts) Approved Submittals Final Commissioning Report & completed test scripts Warranties, SLAs, etc.
20

Use the Commissioning Process to provision for O&M Training Start with a comprehensive Training Plan Develop a Training Program that is site and client specific Embed training in the construction and acceptance testing phases Utilize the site SOMMs to support and supplement training materials Design training to match O&M staff position descriptions Integrate project specific technical training with Corporate policy & procedures training Training material should be editable so it can be updated and kept current Training delivery should be recorded for future use (new-hires, remedial training, periodic re-training, and refresher training prior to infrequently performed activities)
21
O&M Staffing and Organization

Remember the following should be addressed during the programming phase and should influence how the facility is designed and built!
O&M Staffing & Organization

Use an Enterprise reporting structure Critical Facilities O&M should be a department within the IT division Fundamentally eliminate the silos Recognize the mutual support and dependence of IT and Facilities on each other Combined goals drives collaboration and planning (sink or swim together) Eliminate competition for resources (staff & $$$) Combine NOC and Facilities Command Center operations into an Enterprise Command & Control Center Formalize and communicate load vs. capacity tracking, trending, and management
23

Staff the O&M department to meet site specific needs Start with a site specific O&M Job Analysis (dont confuse jobs with positions) Compile list of Jobs (tasks) required to O&M the site infrastructure Differentiate between operations and maintenance tasks Develop Job Descriptions that group tasks and assess what requisite skills and knowledge are required Identify all that pertain to critical facilities Consider what qualifications, certifications, or other credentials are necessary where appropriate Decide how critical vs. non-critical aspects will be structured Decide how operations vs. maintenance aspects will be structured
24

Staff the O&M department to meet site specific needs - continued Develop Position Descriptions Position Descriptions describe the jobs and associated tasks, duties and responsibilities expected for a specific staff position Assemble with the following considerations:

Combined O&M or separate Operations from Maintenance Jack-of-all-trades or Trade-specific (mechanic, electrician, controls tech, etc.) True 24x7 coverage or Business Day + On-Call for off hours
Compile a list of duties and responsibilities based on the tasks and associated skills and knowledge from the Job descriptions
Decide how to staff each position (Direct employees, in-house contractors, SLAs, or a combination)
25

Staff the O&M department to meet site specific needs - continued Develop an Organization and Reporting Structure Enterprise level organization Facilities O&M level organization Critical Facilities vs. Non-Critical Facilities Operations with shift coverage Maintenance broken down by trades

Electrical (Med Volt, Low Volt, Gensets, UPS, etc.) Mechanical (Central Plant, HVAC, Chem Treatment, etc) Monitoring & Controls (BMS, EPMS, calibrations, etc.) Vendors, Contractors, SLAs, etc.
Administration (CMMS, procurement, work approvals, etc.)
26
Operations & Maintenance Processes

Start with well defined Facilities Management mission Develop a mission or Vision statement Create a Strategic Plan with well defined goals and tactical objectives Develop programs and processes first, then supporting policies & procedures
28

Establish Operations Program and Processes Work authorization process (routine, PM, emergency, etc.) Event notification & escalation protocols Define Operations Key Performance Indicators (KPIs) What defines operational excellence and how is it measured? Critical vs. Non-critical boundary and operating protocols Practice (dry-runs) Double verification Mimic boards or other real-time configuration status Valve and switch position status tags Lock-out/Tag-out and other Safety Programs
29

Establish Operations Procedures Specific to each system Normal, maintenance (configuration), emergency, and recovery procedures Include pre-requisites and cautions Work authorization requirements Safety considerations (lock-out/tag-out, PPE, etc.) BMS and alarm lock-outs Verification of status of redundant or backup systems, etc.
30

Establish Operations Procedures - Continued Provide step-by-step procedure including action to be taken, expected result (alarm, indication, action, etc.), and means to collect resulting data (actual alarm, reading, action, etc.). Embed digital photos of equipment, components, and especially displays and control panels Annotate with arrows and relate to specific steps, actions, readings, etc. Consider using a flowchart format Include contingency and direction if expected does not match predicted Include completion/closeout checklist to ensure all aspects of system are restored to normal operating status (reactivate BMS, remove switch tags, lockouts, notify NOC, etc.)
31

Establish Maintenance Program and Processes Define Maintenance Key Performance Indicators (KPIs) What defines maintenance excellence and how is it measured? Develop programs to manage assets, configurations, and documentation
Asset Management System (AMS) Bar-code equipment and components Computerize inventory, location, and who is authorized to pull stock Include shelf-life, PMs (rotate shafts, etc.), and other storage considerations Source, vendors, warranties, spec sheets, etc.
32

Establish Maintenance Processes - Continued Configuration Management System (CMS) Computerized database of deployed assets & interconnectivity Sum loads and capacities to track & alarm at defined thresholds Can be integrated with Work Flow processes so asset deployment, relocation, or decommissioning automatically generates updates to CMS Can combine both IT, Network, and Facilities infrastructure Utilize actual real-time metering (BMS, branch-circuit monitoring, Tivoli, HP Openview, etc.) where available Use eDiscovery for auto audits where possible Set thresholds low enough to ensure redundancy and fail-over strategies remain valid
33

Establish Maintenance Processes - Continued Computerized Maintenance Management System (CMMS) Implementation tool for the Maintenance Program Include install date, warranty duration, lifespan/replacement date Provides input to develop and maintain 5-year & 10-year Capital Plans Meet or exceed manufacturer recommendations Stagger dates for work load leveling, seasonal considerations, & freeze windows Tie to Maintenance Procedures Separate daily, weekly, monthly, quarterly, etc Develop logical ID convention that facilitates filtering, searching, and reporting ease (from design phase!!!)
34

Establish Maintenance Processes - Continued Types of Maintenance Programs Corrective run to failure Preventive routine based on frequency or run-hours Predictive On-line condition monitoring with thresholds; extrapolate trends to predict needs (predict failures) Vibration analysis (snap-shot or continuous via multiplex) Thermography (IR Scans) and temperature sensors Tribology (lubricant analysis) Laser Alignment Computerized Balancing Combination of all above Input field notes and create equipment histories and review routinely
35

Establish Maintenance Procedures Specific to each system and equipment Separate procedure for daily, weekly, monthly, quarterly, etc. Include pre-requisites and cautions (similar to Ops procedures) Provide step-by-step procedure (similar to Ops procedures) Include acceptance criteria, tolerances, settings, etc. Include access considerations (ladder, confined space, PPE, key#) Include ability to take field notes, comments, technician ID, etc. Include completion checklist to ensure all aspects of system are restored to normal operating status (reactivate BMS, remove switch tags, lockouts, etc)
36

Implement BIMS (Building Information Modeling System) BIMS products are still relatively new to the industry BIMS software uses databases to link electronic data, records, and documents including AutoCAD drawings BIMS includes virtual 3D models in layers such that you can simulate the construction and identify conflicts, fouls, and sequence conflicts BIMS can link AutoCAD objects to associated specs, quantities, suppliers/vendors, submittals, warranties, inventory, ordering info, maintenance history, and almost any other electronic record BIMS can link CDs to CMMS, AMS, CMS, etc. BIMS implementation, use, and upkeep requires significant pre-planning and resource allocation
37

Existing Facilities Build / improve programs to support management, staff, and infrastructure using the Sustainable Operations Services (SOS) Methodology The SOS Methodology: 1) Define the requirements and establish acceptance criteria 2) Evaluate / Assess actual conditions and develop a gap report 3) Make recommendations to bridge the gap 4) Build the needed program(s) based on client/site-specific Best Practices 5) Build the associated processes and procedures 6) Train management and staff on the program(s), processes, & procedures 7) Implement the program(s), processes, and procedures 8) Test, verify, audit the results 9) Reassess and apply Continuous Process Improvement
38
Data Center O&M Best Practices

As with Critical Facility design, there are few absolute right and wrong solutions. The O&M Best Practice for a given facility depends upon many (often conflicting or competing) variables. What is most important, is plan ahead, select a solution, and then execute in a high quality manner.
39
Sustainable Operations
Data Centers will undergo change during their lifespan (expansions, infrastructure upgrades, & technology refresh, etc.) Industry Best Practices will change and evolve Long-term Sustained Operations require programs with integral performance measurement and Continuous Process Improvement
40
Inspect What You Expect
Commission > Transition > Sustain
Comments Questions? 650 Castroand Street, Suite 120-495

Mountain View, CA 94041 Phone: 866-331-5491 For more Information: mnaugle@primaryintegration.com www.PrimaryIntegration.com Sustainable Operations requires optimal performance of management, staff, and infrastructure
THANK YOU! Primary Integration Solutions, Inc
41

7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011

Hochgeladen von

Copyright:

Verfügbare Formate

DATA CENTER O&M BEST PRACTICES

Presented by: Terry Rodgers, CPE, CPMP

Data Center O&M Best Practices

Designing for Maintainability

Designing for Maintainability

Designing for Maintainability

Designing for Maintainability

Designing for Maintainability

Designing for Maintainability

Designing for Maintainability

Designing for Maintainability

Designing for Maintainability

Commissioning, Construction, Startup & Testing

Commissioning, Construction, Startup & Testing

Commissioning, Construction, Startup & Testing

Commissioning, Construction, Startup & Testing

Commissioning, Construction, Startup & Testing

Commissioning, Construction, Startup & Testing

SOMMs are a compilation of data and documents from:

Commissioning, Construction, Startup & Testing

O&M Staffing and Organization

O&M Staffing & Organization

O&M Staffing & Organization

O&M Staffing & Organization

O&M Staffing & Organization

Administration (CMMS, procurement, work approvals, etc.)

Operations & Maintenance Processes

Operations & Maintenance Processes

Operations & Maintenance Processes

Operations & Maintenance Processes

Operations & Maintenance Processes

Operations & Maintenance Processes

Operations & Maintenance Processes

Operations & Maintenance Processes

Operations & Maintenance Processes

Operations & Maintenance Processes

Operations & Maintenance Processes

Operations & Maintenance Processes

Data Center O&M Best Practices

Inspect What You Expect

Commission > Transition > Sustain

Comments Questions? 650 Castroand Street, Suite 120-495

THANK YOU! Primary Integration Solutions, Inc

Das könnte Ihnen auch gefallen