Beruflich Dokumente
Kultur Dokumente
IBM PureFlex System and IBM Flex System Products and Technology
Describes the IBM Flex System Enterprise Chassis and compute node technology Provides details about available I/O modules and expansion options Explains networking and storage configurations
ibm.com/redbooks
International Technical Support Organization IBM PureFlex System and IBM Flex System Products and Technology October 2013
SG24-7984-03
Note: Before using this information and the product it supports, read the information in Notices on page xi.
Fourth Edition (October 2013) This edition applies: IBM PureFlex System IBM Flex System Enterprise Chassis IBM Flex System Manager IBM Flex System x220 Compute Node IBM Flex System x222 Compute Node IBM Flex System x240 Compute Node IBM Flex System x440 Compute Node IBM Flex System p260 Compute Node IBM Flex System p270 Compute Node IBM Flex System p24L Compute Node IBM Flex System p460 Compute Node IBM Flex System V7000 Storage Node IBM 42U 1100mm Enterprise V2 Dynamic Rack IBM PureFlex System 42U Rack and 42U Expansion Rack
Copyright International Business Machines Corporation 2012, 2013. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
Contents
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Now you can become a published author, too! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stay connected to IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii xiv xvi xvi xvi
Summary of changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii October 2013, Fourth Edition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii August 2013, Third Edition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii February 2013, Second Edition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii Chapter 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 IBM PureFlex System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 IBM Flex System overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.1 IBM Flex System Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.2 IBM Flex System Enterprise Chassis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2.3 Compute nodes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.2.4 Expansion nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.2.5 Storage nodes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.2.6 I/O modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.3 This book. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Chapter 2. IBM PureFlex System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Configurators for IBM PureFlex System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 PureFlex solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 PureFlex Solution for IBM i . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.2 PureFlex Solution for SmartCloud Desktop Infrastructure . . . . . . . . . . . . . . . . . . 2.4 IBM PureFlex System Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.1 Available Express configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.2 Chassis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.3 Compute nodes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.4 IBM Flex System Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.5 PureFlex Express storage requirements and options . . . . . . . . . . . . . . . . . . . . . . 2.4.6 Video, keyboard, mouse option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.7 Rack cabinet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.8 Available software for Power Systems compute nodes . . . . . . . . . . . . . . . . . . . . 2.4.9 Available software for x86-based compute nodes . . . . . . . . . . . . . . . . . . . . . . . . 2.5 IBM PureFlex System Enterprise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.1 Enterprise configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.2 Chassis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.3 Top-of-rack switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.4 Compute nodes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.5 IBM Flex System Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.6 PureFlex Enterprise storage options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Copyright IBM Corp. 2012, 2013. All rights reserved.
11 12 13 14 15 15 16 17 17 20 20 21 21 24 25 25 26 27 27 30 30 31 31 32 iii
2.5.7 Video, keyboard, and mouse option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.8 Rack cabinet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.9 Available software for Power Systems compute node . . . . . . . . . . . . . . . . . . . . . 2.5.10 Available software for x86-based compute nodes . . . . . . . . . . . . . . . . . . . . . . . 2.6 Services for IBM PureFlex System Express and Enterprise . . . . . . . . . . . . . . . . . . . . . 2.6.1 PureFlex FCoE Customization Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6.2 PureFlex Services for IBM i. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6.3 Software and hardware maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 IBM SmartCloud Entry for Flex system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 3. Systems management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Management network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Chassis Management Module. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Compute node management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Integrated Management Module II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Flexible service processor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.3 I/O modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 IBM Flex System Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.1 IBM Flex System Manager functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.2 Hardware overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.3 Software features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.4 User interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.5 Mobile System Management application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.6 Flex System Manager CLI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
34 35 35 35 36 37 38 38 39 41 42 43 43 44 46 47 47 48 49 50 50 54 58 66 66 68
Chapter 4. Chassis and infrastructure configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.1.1 Front of the chassis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 4.1.2 Midplane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 4.1.3 Rear of the chassis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 4.1.4 Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 4.1.5 Air filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 4.1.6 Compute node shelves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 4.1.7 Hot plug and hot swap components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 4.2 Power supplies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 4.3 Fan modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.4 Fan logic module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 4.5 Front information panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 4.6 Cooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 4.7 Power supply selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 4.7.1 Power policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 4.7.2 Number of power supplies required for N+N and N+1 . . . . . . . . . . . . . . . . . . . . . 95 4.8 Fan module population . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 4.9 Chassis Management Module. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 4.10 I/O architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.11 I/O modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 4.11.1 I/O module LEDs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 4.11.2 Serial access cable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 4.11.3 I/O module naming scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 4.11.4 Switch to adapter compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
iv
IBM PureFlex System and IBM Flex System Products and Technology
4.11.5 IBM Flex System EN6131 40Gb Ethernet Switch . . . . . . . . . . . . . . . . . . . . . . . 4.11.6 IBM Flex System Fabric CN4093 10Gb Converged Scalable Switch . . . . . . . . 4.11.7 IBM Flex System Fabric EN4093 and EN4093R 10Gb Scalable Switch . . . . . 4.11.8 IBM Flex System Fabric SI4093 System Interconnect Module . . . . . . . . . . . . . 4.11.9 IBM Flex System EN4091 10Gb Ethernet Pass-thru Module . . . . . . . . . . . . . . 4.11.10 IBM Flex System EN2092 1Gb Ethernet Scalable Switch . . . . . . . . . . . . . . . 4.11.11 IBM Flex System FC5022 16Gb SAN Scalable Switch. . . . . . . . . . . . . . . . . . 4.11.12 IBM Flex System FC3171 8Gb SAN Switch . . . . . . . . . . . . . . . . . . . . . . . . . . 4.11.13 IBM Flex System FC3171 8Gb SAN Pass-thru. . . . . . . . . . . . . . . . . . . . . . . . 4.11.14 IBM Flex System IB6131 InfiniBand Switch . . . . . . . . . . . . . . . . . . . . . . . . . . 4.12 Infrastructure planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.12.1 Supported power cords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.12.2 Supported PDUs and UPS units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.12.3 Power planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.12.4 UPS planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.12.5 Console planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.12.6 Cooling planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.12.7 Chassis-rack cabinet compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.13 IBM 42U 1100mm Enterprise V2 Dynamic Rack . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.14 IBM PureFlex System 42U Rack and 42U Expansion Rack . . . . . . . . . . . . . . . . . . . 4.15 IBM Rear Door Heat eXchanger V2 Type 1756 . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 5. Compute nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 IBM Flex System Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 IBM Flex System x220 Compute Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.2 Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.3 Chassis support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.4 System architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.5 Processor options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.6 Memory options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.7 Internal disk storage controllers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.8 Supported internal drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.9 Embedded 1 Gb Ethernet controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.10 I/O expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.11 Integrated virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.12 Systems management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.13 Operating system support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 IBM Flex System x222 Compute Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.2 Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.3 Chassis support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.4 System architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.5 Processor options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.6 Memory options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.7 Supported internal drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.8 Expansion Node support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.9 Embedded 10Gb Virtual Fabric adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.10 Mid-mezzanine I/O adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.11 Integrated virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.12 Systems management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.13 Operating system support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 IBM Flex System x240 Compute Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
117 121 129 136 142 144 148 155 158 160 161 161 162 162 168 169 169 170 172 178 180 185 186 186 186 190 190 191 193 193 201 206 209 209 211 211 215 216 216 219 219 220 222 223 225 226 226 228 231 232 234 234
Contents
5.4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.2 Features and specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.3 Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.4 Chassis support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.5 System architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.6 Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.7 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.8 Standard onboard features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.9 Local storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.10 Integrated virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.11 Embedded 10 Gb Virtual Fabric adapter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.12 I/O expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.13 Systems management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.14 Operating system support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 IBM Flex System x440 Compute Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.2 Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.3 Chassis support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.4 System architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.5 Processor options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.6 Memory options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.7 Internal disk storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.8 Embedded 10Gb Virtual Fabric. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.9 I/O expansion options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.10 Network adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.11 Storage host bus adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.12 Integrated virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.13 Light path diagnostics panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.14 Operating systems support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 IBM Flex System p260 and p24L Compute Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6.1 Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6.2 System board layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6.3 IBM Flex System p24L Compute Node. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6.4 Front panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6.5 Chassis support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6.6 System architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6.7 Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6.8 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6.9 Active Memory Expansion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6.10 Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6.11 I/O expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6.12 System management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6.13 Operating system support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7 IBM Flex System p270 Compute Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7.1 Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7.2 System board layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7.3 Comparing the p260 and p270 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7.4 Front panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7.5 Chassis support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7.6 System architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7.7 IBM POWER7+ processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7.8 Memory subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7.9 Active Memory Expansion feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi
IBM PureFlex System and IBM Flex System Products and Technology
235 237 239 239 240 242 245 258 259 266 268 269 271 274 275 275 278 279 280 281 282 284 290 291 294 295 295 296 297 298 298 301 301 302 304 304 305 308 310 313 315 316 317 318 319 320 321 322 323 324 325 327 329
5.7.10 Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7.11 I/O expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7.12 System management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7.13 Operating system support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8 IBM Flex System p460 Compute Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8.2 System board layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8.3 Front panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8.4 Chassis support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8.5 System architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8.6 Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8.7 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8.8 Active Memory Expansion feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8.9 Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8.10 Local storage and cover options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8.11 Hardware RAID capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8.12 I/O expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8.13 System management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8.14 Integrated features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8.15 Operating system support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.9 IBM Flex System PCIe Expansion Node. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.9.1 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.9.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.9.3 Supported PCIe adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.9.4 Supported I/O expansion cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.10 IBM Flex System Storage Expansion Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.10.1 Supported nodes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.10.2 Features on Demand upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.10.3 Cache upgrades . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.10.4 Supported HDD and SSD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.11 I/O adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.11.1 Form factor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.11.2 Naming structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.11.3 Supported compute nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.11.4 Supported switches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.11.5 IBM Flex System EN2024 4-port 1Gb Ethernet Adapter. . . . . . . . . . . . . . . . . . 5.11.6 IBM Flex System EN4132 2-port 10Gb Ethernet Adapter. . . . . . . . . . . . . . . . . 5.11.7 IBM Flex System EN4054 4-port 10Gb Ethernet Adapter. . . . . . . . . . . . . . . . . 5.11.8 IBM Flex System EN6132 2-port 40Gb Ethernet Adapter. . . . . . . . . . . . . . . . . 5.11.9 IBM Flex System CN4054 10Gb Virtual Fabric Adapter . . . . . . . . . . . . . . . . . . 5.11.10 IBM Flex System CN4058 8-port 10Gb Converged Adapter . . . . . . . . . . . . . 5.11.11 IBM Flex System EN4132 2-port 10Gb RoCE Adapter. . . . . . . . . . . . . . . . . . 5.11.12 IBM Flex System FC3172 2-port 8Gb FC Adapter . . . . . . . . . . . . . . . . . . . . . 5.11.13 IBM Flex System FC3052 2-port 8Gb FC Adapter . . . . . . . . . . . . . . . . . . . . . 5.11.14 IBM Flex System FC5022 2-port 16Gb FC Adapter . . . . . . . . . . . . . . . . . . . . 5.11.15 IBM Flex System FC5024D 4-port 16Gb FC Adapter . . . . . . . . . . . . . . . . . . . 5.11.16 IBM Flex System FC5052 2-port and FC5054 4-port 16Gb FC Adapters. . . . 5.11.17 IBM Flex System FC5172 2-port 16Gb FC Adapter . . . . . . . . . . . . . . . . . . . . 5.11.18 IBM Flex System IB6132 2-port FDR InfiniBand Adapter . . . . . . . . . . . . . . . . 5.11.19 IBM Flex System IB6132 2-port QDR InfiniBand Adapter. . . . . . . . . . . . . . . . 5.11.20 IBM Flex System IB6132D 2-port FDR InfiniBand Adapter. . . . . . . . . . . . . . .
329 333 333 334 335 335 338 338 340 341 342 345 349 350 351 353 353 354 355 355 356 357 359 361 362 363 364 366 367 368 370 371 372 373 374 376 377 378 380 381 384 387 389 391 393 394 396 398 400 401 403
Contents
vii
Choosing the Ethernet switch I/O module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Virtual local area networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Scalability and performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . High Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.1 Highly available topologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.2 Spanning Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.3 Link aggregation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.4 NIC teaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.5 Trunk failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.6 Virtual Router Redundancy Protocol. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 FCoE capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 Virtual Fabric vNIC solution capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6.1 Virtual Fabric mode vNIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6.2 Switch-independent mode vNIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7 Unified Fabric Port feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.8 Easy Connect concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.9 Stacking feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.10 Openflow support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.11 802.1Qbg Edge Virtual Bridge support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.12 SPAR feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.13 Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.13.1 Management tools and their capabilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.14 Summary and conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 7. Storage integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 IBM Flex System V7000 Storage Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.1 V7000 Storage Node types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.2 Controller Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.3 Expansion Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.4 SAS cabling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.5 Host interface cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.6 Fibre Channel over Ethernet with a V7000 Storage Node . . . . . . . . . . . . . . . . . 7.1.7 V7000 Storage Node drive options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.8 Features and functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.9 Licenses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.10 Configuration restrictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 External storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.1 IBM Storwize V7000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.2 IBM XIV Storage System series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.3 IBM System Storage DS8000 series. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.4 IBM System Storage DS5000 series. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.5 IBM System Storage V3700 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.6 IBM System Storage DS3500 series. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.7 IBM network-attached storage products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.8 IBM FlashSystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.9 IBM System Storage TS3500 Tape Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.10 IBM System Storage TS3310 series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.11 IBM System Storage TS3200 Tape Library . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.12 IBM System Storage TS3100 Tape Library . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Fibre Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 FC requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.2 FC switch selection and fabric interoperability rules . . . . . . . . . . . . . . . . . . . . . . 7.4 FCoE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
406 408 409 411 413 416 417 419 420 421 422 423 424 426 427 429 430 432 433 433 434 436 437 439 440 444 445 450 452 454 454 455 455 457 458 459 460 462 463 463 464 464 465 465 466 467 467 468 468 468 469 473
viii
IBM PureFlex System and IBM Flex System Products and Technology
iSCSI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . HA and redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Backup solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8.1 Dedicated server for centralized LAN backup. . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8.2 LAN-free backup for nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.9 Boot from SAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.9.1 Implementing Boot from SAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.9.2 iSCSI SAN Boot specific considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Abbreviations and acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483 Related publications and education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IBM education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485 485 486 486 487
Contents
ix
IBM PureFlex System and IBM Flex System Products and Technology
Notices
This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurements may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs.
xi
Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. These and other IBM trademarked terms are marked on their first occurrence in this information with the appropriate symbol ( or ), indicating US registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at http://www.ibm.com/legal/copytrade.shtml The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:
Active Cloud Engine Active Memory AIX AIX 5L AS/400 BladeCenter DB2 DS4000 DS8000 Easy Tier EnergyScale eServer FICON FlashCopy FlashSystem IBM IBM FlashSystem IBM Flex System IBM Flex System Manager IBM SmartCloud iDataPlex Linear Tape File System Netfinity POWER Power Systems POWER6 POWER6+ POWER7 POWER7+ PowerPC PowerVM PureApplication PureData PureFlex PureSystems Real-time Compression Redbooks Redbooks (logo) ServerProven ServicePac Storwize System Storage System Storage DS System x Tivoli Tivoli Storage Manager FastBack VMready X-Architecture XIV
The following terms are trademarks of other companies: Intel, Intel Xeon, Pentium, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Linear Tape-Open, LTO, Ultrium, the LTO Logo and the Ultrium logo are trademarks of HP, IBM Corp. and Quantum in the U.S. and other countries. Microsoft, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Java, and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. UNIX is a registered trademark of The Open Group in the United States and other countries. Other company, product, or service names may be trademarks or service marks of others.
xii
IBM PureFlex System and IBM Flex System Products and Technology
Preface
To meet todays complex and ever-changing business demands, you need a solid foundation of compute, storage, networking, and software resources. This system must be simple to deploy, and be able to quickly and automatically adapt to changing conditions. You also need to be able to take advantage of broad expertise and proven guidelines in systems management, applications, hardware maintenance, and more. The IBM PureFlex System combines no-compromise system designs along with built-in expertise and integrates them into complete, optimized solutions. At the heart of PureFlex System is the IBM Flex System Enterprise Chassis. This fully integrated infrastructure platform supports a mix of compute, storage, and networking resources to meet the demands of your applications. The solution is easily scalable with the addition of another chassis with the required nodes. With the IBM Flex System Manager, multiple chassis can be monitored from a single panel. The 14 node, 10U chassis delivers high-speed performance complete with integrated servers, storage, and networking. This flexible chassis is simple to deploy now, and to scale to meet your needs in the future. This IBM Redbooks publication describes IBM PureFlex System and IBM Flex System. It highlights the technology and features of the chassis, compute nodes, management features, and connectivity options. Guidance is provided about every major component, and about networking and storage connectivity. This book is intended for customers, Business Partners, and IBM employees who want to know the details about the new family of products. It assumes that you have a basic understanding of blade server concepts and general IT knowledge.
xiii
Authors
This book was produced by a team of specialists from around the world working at the International Technical Support Organization, Raleigh Center. David Watts is a Consulting IT Specialist at the IBM ITSO Center in Raleigh. He manages residencies and produces IBM Redbooks publications on hardware and software topics that are related to IBM Flex System, IBM System x, and BladeCenter servers and associated client platforms. He has authored over 200 books, papers, and Product Guides. He holds a Bachelor of Engineering degree from the University of Queensland (Australia), and has worked for IBM in the United States and Australia since 1989. David is an IBM Certified IT Specialist and a member of the IT Specialist Certification Review Board. Randall Davis is a Senior IT Specialist working in the System x pre-sales team for IBM Australia as a Field Technical Sales Support (FTSS) specialist. He regularly performs System x, BladeCenter, and Storage demonstrations for customers at the IBM Demonstration Centre in Melbourne, Australia. He also helps instruct Business Partners and customers on how to configure and install the BladeCenter. His areas of expertise are the IBM BladeCenter, System x servers, VMware, and Linux. Randall started at IBM as a System 36 and AS/400 Engineer in 1989. Dave Ridley is the PureFlex and Flex System Technical Product Manager for IBM in the United Kingdom and Ireland. His role includes product transition planning, supporting marketing events, press briefings, managing the UK loan pool, running early ship programs, and supporting the local sales and technical teams. He is based in Horsham in the United Kingdom, and has worked for IBM since 1998. In addition, he has been involved with IBM x86 products for 27 years.
Thanks to the authors of the previous editions of this book. Authors of the second edition, IBM PureFlex System and IBM Flex System Products and Technology, published in February 2013, were: David Watts Dave Ridley Authors of the first edition, IBM PureFlex System and IBM Flex System Products and Technology, published in July 2012, were: David Watts Randall Davis Richard French Lu Han Dave Ridley Cristian Rojas
xiv
IBM PureFlex System and IBM Flex System Products and Technology
Thanks to the following people for their contributions to this project: From IBM marketing: TJ Aspden Michael Bacon John Biebelhausen Mark Cadiz Bruce Corregan Mary Beth Daughtry Meleata Pinto Mike Easterly Diana Cunniffe Kyle Hampton From IBM development: Mike Anderson Sumanta Bahali Wayne Banks Barry Barnett Keith Cramer Mustafa Dahnoun Dean Duff Royce Espey Kaena Freitas Jim Gallagher Dottie Gardner Sam Gaver Phil Godbolt Mike Goodman John Gossett Tim Hiteshew Andy Huryn Bill Ilas Don Keener Caroline Metry Meg McColgan Mark McCool Rob Ord Greg Pruett Mike Solheim Fang Su Vic Stankevich Tan Trinh Rochelle White Dale Weiler Mark Welch Al Willard Botond Kiss Shekhar Mishra Sander Kim Dean Parker Hector Sanchez David Tareen David Walker Randi Wood Bob Zuber
From the International Technical Support Organization: Kevin Barnes Tamikia Barrow Mary Comianos Deana Coble Others from IBM around the world: Kerry Anders Simon Casey Bill Champion Jonathan A Tyrrell Others from other companies: Tom Boucher, Emulex Brad Buland, Intel Jeff Lin, Emulex Chris Mojica, QLogic Brent Mosbrook, Emulex Jimmy Myers, Brocade Haithuy Nguyen, Mellanox Brian Sparks, Mellanox Matt Wineberg, Brocade Michael L. Nelson Kiron Rakkar Matt Slavin Fabien Willmann Shari Deiana Cheryl Gera Ilya Krutov Karen Lawrence
Preface
xv
Comments welcome
Your comments are important to us! We want our books to be as helpful as possible. Send us your comments about this book or other IBM Redbooks publications in one of the following ways: Use the online Contact us review Redbooks form found at: ibm.com/redbooks Send your comments in an email to: redbooks@us.ibm.com Mail your comments to: IBM Corporation, International Technical Support Organization Dept. HYTD Mail Station P099 2455 South Road Poughkeepsie, NY 12601-5400
xvi
IBM PureFlex System and IBM Flex System Products and Technology
Summary of changes
This section describes the technical changes that were made in this edition of the book and in previous editions. This edition might also include minor corrections and editorial changes that are not identified. Summary of Changes for SG24-7984-03 for IBM PureFlex System and IBM Flex System Products and Technology as created or updated on October 15, 2013 3:46 pm.
New information
The following new products were added to the book: IBM PureFlex System Express IBM PureFlex System Enterprise IBM SmartCloud Entry 3.2 These products are described in Chapter 2, IBM PureFlex System on page 11. Important: The Flex System components that were announced in October 2013 will be covered in the next edition of this book.
New information
The following new products and options were added to the book: IBM Flex System x222 Compute Node IBM Flex System p260 Compute Node (POWER7+ SCM) IBM Flex System p270 Compute Node (POWER7+ DCM) IBM Flex System p460 Compute Node (POWER7+ SCM) IBM Flex System EN6132 2-port 40Gb Ethernet Adapter IBM Flex System FC5052 2-port 16Gb FC Adapter IBM Flex System FC5054 4-port 16Gb FC Adapter IBM Flex System FC5172 2-port 16Gb FC Adapter IBM Flex System FC5024D 4-port 16Gb FC Adapter IBM Flex System IB6132D 2-port FDR InfiniBand Adapter IBM Flex System Fabric SI4093 System Interconnect Module IBM Flex System EN6131 40Gb Ethernet Switch
xvii
New information
The following new products and options were added to the book: IBM SmartCloud Entry V2.4 IBM Flex System Manager V1.2 IBM Flex System Fabric EN4093R 10Gb Scalable Switch IBM Flex System Fabric CN4093 10Gb Converged Scalable Switch FoD license upgrades for the IBM Flex System FC5022 16Gb SAN Scalable Switch IBM PureFlex System 42U Rack 2100-W power supply option for the Enterprise Chassis New options and models of the IBM Flex System x220 Compute Node IBM Flex System x440 Compute Node Additional solid-state drive options for all x86 compute nodes IBM Flex System p260 Compute Node, model 23X with IBM POWER7+ processors New memory options for the IBM Power Systems compute nodes IBM Flex System Storage Expansion Node IBM Flex System PCIe Expansion Node IBM Flex System CN4058 8-port 10Gb Converged Adapter IBM Flex System EN4132 2-port 10Gb RoCE Adapter IBM Flex System V7000 Storage Node
Changed information
The following updates were made to existing product information: Updated the configurations of IBM PureFlex System Express, Standard, and Enterprise Switch stacking feature of Ethernet switches FCoE and iSCSI support
xviii
IBM PureFlex System and IBM Flex System Products and Technology
Chapter 1.
Introduction
During the last 100 years, information technology moved from a specialized tool to a pervasive influence on nearly every aspect of life. From tabulating machines that counted with mechanical switches or vacuum tubes to the first programmable computers, IBM has been a part of this growth. The goal has always been to help customers to solve problems. IT is a constant part of business and of general life. The expertise of IBM in delivering IT solutions has helped the planet become more efficient. As organizational leaders seek to extract more real value from their data, business processes, and other key investments, IT is moving to the strategic center of business. To meet these business demands, IBM has introduced a new category of systems. These systems combine the flexibility of general-purpose systems, the elasticity of cloud computing, and the simplicity of an appliance that is tuned to the workload. Expert integrated systems are essentially the building blocks of capability. This new category of systems represents the collective knowledge of thousands of deployments, established guidelines, innovative thinking, IT leadership, and distilled expertise. The offerings are designed to deliver value in the following ways: Built-in expertise helps you to address complex business and operational tasks automatically. Integration by design helps you to tune systems for optimal performance and efficiency. Simplified experience, from design to purchase to maintenance, creates efficiencies quickly. These offerings are optimized for performance and virtualized for efficiency. These systems offer a no-compromise design with system-level upgradeability. The capability is built for cloud, containing built-in flexibility and simplicity.
IBM PureFlex System is an expert integrated system. It is an infrastructure system with built-in expertise that deeply integrates with the complex IT elements of an infrastructure. This chapter describes the IBM PureFlex System and the components that make up this compelling offering and includes the following topics: 1.1, IBM PureFlex System on page 3 1.2, IBM Flex System overview on page 6 1.3, This book on page 10
IBM PureFlex System and IBM Flex System Products and Technology
Chapter 1. Introduction
The PureFlex System includes the following configurations: IBM PureFlex System Express, which is designed for small and medium businesses and is the most affordable entry point for PureFlex System. IBM PureFlex System Standard, which is optimized for application servers with supporting storage and networking, and is designed to support your key ISV solutions. IBM PureFlex System Enterprise, which is optimized for transactional and database systems. It has built-in redundancy for highly reliable and resilient operation to support your most critical workloads.
IBM PureFlex System and IBM Flex System Products and Technology
1 IBM Flex System Manager with 1-year service and support 2 2/6 4/8 Yes (redundant controller) Yes (redundant controller) Base with 1-year software maintenance agreement Optional Real Time Compression
1 IBM Flex System Manager Advanced with 3-year service and support 2 4/6 6/8 Yes (redundant controller) Yes (redundant controller) Base with 3-year software maintenance agreement Real Time Compression
1 Flex System Manager Advanced with 3-year service and support 2 6/6 8/8 Yes (redundant controller) Yes (redundant controller) Base with 3-year software maintenance agreement Real Time Compression
a. Select the IBM Flex System FC3171 8Gb SAN Switch or IBM Flex System FC5022 24-port 16Gb ESB SAN Scalable Switch module. b. Select the IBM Flex System V7000 Storage Node that is installed inside the Enterprise chassis or the external IBM Storwize V7000 Disk System.
The fundamental building blocks of the three IBM PureFlex System solutions are the compute nodes, storage nodes, and networking of the IBM Flex System Enterprise Chassis.
Chapter 1. Introduction
IBM PureFlex System and IBM Flex System Products and Technology
Resource pooling: Pooled network switching, with placement advisors that consider virtual machine (VM) compatibility, processor, availability, and energy. Intelligent automation: Automated and dynamic VM placement that is based on usage, hardware predictive failure alerts, and host failures. Figure 1-2 shows the IBM Flex System Manager appliance.
Chapter 1. Introduction
IBM PureFlex System and IBM Flex System Products and Technology
Figure 1-4 shows a four socket IBM POWER7 compute node, the p460.
The nodes are complemented with leadership I/O capabilities of up to 16 channels of high-speed I/O lanes per standard wide node bay and 32 lanes per full wide node bay. Various I/O adapters and matching I/O Modules are available.
Chapter 1. Introduction
Figure 1-5 IBM Flex System Fabric EN4093R 10Gb Scalable Switch
10
IBM PureFlex System and IBM Flex System Products and Technology
Chapter 2.
11
2.1 Introduction
IBM PureFlex System provides an integrated computing system that combines servers, enterprise storage, networking, virtualization, and management into a single structure. You can use its built-in expertise to manage and flexibly deploy integrated patterns of virtual and hardware resources through unified management. PureFlex System includes the following features: Configurations that ease acquisition experience and match your needs. Optimized to align with targeted workloads and environments. Designed for cloud with the SmartCloud Entry option. Choice of architecture, operating system, and virtualization engine. Designed for simplicity with integrated, single-system management across physical and virtual resources. Ships as a single integrated entity directly to you. Includes factory integration and lab services optimization. Revised in the fourth quarter of 2013, IBM PureFlex System now consolidates the three previous offerings (Express, Standard, and Enterprise) into two simplified pre-integrated offerings (Express and Enterprise) that support the latest compute, storage, and networking requirements. Clients can select from either of these offerings that help simplify ordering and configuration. As a result, PureFlex System helps cut the cost, time, and complexity of system deployments, which reduces the time to gain real value. Latest enhancements include support for the latest compute nodes, I/O modules, and I/O adapters with the latest release of software, such as IBM SmartCloud Entry with the latest Flex System Manager release. PureFlex 4Q 2013 includes the following enhancements: New PureFlex Express New PureFlex Enterprise New Rack offerings for Express: 25U, 42U (or none) New compute nodes: x222, p270, p460 New networking support: 10 GbE Converged New SmartCloud Entry v3.2 offering The IBM PureFlex System includes the following offerings: Express: An infrastructure system for small-sized and midsized businesses; the most cost-effective entry point with choice and flexibility to upgrade to higher function. For more information, see 2.4, IBM PureFlex System Express on page 17. Enterprise: An infrastructure system that is optimized for scalable cloud deployments with built-in redundancy for highly reliable and resilient operation to support critical applications and cloud services. For more information, see 2.5, IBM PureFlex System Enterprise on page 27.
12
IBM PureFlex System and IBM Flex System Products and Technology
2.2 Components
A PureFlex System configuration features the following main components: A preinstalled and configured IBM Flex System Enterprise Chassis. Choice of compute nodes with IBM POWER7, POWER7+, or Intel Xeon E5-2400 and E5-2600 processors. IBM Flex System Manager that is preinstalled with management software and licenses for software activation. IBM Flex System V7000 Storage Node or IBM Storwize V7000 external storage system. The following hardware components are preinstalled in the IBM PureFlex System rack: Express: 25U, 42U rack, or no rack configured Enterprise: 42U rack only Choice of software: Operating system: IBM AIX, IBM i, Microsoft Windows, Red Hat Enterprise Linux, or SUSE Linux Enterprise Server Virtualization software: IBM PowerVM, KVM, VMware vSphere, or Microsoft Hyper V SmartCloud Entry 3.2 (for more information, see 2.7, IBM SmartCloud Entry for Flex system on page 39) Complete pre-integrated software and hardware Optional onsite services available to get you up and running and provide skill transfer The hardware differences between Express and Enterprise are summarized in Table 2-1. The base configuration of the two offerings is shown that can be further customized within the IBM configuration tools.
Table 2-1 PureFlex System hardware overview configurations Components PureFlex Rack Flex System Enterprise Chassis Chassis power supplies/ fans Flex System Manager Compute nodes (one minimum) POWER or x86 based VMware ESXi USB key Top of rack switches Integrated 1GbE switch Integrated 10GbE switch Integrated 16Gb Fibre Channel Converged 10GbE switch (FCoE) IBM Storwize V7000 or V7000 Storage Node PureFlex Express Optional: 42U, 25U or no rack Required. Single chassis only 2/6 Required p260, p270, p460, x220, x222, x240, x440 Selectable on x86 nodes Optional: Integrated by client Selectable (redundant) Selectable (redundant) Selectable (redundant) Selectable (Redundant or non-redundant) Required & selectable PureFlex Enterprise Required: 42U Rack Required: Multi-chassis, 1, 2, or 3 chassis 4/8 Required p260, p270, p460, x220, x222, x240, x440 Selectable on x86 nodes Integrated by IBM Selectable (redundant) Selectable (redundant) Selectable (redundant) Selectable (redundant) Required and selectable
13
PureFlex System software can also be customized in a similar manner to the hardware components of the two offerings. Enterprise has a slightly different composition of software defaults than Express, which are summarized in Table 2-2.
Table 2-2 PureFlex software defaults overview Software Storage Flex System Manager (FSM) IBM Virtualization Virtualization customer installed Operating systems Express Enterprise
Storwize V7000 or Flex System V7000 Base Real Time Compression (optional) FSM Standard Upgradeable to Advanced PowerVM Standard Upgradeable to Enterprise FSM advanced Selectable to Standarda PowerVM Enterprise Selectable to Standard
VMware, Microsoft Hyper-V, KVM, Red Hat, and SUSE Linux AIX Standard (V6 and V7), IBM i (7.1, 6.1). RHEL (6), SUSE (SLES 11) Customer installed: Windows Server, RHEL, SLES Power SC Standard (AIX only) Tivoli Provisioning Manager (x86 only) SmartCloud Entry (optional) Standard 1 year, upgradeable to three years
14
IBM PureFlex System and IBM Flex System Products and Technology
15
16
IBM PureFlex System and IBM Flex System Products and Technology
Configurations
There are seven different configurations that are orderable within the PureFlex express offering. These offerings cover various redundant and non-redundant configurations with the different types of protocol and storage controllers.
17
1A
10 GbE
2A
10 GbE
2B
10GbE
3A
1 GbE
3B
1 GbE
4A
10 GbE
4B
10 GbE
FCoE
FCoE
FCoE
16 Gb
16 Gb
16 Gb
16 Gb
Storwize V7000
Storwize V7000
Storwize V7000
1 Chassis with 2 Chassis management modules, fans, and power supple units (PSUs) None or 42U or 25U (+PDUs) Optional DVD only DVD and Tape
Storage Options: (24 HDD, 22 HDD + 2 SSD, 20 HDD + 4 SSD or Custom) Storwize expansion (limit to single rack in Express, overflow storage rack in Enterprise), nine units per controller Up to two Storwize V7000 controllers and up to nine IBM Flex System V7000 Storage Nodes. VIOS, AIX, IBM i, and SCE on first Controller P260, p270, p460, x222, x240, x220, x440 CN4058 8-port 10Gb Converged Adapter EN2024 4-port 1Gb Ethernet Adapter EN4054 4-port 10Gb Ethernet Adapter
V7000 Content Nodes POWER Nodes Ethernet I/O Adapters POWER nodes Fibre Channel I/O Adapters x86 Nodes Ethernet I/O adapters
Not applicable
x86 Nodes Fibre Channel I/O Adapters ESXi USB Key Port FoD Activations IBM i PureFlex Solution
Not applicable
FC5022 16Gb 2-port Fibre Channel adapter FC3052 8Gb 2-port Fibre Channel adapter FC5024D 4-port Fibre Channel adapter
Optional with x86 Nodes Ports are computed during configuration based on chassis switch, node type, and the I/O adapter selection.
Not configur-abl e
Available
Available
Not configurable
Not configurable
18
IBM PureFlex System and IBM Flex System Products and Technology
Example configuration
There are seven configurations for PureFlex Express as described in Table 2-4 on page 18. Configuration 2B features a single chassis with an external V7000 Storwize controller. This solution uses FCoE and includes the Converged Switch module CN3093 to provide an FC Forwarder. This means that only converged adapters must be installed on the node and that the CN4093 breaks out Ethernet and Fibre Channel externally from the chassis. Figure 2-1 shows the connections, including the Fibre Channel and Ethernet data networks and the management network that is presented to the Access Points within the PureFlex Rack. The green box signifies the chassis and its components with the inter-switch link between the two switches. Because this is an Express solution, it is an entry configuration.
CMM
ISL
CMM
Access Points
Storwize V7000
Midplane Connections Management 1GbE Data 10GbE Data 40GbE Data 8Gb FC
Label Label
Figure 2-1 PureFlex Express with FCoE and external V7000 Storwize
19
2.4.2 Chassis
The IBM Flex System Enterprise Chassis contains all the components of the PureFlex Express configuration with the exception of the IBM Storwize V7000 and any expansion enclosure. The chassis is installed in a 25U or 42U rack. The compute nodes, storage nodes, switch modules and IBM Flex System Manager are installed in the chassis. When the V7000 Storage Node is chosen as the storage type, a no rack option is also available. Table 2-5 lists the major components of the Enterprise Chassis, including the switches and options. Feature codes: The tables in this section do not list all feature codes. Some features are not listed here for brevity.
Table 2-5 Components of the chassis and switches AAS feature code 7893-92X 7955-01M A0TF ESW7 ESW2 EB28 EB29 3286 3771 5370 9039 3592 XCC feature code 8721-HC1 8731-AC1 3598 A3J6 A3HH 5053 3268 5075 A2RQ 5084 A0TM A0UE Description IBM Flex System Enterprise Chassis IBM Flex System Manager IBM Flex System EN2092 1Gb Ethernet Scalable Switch IBM Flex System Fabric EN4093R 10Gb Scalable Switch IBM Flex System Fabric CN4093 10Gb Converged Scalable Switch IBM SFP+ SR Transceiver IBM SFP RJ45 Transceiver IBM 8Gb SFP+ Software Optical Transceiver IBM Flex System FC5022 24-port 16Gb ESB SAN Scalable Switch Brocade 8Gb SFP+ Software Optical Transceiver Base Chassis Management Module Additional Chassis Management Module
20
IBM PureFlex System and IBM Flex System Products and Technology
Description IBM Flex System p260 Compute Node (POWER7+) IBM Flex System p460 Compute Node IBM Flex System p460 Compute Node (POWER7+) IBM Flex System p270 Compute Node (POWER7+)
Table 2-7 x86 based compute nodes AAS feature code ECS7 ECSB 0457 ECSB MTM 7906-25X 7916-27X 7863-10X 7917-45X Description IBM Flex System x220 Compute Node IBM Flex System x222 Compute Node IBM Flex System x240 Compute Node IBM Flex System x440 Compute Node
21
Solid-state drives (SSDs) are optional. However, if they are added to the configuration, they are normally used for the V7000 Easy Tier function, which improves system performance.
The IBM Storwize V7000 consists of the following components, disk, and software options: IBM Storwize V7000 Controller (2076-124) SSDs: 200 GB 2.5-inch 400 GB 2.5-inch Hard disk drives (HDDs): 300 GB 2.5-inch 10K 300 GB 2.5-inch 15K 600 GB 2.5-inch 10K 800 GB 2.5-inch 10K 900 GB 2.5-inch 10K 1 TB 2.5-inch 7.2K 1.2 TB 2.5-inch 10K
Expansion Unit (2076-224): up to 9 per V7000 Controller IBM Storwize V7000 Expansion Enclosure (24 disk slots) Optional software: IBM Storwize V7000 Remote Mirroring IBM Storwize V7000 External Virtualization IBM Storwize V7000 Real-time Compression
22
IBM PureFlex System and IBM Flex System Products and Technology
The IBM Flex System V7000 Storage Node consists of the following components, disk, and software options: IBM Storwize V7000 Controller (4939-A49) SSDs: 200GB 2.5-inch 400GB 2.5-inch 800 GB 2.5-inch HDDs: 300GB 2.5-inch 10K 300GB 2.5-inch 15K 600GB 2.5-inch 10K 800GB 2.5-inch 10K 900GB 2.5-inch 10K 1 TB 2.5-inch 7.2K 1.2TB 2.5-inch 10K
Expansion Unit (4939-A29) IBM Storwize V7000 Expansion Enclosure (24 disk slots) Optional software: IBM Storwize V7000 Remote Mirroring IBM Storwize V7000 External Virtualization IBM Storwize V7000 Real-time Compression
23
The 7226 enclosure media devices offers support for SAS, USB, and Fibre Channel connectivity, depending on the drive. Support in a PureFlex configuration includes the external USB and Fibre Channel connections. Table 2-8 shows the Multi-Media Enclosure and available PureFlex options.
Table 2-8 Multi-Media Enclosure and options Machine type 7226 7226-1U3 7226-1U3 7226-1U3 Feature Code Model 1U3 5763 8248 8348 Description Multi-Media Enclosure DVD Sled with DVD-RAM USB Drive Half-high LTO Ultrium 5FC Tape Drive Half-high LTO Ultrium 6 FC Tape Drive
The console is a 19-inch, rack-mounted 1U unit that includes a language-specific IBM Travel Keyboard. The console kit is used with the Console Breakout cable that is shown in figure Figure 2-6. This cable provides serial, video, and two USB ports. The Console Breakout cable can be attached to the keyboard, video, and mouse (KVM) connector on the front panel of x86 based compute nodes, including the FSM.
24
IBM PureFlex System and IBM Flex System Products and Technology
The CMM in the chassis also allows direct connection to nodes via the internal chassis management network that communicates to the FSP or iMM2 on the node to allow remote out-of-band management.
25
26
IBM PureFlex System and IBM Flex System Products and Technology
Configurations
There are eight different orderable configurations within the enterprise PureFlex offerings. These offerings cover various redundant and non-redundant configurations along with the different types of protocol and storage controllers.
27
Table 2-11 summarizes the PureFlex Enterprise offerings that are fully configurable within the IBM configuration tools.
Table 2-11 PureFlex Enterprise Offerings Configuration
Networking Ethernet Networking Fibre Channel Number of Switches up to 18 maximum.a
5A
10 GbE FCoE
5B
10 GbE FCoE
6A
10GbE FCoE
6B
10 GbE FCoE
7A
10 GbE 16 Gb
7B
10 GbE 16 Gb
8A
10 GbE 16 Gb
8B
10GbE 16Gb
Storwize V7000
1, 2, or 3x Chassis with two Chassis management modules, fans, and PSUs 42U Rack mandatory Optional DVD only DVD and tape
Storage Options: (24 HDD, 22 HDD + 2 SSD, 20 HDD + 4 SSD or Custom) Storwize expansion (limit to single rack in Express, overflow storage rack in Enterprise): nine units per controller Up to two Storwize V7000 controllers, up to nine IBM Flex System V7000 Storage Nodes VIOS, AIX, IBM i, and SCE on first Controller
V7000 Content Nodes POWER Nodes Ethernet I/O Adapters POWER nodes Fibre Channel I/O Adapters x86 Nodes Ethernet I/O adapters x86 Nodes Fibre Channel I/O Adapters ESXi USB Key Port FoD Activations IBM i PureFlex Solution VDI PureFlex Solution
P260, p270, p460, x222, x240, x220, x440 CN4058 8-port 10Gb Converged Adapter EN4054 4-port 10Gb Ethernet Adapter
Not applicable
Not applicable
FC5022 2-port 16Gb FC Adapter FC3052 2-port 8Gb FC Adapter FC5024D 4-port 16Gb FC Adapter
Optional; for x86 compute nodes only Ports are computed during configuration based upon chassis switch, node type, and the I/O adapter selection. Not configurable
Supported
Not configurable
28
IBM PureFlex System and IBM Flex System Products and Technology
Example configuration
There are eight different configuration starting points for PureFlex Enterprise, as described in Table 2-11 on page 28. These configurations can be enhanced further with multi-chassis and other storage configurations. Figure 2-7 shows an example of the wiring for base configuration 6B, which is an Enterprise PureFlex that uses an external Storwize V7000 enclosure and CN4093 10Gb Converged Scalable Switch converged infrastructure switches. Also included are external SAN B24 switches and Top-of-Rack (TOR) G8264 Ethernet switches. The TOR switches allow for the data networks to allow other chassis to be configured into this solution (not shown).
G8264 TOR Access Points 40Gb ISL CN4093 CN4093 G8264 TOR
Node Bays 1 to 14
CMM
CMM
Midplane Connections Management 1GbE Data 10GbE Data 40GbE Data 8Gb FC
Label Label
29
There is a management network that is included in this configuration that is composed of a 1GbE G8062 network switch. The Access points within the PureFlex chassis provide connections from the clients network into the internal networking infrastructure of the PureFlex system and connections into to the Management network.
2.5.2 Chassis
Table 2-12 lists the major components of the IBM Flex System Enterprise Chassis, including the switches. Feature codes: The tables in this section do not list all feature codes. Some features are not listed here for brevity.
Table 2-12 Components of the chassis and switches AAS feature code 7893-92X 7955-01M A0TF ESW2 ESW7 EB28 EB29 3286 3771 5370 9039 3592 XCC feature code 8721-HC1 8731-AC1 3598 A3HH A3J6 5053 3268 5075 A2RQ 5084 A0TM A0UE Description IBM Flex System Enterprise Chassis IBM Flex System Manager IBM Flex System EN2092 1Gb Ethernet Scalable Switch IBM Flex System Fabric CN4093 10Gb Converged Scalable Switch IBM Flex System Fabric EN4093R 10Gb Scalable Switch IBM SFP+ SR Transceiver IBM SFP RJ45 Transceiver IBM 8Gb SFP+ Software Optical Transceiver IBM Flex System FC5022 24-port 16Gb ESB SAN Scalable Switch Brocade 8Gb SFP+ Software Optical Transceiver Base Chassis Management Module Other Chassis Management Module
30
IBM PureFlex System and IBM Flex System Products and Technology
Table 2-15 x86 based compute nodes AAS feature code ECS7 ECSB 0457 ECS8 MTM 7906-25X 7916-27X 7863-10X 7917-45X Description IBM Flex System x220 Compute Node IBM Flex System x222 Compute Node IBM Flex System x240 Compute Node IBM Flex System x440 Compute Node
31
IBM Open Fabric Manager Optional FSM advanced, adds VM Control Enterprise license
Expansion Unit (2076-224): Up to nine per V7000 Controller IBM Storwize V7000 Expansion Enclosure (24 disk slots)
32
IBM PureFlex System and IBM Flex System Products and Technology
Optional software: IBM Storwize V7000 Remote Mirroring IBM Storwize V7000 External Virtualization IBM Storwize V7000 Real-time Compression
Expansion Unit (4939-A29) IBM Storwize V7000 Expansion Enclosure (24 disk slots) Optional software: IBM Storwize V7000 Remote Mirroring IBM Storwize V7000 External Virtualization IBM Storwize V7000 Real-time Compression
33
The 7226 enclosure media devices offers support for SAS, USB, and Fibre Channel connectivity, depending on the drive. Support in a PureFlex configuration includes the external USB and Fibre Channel connections. Table 2-16 shows the Multi-Media Enclosure and available PureFlex options.
Table 2-16 Multi-Media Enclosure and options Machine/Type 7226 7226-1U3 7226-1U3 7226-1U3 Feature Code Model 1U3 5763 8248 8348 Description Multi-Media Enclosure DVD Sled with DVD-RAM USB Drive Half-high LTO Ultrium 5 FC Tape Drive Half-high LTO Ultrium 6 FC Tape Drive
The console is a 19-inch, rack-mounted 1U unit that includes a language-specific IBM Travel Keyboard. The console kit is used with the Console Breakout cable that is shown in Figure 2-10. This cable provides serial, video, and two USB ports. The Console Breakout cable can be attached to the KVM connector on the front panel of x86 based compute nodes, including the FSM.
34
IBM PureFlex System and IBM Flex System Products and Technology
The CMM in the chassis also allows direct connection to nodes via the internal chassis management network that communicates to the FSP or iMM2 on the node, which allows remote out-of-band management.
35
RHEL SLES
One node FSM Configuration Discovery, Inventory Review Internal Storage configuration Basic Network Integration using pre-configured switches (factory default) No external SAN integration No FCoE changes No Virtualization No Cloud Skills Transfer
36
IBM PureFlex System and IBM Flex System Products and Technology
Function delivered
PureFlex Extra Chassis Add-on 5 days Configure up to 14 nodes within one chassis Up to two virtualization engines (ESXi, KVM, or PowerVM) Configure up to 14 nodes within one chassis Up to two virtualization engines (ESXi, KVM, or PowerVM) Configure up to 14 nodes within one chassis Up to two virtualization engines (ESXi, KVM, or PowerVM)
Basic virtualization (VMware, KVM, and VMControl) No external SAN Integration No Cloud Up to four nodes
Advanced virtualization Server pools or VMware cluster configured (VMware or VMControl) No external SAN integration No FCoE Config Changes No Cloud Configure SmartCloud Entry Basic External network integration No FCoE Config changes No external SAN integration First chassis is configured with 13 nodes
Not included
Not included
Included
Included
Not included
Not included
Not included
Included
In addition to the offerings that are listed in Table 2-18 on page 36, two other services offerings are now available for PureFlex System and PureFlex IBM i Solution: PureFlex FCoE Customization Service and PureFlex Services for IBM i.
37
38
IBM PureFlex System and IBM Flex System Products and Technology
39
Optimize resources, which reduces the number of virtualized images and the storage required for them. Reduce time to value for new workloads with the following simple VM management options: Deploy application images across compute and storage resources. Offer users self-service for improved responsiveness. Enable security through VM isolation, project-level user access controls. Simplify deployment; there is no need to know all the details of the infrastructure. Protect your investment with support for existing virtualized environments. Optimize performance on IBM systems with dynamic scaling, expansive capacity and continuous operation. Improve efficiency with a private cloud that includes the following capabilities: Delegate provisioning to authorized users to improve productivity. Implement pay-per-use with built-in workload metering. Standardize deployment to improve compliance and reduce errors with policies and templates. Simplify management of projects, billing, approvals and metering with an intuitive user interface. Ease maintenance and problem diagnosis with integrated views of both physical and virtual resources. For more information about IBM SmartCloud Entry on Flex System, see this website: http://www.ibm.com/systems/flex/smartcloud/bto/entry/
40
IBM PureFlex System and IBM Flex System Products and Technology
Chapter 3.
Systems management
IBM Flex System Manager (the management component of IBM Flex System Enterprise Chassis) and compute nodes are designed to help you get the most out of your IBM Flex System installation. They also allow you to automate repetitive tasks. These management interfaces can significantly reduce the number of manual navigational steps for typical management tasks. They offer simplified system setup procedures by using wizards and built-in expertise to consolidate monitoring for physical and virtual resources. It must be noted that from August 2013, Power Systems Nodes installed within a Flex System chassis can alternatively be managed by Hardware Management Console (HMC) and Integrated Virtualization Manager (IVM). This allows clients with existing rack-based Power Systems servers to use a single management tool to manage their rack and Flex System nodes. However, systems management that is implemented in this way means none of the cross element management functions that are available with FSM (such as management of x86 Nodes, Storage, Networking, system pooling, or advanced virtualization function) are available. For the most complete and sophisticated broad management of a Flex System environment, the FSM is recommended. This chapter includes the following topics: 3.1, Management network on page 42 3.2, Chassis Management Module on page 43 3.3, Security on page 46 3.4, Compute node management on page 47 3.5, IBM Flex System Manager on page 50
41
Eth1: Embedded 2-port 10 GbE controller with Virtual Fabric Connector Eth0: Special GbE management network adapter
Enterprise Chassis Flex System Manager System x compute node Power Systems compute node
Eth0 IMM
CMM
Port
I/O bay 1
I/O bay 2
CMM
CMM
Management workstation
42
IBM PureFlex System and IBM Flex System Products and Technology
Tip: The management node console can be connected to the data network for convenient access. One of the key functions that the data network supports is the discovery of operating systems on the various network endpoints. Discovery of operating systems by the FSM is required to support software updates on an endpoint, such as a compute node. The FSM Checking and Updating Compute Nodes wizard assists you in discovering operating systems as part of the initial setup.
3.2.1 Overview
The CMM is a hot-swap module that provides basic system management functions for all devices that are installed in the Enterprise Chassis. An Enterprise Chassis comes with at least one CMM and supports CMM redundancy. The CMM is shown in Figure 3-2.
43
Through an embedded firmware stack, the CMM implements functions to monitor, control, and provide external user interfaces to manage all chassis resources. You can use the CMM to perform the following functions, among others: Define login IDs and passwords. Configure security settings such as data encryption and user account security. The CMM contains an LDAP client that can be configured to provide user authentication through one or more LDAP servers. The LDAP server (or servers) to be used for authentication can be discovered dynamically or manually pre-configured. Select recipients for alert notification of specific events. Monitor the status of the compute nodes and other components. Find chassis component information. Discover other chassis in the network and enable access to them. Control the chassis, compute nodes, and other components. Access the I/O modules to configure them. Change the startup sequence in a compute node. Set the date and time. Use a remote console for the compute nodes. Enable multi-chassis monitoring. Set power policies and view power consumption history for chassis components.
3.2.2 Interfaces
The CMM supports a web-based graphical user interface that provides a way to perform chassis management functions within a supported web browser. You can also perform management functions through the CMM command-line interface (CLI). Both the web-based and CLI interfaces are accessible through the single RJ45 Ethernet connector on the CMM, or from any system that is connected to the same network. The CMM has the following default IPv4 settings: IP address: 192.168.70.100 Subnet: 255.255.255.0 User ID: USERID (all capital letters) Password: PASSW0RD (all capital letters, with a zero instead of the letter O) The CMM does not have a fixed static IPv6 IP address by default. Initial access to the CMM in an IPv6 environment can be done by using the IPv4 IP address or the IPv6 link-local address. The IPv6 link-local address is automatically generated based on the MAC address of the CMM. By default, the CMM is configured to respond to DHCP first before it uses its static IPv4 address. If you do not want this operation to take place, connect locally to the CMM and change the default IP settings. For example, you can connect locally by using a notebook. The web-based GUI brings together all the functionality that is needed to manage the chassis elements in an easy-to-use fashion consistently across all System x IMM2 based platforms.
44
IBM PureFlex System and IBM Flex System Products and Technology
Figure 3-4 shows an example of the Chassis Management Module front page after login.
45
3.3 Security
The focus of IBM on smarter computing is evident in the improved security measures that are implemented in IBM Flex System Enterprise Chassis. Todays world of computing demands tighter security standards and native integration with computing platforms. For example, the push towards virtualization increased the need for more security. This increase comes as more mission-critical workloads are consolidated on to fewer and more powerful servers. The IBM Flex System Enterprise Chassis takes a new approach to security with a ground-up chassis management design to meet new security standards. The following security enhancements and features are provided in the chassis: Single sign-on (central user management) End-to-end audit logs Secure boot: IBM Tivoli Provisioning Manager and CRTM Intel TXT technology (Intel Xeon -based compute nodes) Signed firmware updates to ensure authenticity Secure communications Certificate authority and management Chassis and compute node detection and provisioning Role-based access control Security policy management Same management protocols that are supported on BladeCenter AMM for compatibility with earlier versions Insecure protocols are disabled by default in CMM, with Locks settings to prevent user from inadvertently or maliciously enabling them Supports up to 84 local CMM user accounts Supports up to 32 simultaneous sessions Planned support for DRTM CMM supports LDAP authentication The Enterprise Chassis ships Secure, and supports the following security policy settings: Secure: Default setting to ensure a secure chassis infrastructure and includes the following features: Strong password policies with automatic validation and verification checks Updated passwords that replace the manufacturing default passwords after the initial setup Only secure communication protocols such as Secure Shell (SSH) and Secure Sockets Layer (SSL) Certificates to establish secure, trusted connections for applications that run on the management processors Legacy: Flexibility in chassis security, which includes the following features: Weak password policies with minimal controls Manufacturing default passwords that do not have to be changed
46
IBM PureFlex System and IBM Flex System Products and Technology
Unencrypted communication protocols, such as Telnet, SNMPv1, TCP Command Mode, FTP Server, and TFTP Server The centralized security policy makes Enterprise Chassis easy to configure. In essence, all components run with the same security policy that is provided by the CMM. This consistency ensures that all I/O modules run with a hardened attack surface. The CMM and the IBM Flex System Manager management node each have their own independent security policies that control, audit, and enforce the security settings. The security settings include the network settings and protocols, password and firmware update controls, and trusted computing properties such as secure boot. The security policy is distributed to the chassis devices during the provisioning process.
The management controllers for the various Enterprise Chassis components have the following default IPv4 addresses: CMM:192.168.70.100 Compute nodes: 192.168.70.101-114 (corresponding to the slots 1-14 in the chassis) I/O Modules: 192.168.70.120-123 (sequentially corresponding to chassis bay numbering) In addition to the IPv4 address, all I/O modules support link-local IPv6 addresses and configurable external IPv6 addresses.
47
Remote presence: Increased color depth and resolution for more detailed server video Active X client in addition to Java client Increased memory capacity (~50 MB) provides convenience for remote software installations No IMM2 reset is required on configuration changes because they become effective immediately without reboot Hardware management of non-volatile storage Faster Ethernet over USB 1 Gb Ethernet management capability Improved system power-on and boot time More detailed information for UEFI detected events enables easier problem determination and fault isolation User interface meets accessibility standards (CI-162 compliant) Separate audit and event logs Trusted IMM with significant security enhancements (CRTM/TPM, signed updates, authentication policies, and so on) Simplified update and flashing mechanism Addition of Syslog alerting mechanism provides you with an alternative to email and SNMP traps Support for Features on-demand (FoD) enablement of server functions, option card features, and System x solutions and applications First Failure Data Capture: One button web press starts data collection and download For more information about IMM2, see Chapter 5, Compute nodes on page 185. For more information, see the following publications: Integrated Management Module II Users Guide: http://ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5086346 IMM and IMM2 Support on IBM System x and BladeCenter Servers, TIPS0849: http://www.redbooks.ibm.com/abstracts/tips0849.html
48
IBM PureFlex System and IBM Flex System Products and Technology
The FSP provides an SOL interface, which is available by using the CMM and the console command. The Power Systems compute nodes do not have an on-board video chip, and do not support keyboard, video, and mouse (KVM) connections. Server console access is obtained by a SOL connection only. SOL provides a means to manage servers remotely by using a CLI over a Telnet or SSH connection. SOL is required to manage servers that do not have KVM support or that are attached to the FSM. SOL provides console redirection for Software Management Services (SMS) and the server operating system. The SOL feature redirects server serial-connection data over a LAN without requiring special cabling by routing the data through the CMM network interface. The SOL connection enables Power Systems compute nodes to be managed from any remote location with network access to the CMM. SOL offers the following functions: Remote administration without KVM Reduced cabling and no requirement for a serial concentrator Standard Telnet/SSH interface, which eliminates the requirement for special client software The CMM CLI provides access to the text-console command prompt on each server through an SOL connection. This configuration allows the Power Systems compute nodes to be managed from a remote location.
49
Ability to detect firmware/hardware hangs, and ability to pull a crash-failure memory dump file to an FTP (sFTP) server. Selectable primary and backup firmware banks as the current operational firmware. Ability to send events, SNMP traps, and event logs to the CMM, including security audit logs. IPv4 and IPv6 on by default. The CMM management port supports IPv4 and IPv6 (IPV6 support includes the use of link local addresses. Port mirroring capabilities: Port mirroring of CMM ports to internal and external ports. For security reasons, the ability to mirror the CMM traffic is hidden and is available only to development and service personnel Management virtual local area network (VLAN) for Ethernet switches: A configurable management 802.1q tagged VLAN in the standard VLAN range of 1 - 4094. It includes the CMMs internal management ports and the I/O modules internal ports that are connected to the nodes.
IBM Flex System Manager base feature set that is preinstalled offers the following function: Support up to 16 managed chassis Support for up to 224 Nodes Support up to 5,000 managed elements Auto-discovery of managed elements Overall health status Monitoring and availability Hardware management Security management Administration Network management (Network Control) Storage management (Storage Control) Virtual machine lifecycle management (VMControl Express) The IBM Flex System Manager Advanced feature set upgrade offers the following advanced features: Image management (VMControl Standard) Pool management (VMControl Enterprise) Advanced network monitoring and quality of service (QoS) configuration (Service Fabric Provisioning) The Fabric Provisioning upgrade offers advanced network monitoring and quality of service (QoS) configuration (Service Fabric Provisioning). Fabric provisioning functionality is included in the advanced feature set. It is also available as a separate Fabric Provisioning feature upgrade for the base feature set. It also can be ordered as a separate fabric provisioning upgrade for the Flex System Manager node via the HVEC order route. Upgrade licenses: The Advanced Upgrade and the Fabric Provisioning feature upgrade are mutually exclusive. Either the Advance Upgrade or the Fabric Provisioning feature can be applied on top of the base feature set license, but not both. The Service Fabric provisioning upgrade is not selectable in AAS. The part number to order the management node is shown in Table 3-2.
Table 3-2 Ordering information for IBM Flex System Manager node HVEC 8731-A1xa AAS 7955-01Mb Description IBM Flex System Manager node
a. x in the Part number represents a country-specific letter (for example, the EMEA part number is 8731A1G, and the US part number is 8731A1U). Ask your local IBM representative for specifics. b. This part number is ordered as part of the IBM PureFlex System
The part numbers to order FoD software entitlement licenses are shown in the following tables. The part numbers for the same features are different in different countries. Ask your local IBM representative for specifics. Table 3-3 on page 52 shows the following sets of part numbers: Column 1: for Latin America and Europe/Middle East/Africa Column 2: For US, Canada, Asia Pacific, and Japan
51
Table 3-3 HVEC Ordering information for FoD licenses Part number LA & EMEA US, CAN, AP, JPN Description
Base feature set 95Y1174 95Y1179 90Y4217 90Y4222 IBM Flex System Manager Per Managed Chassis with 1-Year Software Support and Subscription (software S&S) IBM Flex System Manager Per Managed Chassis with 3-Year software S&S
Advanced feature set upgradea 94Y9219 94Y9220 90Y4249 00D7554 IBM Flex System Manager, Advanced Upgrade, Per Managed Chassis with 1-Year software S&S IBM Flex System Manager, Advanced Upgrade, Per Managed Chassis with 3-Year software S&S
Fabric Provisioning feature upgradea 95Y1178 95Y1183 90Y4221 90Y4226 IBM Flex System Manager Service Fabric Provisioning with 1-Year S&S IBM Flex System Manager Service Fabric Provisioning with 3-Year S&S
a. The Advanced Upgrade and Fabric Provisioning licenses are applied on top of the IBM FSM base license
Table 3-4 shows the indicator codes that are selected when configuring Flex System Manager in AAS by using e-config. This also selects the relevant options for one or three years of S&S that is included in the configurator output.
Table 3-4 7955-01M Flex System Manager feature codes Description
Feature code
Advanced feature set upgradea EB31 EB32 FSM Platform Software Bundle Pre-load Indicator FSM Platform Virtualization Software Bundle Pre-load Indicator
a. The FSM Platform Virtualization Software Bundle Pre-load Indicator are applied on top of the FSM Platform Software Bundle Pre-load Indicator
Example 1
A client wants to manage four Flex System chassis with one FSM, no advanced license function, with three years of support and subscription (S&S). The client purchases the following products: One Flex System Manager node Four IBM Flex System Manager per managed chassis with 3-Year SW S&S
52
IBM PureFlex System and IBM Flex System Products and Technology
Table 3-5 shows the part numbers and quantity that is required. The following sets of part numbers are shown: Column 1: For Latin America and Europe/Middle East/Africa Column 2: For US, Canada, Asia Pacific, and Japan
Table 3-5 Example 1 part numbers Part number Qty 1 4 LA & EMEA 8731-A1xa 95Y1179 US, CAN, AP, JPN 8731-A1xa 90Y4222 Description IBM Flex System Manager node IBM Flex System Manager Per Managed Chassis with three-year software S&S
a. x in the Part number represents a country-specific letter (for example, the EMEA part number is 8731A1G, and the US part number is 8731A1U). Ask your local IBM representative for specifics.
Example 2
The client wants to manage four Flex System chassis in total, two chassis are located on one site and two on another, with a local FSM installed in a chassis at each of these sites. They require advance functionality with three-year S&S. A client purchases the following products: Two Flex System Manager Nodes Four IBM Flex System Manager per managed chassis with three-year software S&S Four IBM Flex System Manager, Advanced Upgrade, Per Managed Chassis with three-year software S&S. Table 3-6 shows the part numbers and quantity that are required. The following sets of part numbers are shown: Column 1: For Latin America and Europe/Middle East/Africa Column 2: For US, Canada, Asia Pacific, and Japan
Table 3-6 Example 2 part numbers Part number Qty 2 4 4 LA & EMEA 8731-A1xa 95Y1179 94Y9220 US, CAN, AP, JPN 8731-A1xa 90Y4222 00D7554 Description IBM Flex System Manager node IBM Flex System Manager Per Managed Chassis with three-year software S&S IBM Flex System Manager, Advanced Upgrade, Per Managed Chassis with three-year software S&S
a. x in the Part number represents a country-specific letter (for example, the EMEA part number is 8731A1G, and the US part number is 8731A1U). Ask your local IBM representative for specifics.
53
54
IBM PureFlex System and IBM Flex System Products and Technology
Figure 3-6 shows the internal layout and major components of the FSM.
Cover
Heat sink Microprocessor Microprocessor heat sink filler SSD and HDD backplane Hot-swap storage cage SSD interposer SSD drives I/O expansion adapter ETE adapter
DIMM
Figure 3-6 Exploded view of the IBM Flex System Manager node, showing major components
55
The FSM comes preconfigured with the components that are described in Table 3-7.
Table 3-7 Features of the IBM Flex System Manager node Feature Model numbers Processor Memory SAS Controller Disk Integrated NIC Systems Management Description 8731-A1x (XCC, x-config) 7955-01M (AAS, e-config) 1x Intel Xeon processor E5-2650 8C 2.0 GHz 20 MB Cache 1600 MHz 95 W 8 x 4 GB (1x4 GB, 1Rx4, 1.35 V) PC3L-10600 CL9 ECC DDR3 1333 MHz LP RDIMM One LSI 2004 SAS Controller 1 x IBM 1TB 7.2K 6 Gbps NL SATA 2.5" SFF HS HDD 2 x IBM 200GB SATA 1.8" MLC SSD (configured in an RAID-1) Embedded dual-port 10 Gb Virtual Fabric Ethernet controller (Emulex BE3) Dual-port 1 GbE Ethernet controller on a management adapter (Broadcom 5718) Integrated Management Module II (IMM2) Management network adapter
Figure 3-7 shows the internal layout of the FSM. Filler slot for Processor 2 Processor 1
Drive bays
Figure 3-7 Internal view that shows the major components of IBM Flex System Manager
56
IBM PureFlex System and IBM Flex System Products and Technology
Front controls
The FSM has similar controls and LEDs as the IBM Flex System x240 Compute Node. Figure 3-8 shows the front of an FSM with the location of the control and LEDs highlighted.
Solid state drive LEDs
a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aaaa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaa 2 aaaaaaaaaaaaaaaaaaaaaaaa a aa aa aa aa aa aaaa aa aa aa aaaa a aaaaaaaaaaaaaaaaaaaa aa a aa aaaa aaaa aaaaaaaaaaaaaaaaaa aaaa aa aa aa aa aa aa aa a aa aa aa aa aa aa aa aa aa aa aa a 1 a aaaaaa aaaa aaaa aa aa aa aa aa aa aa aa aa aa aa aaaaaaaa aaaa aaaa aa aa aa aa aa aa aa aa aa a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aaaaaaaaaaaaaa aa aa aa aa aa aa aa aa aa aa aa a a a aa a aa a aa a aa aa aa aa aa aa aa aa aaaaaaaaaaaaa a aaaaaaaaaaaaaaaaaaaaaa aa a a a a a a a a a a a aaaaaaaaaaaa 0 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Power button/LED
Identify LED
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa a a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aaaa a aaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa a aaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa aaaa aaaa aa aa a aa aaaa aaaa aaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aaaa aa aa aa a a a a a a a a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aaaa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa aa aa aa aa aa aaaaaaaaaaaaaaaaaaaaaa aaa aa aa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaa aaaaaaaaaaaaaaaaaaaaaa aaa a aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaaa a aaaa aaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa aaaa aa aa aaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaa a aa aa aaaaaa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Fault Hard disk drive LED status LED Check log LED
Storage
The FSM ships with 2 x IBM 200 GB SATA 1.8" MLC SSD and 1 x IBM 1 TB 7.2K 6 Gbps NL SATA 2.5" SFF HS HDD drives. The 200 GB SSD drives are configured in an RAID-1 pair that provides roughly 200 GB of usable space. The 1 TB SATA drive is not part of a RAID group.
57
58
IBM PureFlex System and IBM Flex System Products and Technology
Component names and hardware identification numbers Firmware levels Usage rates
Network management: Management of network switches from various vendors Discovery, inventory, and status monitoring of switches Graphical network topology views Support for KVM, pHyp, VMware virtual switches, and physical switches VLAN configuration of switches Integration with server management Per-virtual machine network usage and performance statistics that are provided to VMControl Logical views of servers and network devices that are grouped by subnet and VLAN Network management (advanced feature set or fabric provisioning feature): Defines QoS settings for logical networks Configures QoS parameters on network devices Provides advanced network monitors for network system pools, logical networks, and virtual systems Storage management: Discovery of physical and virtual storage devices Physical and logical topology views Support for virtual images on local storage across multiple chassis Inventory of physical storage configuration Health status and alerts Storage pool configuration Disk sparing and redundancy management Virtual volume management Support for virtual volume discovery, inventory, creation, modification, and deletion
Virtualization management (base feature set) Support for VMware, Hyper-V, KVM, and IBM PowerVM Create virtual servers Edit virtual servers Manage virtual servers Relocate virtual servers Discover virtual server, storage, and network resources, and visualize the physical-to-virtual relationships Virtualization management (advanced feature set) Create new image repositories for storing virtual appliances and discover existing image repositories in your environment Import external, standards-based virtual appliance packages into your image repositories as virtual appliances Capture a running virtual server that is configured the way you want, complete with guest operating system, running applications, and virtual server definition
59
Import virtual appliance packages that exist in the Open Virtual Machine Format (OVF) from the Internet or other external sources Deploy virtual appliances quickly to create virtual servers that meet the demands of your ever-changing business needs Create, capture, and manage workloads Create server system pools, which enable you to consolidate your resources and workloads into distinct and manageable groups Deploy virtual appliances into server system pools Manage server system pools, including adding hosts or more storage space, and monitoring the health of the resources and the status of the workloads in them Group storage systems together by using storage system pools to increase resource usage and automation Manage storage system pools by adding storage, editing the storage system pool policy, and monitoring the health of the storage resources I/O address management: Manages assignments of Ethernet MAC and Fibre Channel WWN addresses. Monitors the health of compute nodes, and automatically, without user intervention, replaces a failed compute node from a designated pool of spare compute nodes by reassigning MAC and WWN addresses. Preassigns MAC addresses, WWN addresses, and storage boot targets for the compute nodes. Creates addresses for compute nodes, saves the address profiles, and deploys the addresses to the slots in the same or different chassis. Other features: Resource-oriented chassis map provides instant graphical view of chassis resource that includes nodes and I/O modules: Fly-over provides instant view of individual server (node) status and inventory Chassis map provides inventory view of chassis components, a view of active statuses that require administrative attention, and a compliance view of server (node) firmware Actions can be taken on nodes such as working with server-related resources, showing and installing updates, submitting service requests, and starting the remote access tools Resources can be monitored remotely from mobile devices, including Apple iOS based devices, Google Android -based devices, and RIM BlackBerry based devices. Flex System Manager Mobile applications are separately available under their own terms and conditions as outlined by the respective mobile markets. Ability to open video sessions and mount media such as DVDs with software updates to their servers from their local workstation Remote KVM connections Remote Virtual Media connections (mount CD/DVD/ISO/USB media) Power operations against servers (Power On/Off/Restart)
Remote console:
Health status (such as processor usage) on all hardware devices from a single chassis view Automatic detection of hardware failures: Provides alerts Takes corrective action Notifies IBM of problems to escalate problem determination
Administrative capabilities, such as setting up users within profile groups, assigning security levels, and security governance Bare metal deployment of hypervisors (VMware ESXi, KVM) through centralized images
Also, warnings are generated if metrics exceed a set of predefined thresholds. The warning includes full details of the specific warning and recommendations to help rectify the situation. Thresholds are presented as green, yellow or red. It is also possible to configure the thresholds. Further, a capacity usage report can be generated that shows overall usage, the current status of key parameters, and a list of historical data. Deploy compute node image enhancements: Increased supported OS Images in repository from 2 to 5
Chapter 3. Systems management
61
Improved MAC address support: pNIC VNIC, Virtual addresses Improved OS support: ESXi Image 5.1.1, REL 6.4, REL KVM platform agent for VMControl and ESXi 5000V Agent V1.1.0 Bare metal deployment patterns included for the new x222 nodes. Configuration Patterns enhancements: Pattern stored in LDAP New path options for initial setup New I/O adapter options Unique x222 configuration pattern support and independent node failover support New Keep settings option for boot order configuration Improved guidance for deployment of a pattern Increased supported OS Images in repository from 2 - 5 Improved MAC address support Improved dialog Ability to edit and deploy patterns during initial FSM setup Multiple improvements in usability of patterns
Enhancements to console: View scheduled Jobs in mouse flyover Compliance issues show in console scoreboard and on chassis map compliance overlay Flex firmware views and Compliance issue views; compliance automatically marked when new updates are imported More backup and restore context-sensitive help IEEE 802.1Qbg support added for Power Nodes Performance enhancement for Inventory export (much faster export) Compare installed fixes for IBM i between one installed system and another Smart Zoning enhancements: Simplified interactions between storage and server, no need to pre-zone Create storage volume enhancements; can automatically zone host and storage when zoning was not previously configured. Only needed zoning operations are performed to ensure host and storage can communicate with each other: If zoning is not enabled, it is enabled If a zone set is not created, it is created If a zone does not exist for host and storage, one is created
Management extended to support System x3950 SAP HANA appliance: Manual discovery and inventory Power Control Remote Access System Configuration System Health and Status Release Management (firmware, software installation and update) Service and Support
62
IBM PureFlex System and IBM Flex System Products and Technology
Yes Yes
Yes Yes
Yes No
Yes No
Table 3-10 on page 64 summarizes the management tasks that are supported by the compute nodes that depend on the agent tier.
63
Table 3-10 Compute node management tasks that are supported by the agent tier Agent tier Managed system type Command automation Hardware alerts Platform alerts Health and status monitoring File transfer Inventory (hardware) Inventory (software) Problems (hardware status) Process management Power management Remote control Remote command line Resource monitors Update manager Agentless in-band No No No No No No Yes No No No No Yes No No Agentless out-of-band No Yes No No No Yes No Yes No Yes Yes No No No Platform Agent No Yes Yes Yes No Yes Yes Yes No No No Yes Yes Yes Common Agent Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No Yes Yes Yes
Table 3-11 shows the supported virtualization environments and their management tasks.
Table 3-11 Supported virtualization environments and management tasks Virtualization environment Management task Deploy virtual servers Deploy virtual farms Relocate virtual servers Import virtual appliance packages Capture virtual servers Capture workloads Deploy virtual appliances Deploy workloads Deploy server system pools Deploy storage system pools AIX and Linuxa Yes No Yes Yes Yes Yes Yes Yes Yes Yes IBM i Yes No No Yes Yes Yes Yes Yes No No VMware vSphere Yes Yes Yes No No No No No No No Microsoft Hyper-V Yes No No No No No No No No No Linux KVM Yes Yes Yes Yes Yes Yes Yes Yes Yes No
64
IBM PureFlex System and IBM Flex System Products and Technology
Table 3-12 shows the supported I/O switches and their management tasks.
Table 3-12 Supported I/O switches and management tasks EN2092 1 Gb Ethernet Yes Yes Yes Yes Yes Yes No EN4093 and EN4093R 10 Gb Ethernet Yes Yes Yes Yes Yes Yes Yes CN4093 10 Gb Converged Yes Yes Yes Yes Yes Yes No FC3171 8 Gb FC Yes Yes Yes Yes Yes Yes No FC5022 16 Gb FC Yes Yes Yes Yes No No No
Management task Discovery Inventory Monitoring Alerts Configuration management Automated logical network provisioning (ALNP) Stacked switch
Table 3-13 shows the supported virtual switches and their management tasks.
Table 3-13 Supported virtual switches and management tasks Virtualization environment Virtual switch Management task Discovery Inventory Configuration management Automated logical network provisioning (ALNP) Linux KVM Platform Agent Yes Yes Yes Yes VMware vSphere VMware Yes Yes Yes Yes IBM 5000V Yes Yes Yes Yes PowerVM PowerVM Yes Yes Yes Yes Hyper-V Hyper-V No No No No
Table 3-14 shows the supported storage systems and their management tasks.
Table 3-14 Supported storage systems and management tasks Storage system Management task Storage device discovery Inventory collection Monitoring (alerts and status) Integrated physical and logical topology views Show relationships between storage and server resources Perform logical and physical configuration View and manage attached devices VMControl provisioning V7000 Storage Node Yes Yes Yes Yes Yes Yes Yes Yes IBM Storwize V7000 Yes Yes Yes No Yes Yes No Yes
65
Web interface
The following browsers are supported by the management software web interface: Mozilla Firefox versions 3.5.x, 3.6.x, 7.0, and Extended Support Release (ESR) 10.0.x Microsoft Internet Explorer versions 7.0, 8.0, and 9.0
66
IBM PureFlex System and IBM Flex System Products and Technology
Chassis Map (hardware view): Check the front and rear graphical hardware views of a chassis. Chassis List (components view): View a list of the hardware components that are installed in a chassis. Inventory Management: See the Vital Product Data (VPD) for a managed resource (for example, serial number or IP address). Multiple chassis management: Manage multiple chassis and multiple management nodes from a single application. Authentication and security: Secure all connections by using encrypted protocols (for example, SSL), and secure persistent credentials on your mobile device. You can download the Mobile System Management application for your mobile device from one of the following app stores: Google Play for the Android operating system iTunes for the Apple iOS BlackBerry App World
Perform actions on CMM, such as Virtual Reseat and Restart Primary CMM Recent Jobs View a list of the recent jobs (last 24 hours) that were run from mobile or desktop. Event Log Easier to toggle between event log and status. Chassis Map (hardware view): Check the front and rear graphical hardware views for a chassis Overlay the graphical views with Power and Error LEDs Inventory Management See the Vital Product Data (VPD) and Firmware Levels for managed resources Authentication and Security: Simpler connection menu Accept unsigned certificates For more information about the application, see the Mobile System Management application page at this website: http://www.ibm.com/systems/flex/fsm/mobile/
67
68
IBM PureFlex System and IBM Flex System Products and Technology
Chapter 4.
69
4.1 Overview
Figure 4-1 shows the Enterprise Chassis as seen from the front. The front of the chassis includes 14 horizontal bays with robust removable dividers that allow nodes and future elements to be installed within the chassis. Nodes can be Compute, Storage, or Expansion type. The nodes can be installed when the chassis is powered. The chassis uses a die-cast mechanical bezel for rigidity to allow shipment of the chassis with nodes installed. This chassis construction allows for tight tolerances between nodes, shelves, and the chassis bezel. These tolerances ensure accurate location and mating of connectors to the midplane.
The Enterprise Chassis includes the following major components: Fourteen standard (half-wide) node bays. It can also support seven, two-bay or three, four-bay nodes with the shelves removed. 8271-A1x: Up to Six 2500W power modules that provide N+N or N+1 redundant power. 8271-LRx: Up to Six 2100W Power Modules that provide N+N or N+1 redundant power. Ten fan modules (eight 80 mm fan modules and two 40 mm fan modules). Four physical I/O modules. An I/O architectural design capable of providing the following features: Up to eight lanes of I/O to an I/O adapter. Each lane capable of up to 16 Gbps. A maximum of 16 lanes of I/O to a half-wide node with two adapters. Various networking solutions that include Ethernet, Fibre Channel, FCoE, and InfiniBand. Two IBM Chassis Management Module (CMMs). The CMM provides single-chassis management support.
70
IBM PureFlex System and IBM Flex System Products and Technology
Table 4-1 lists the quantity of components that comprise the 8271 machine type:
Table 4-1 8721 Enterprise Chassis configuration 8721-A1x 1 1 2 0 4 2 1 2 1 8721-LRx 1 1 0 2 4 2 1 2 1 Description IBM Flex System Enterprise Chassis Chassis Management Module 2500W power supply unit 2100W Power supply unita 80 mm fan modules 40 mm fan modules Console breakout cable C19 to C20 2M power cables Rack mount kit
a. 2100W power supply units are also available through the CTO process
More Console Breakout Cables can be ordered, if required. The console breakout cable connects to the front of an x86 node and allows Keyboard, Video, USB, and Serial to be attached locally to that node. For more information about alternative methods, see 4.12.5, Console planning on page 169. The Chassis Management Module (CMM) includes built-in console re-direction via the CMM Ethernet port.
Table 4-2 Ordering part number and feature code Part number 81Y5286 Feature code A1NF Description IBM Flex System Console Breakout Cable
Figure 4-2 on page 72 shows the component parts of the chassis with the shuttle removed. The shuttle forms the rear of the chassis where the I/O Modules, power supplies, fan modules, and CMMs are installed. The Shuttle is removed only to gain access to the midplane or fan distribution cards in the rare event of a service action.
71
Chassis
I/O module
Shuttle
Within the chassis, a personality card holds vital product data (VPD) and other information that is relevant to the particular chassis. This card can be replaced only under service action, and is not normally accessible. The personality card is attached to the midplane, as shown in Figure 4-4 on page 74.
72
IBM PureFlex System and IBM Flex System Products and Technology
Bay 13
Bay 14
Bay 11
Bay 12
Bay 9
Bay 10
Bay 7
Bay 8
Bay 5
Bay 6
Bay 3
Bay 4
Bay 1
a aaaaaaaaaaaaaaaa a a a a a a a a a a a a a a a a a a a a a aaaa aaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa aa aaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aa aa a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aa aa aa aaaaaa aaaaaa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aa aa aa aaaaaa aaaaaa a a a a a a a a a a a a a a a aaaaaa aa aa aa aaaaaa aaaa aa aa aa aa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a a a a a a aaaaaa aaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaa aa aa aa aa aa aa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Bay 2
The chassis includes the following features on the front: The front information panel on the lower left of the chassis Bays 1 - 14 that support nodes and FSM Lower airflow inlet apertures that provide air cooling for switches, CMMs, and power supplies Upper airflow inlet apertures that provide cooling for power supplies For efficient cooling, each bay in the front or rear in the chassis must contain a device or filler. The Enterprise Chassis provides several LEDs on the front information panel that can be used to obtain the status of the chassis. The Identify, Check log, and the Fault LED are also on the rear of the chassis for ease of use.
73
4.1.2 Midplane
The midplane is the circuit board that connects to the compute nodes from the front of the chassis. It also connects to I/O modules, fan modules, and power supplies from the rear of the chassis. The midplane is located within the chassis and can be accessed by removing the Shuttle assembly. Removing the midplane is rare and only necessary in case of service action. The midplane is passive, which is to say that there are no electronic components on it. The midplane has apertures to allow air to pass through. When no node is installed in a standard node bay, the Air Damper is completely closed for that bay, which gives highly efficient scale up cooling. The midplane has reliable industry standard connectors on both sides for power supplies, fan distribution cards, switches, I/O modules and nodes. The chassis design allows for highly accurate placement and mating of connectors from the nodes, I/O modules, and Power supplies to the midplane, as shown in Figure 4-4.
Midplane front view Node power connectors Management connectors I/O module connectors
The midplane uses a single power domain within the design. This a cost-effective overall solution and optimizes the design for a preferred 10U Height. Within the midplane, there are five separate power and ground planes for distribution of the main 12.2 Volt power domain through the chassis.
74
IBM PureFlex System and IBM Flex System Products and Technology
The midplane also distributes I2C management signals and some 3.3v for powering management circuits. The power supplies source their fan power from the midplane. Figure 4-4 on page 74 shows the connectors on both sides of the midplane.
The following components can be installed into the rear of the chassis: Up to two CMMs. Up to six 2500W or 2100W power supply modules. Up to six fan modules that consist of four 80 mm fan modules and two 40 mm fan modules. More fan modules can be installed for a total of 10 modules. Up to four I/O modules.
4.1.4 Specifications
Table 4-3 shows the specifications of the Enterprise Chassis 8721-A1x.
Table 4-3 Enterprise Chassis specifications Feature Machine type-model Form factor Maximum number of compute nodes supported Specifications System x ordering sales channel: 8721-A1x or 8721-LRx Power Systems sales channel: 7893-92Xa 10U rack mounted unit 14 half-wide (single bay), 7 full-wide (two bays), or 3 double-height full-wide (four bays). Mixing is supported.
75
Feature Chassis per 42U rack Nodes per 42U rack Management
Specifications 4 56 half-wide, or 28 full-wide One or two Chassis Management Modules for basic chassis management. Two CMMs form a redundant pair. One CMM is standard in 8721-A1x and 8271-LRx. The CMM interfaces with the Integrated Management Module II (IMM2) or flexible service processor (FSP) integrated in each compute node in the chassis and also to the integrated storage node. An optional IBM Flex System Managera management appliance provides comprehensive management that includes virtualization, networking, and storage management. Up to eight lanes of I/O to an I/O adapter, with each lane capable of up to 16 Gbps bandwidth. Up to 16 lanes of I/O to a half wide-node with two adapters. Various networking solutions include Ethernet, Fibre Channel, FCoE, and InfiniBand 8721-A1x: Six 2500W power modules that can provide N+N or N+1 redundant power. Two are standard in this model. 8721-LRx: Six 2100W power modules that can provide N+N or N+1 redundant power. Two are standard in this model. Power supplies are 80 PLUS Platinum certified and provide over 94% efficiency at 50% load and 20% load. Power capacity of 2500 watts output rated at 200 VAC. Each power supply contains two independently powered 40 mm cooling fan modules. Ten fan modules (eight 80 mm fan modules and two 40 mm fan modules). Four 80 mm and two 40 mm fan modules are standard in model 8721-A1x and 8721-LRx. Height: 440 mm (17.3) Width: 447 mm (17.6) Depth, measured from front bezel to rear of chassis: 800 mm (31.5 inches) Depth, measured from node latch handle to the power supply handle: 840 mm (33.1 inches) Minimum configuration: 96.62 kg (213 lb) Maximum configuration: 220.45 kg (486 lb) 6.3 to 6.8 bels Operating air temperature 5C to 40C Input power: 200 - 240 VAC (nominal), 50 or 60 Hz Minimum configuration: 0.51 kVA (two power supplies) Maximum configuration: 13 kVA (six 2500W power supplies) 12,900 watts maximum
I/O architecture
Power supplies
Power consumption
a. When you order the IBM Flex System Enterprise Chassis through the Power Systems sales channel, the IBM Flex System Manager is required if PowerVM software is selected on a power node.
For data center planning, the chassis is rated to a maximum operating temperature of 40C. For comparison, BC-H is rated to 35C (110v operation is not supported). The AC operating range is 200 - 240 VAC.
76
IBM PureFlex System and IBM Flex System Products and Technology
The filter is attached to and removed from the chassis, as shown in Figure 4-6.
77
Shelf
Tabs
a. Node must be powered off, in standby before removal. b. I/O Module might require reconfiguration, and removal is disruptive to any communications that are taking place.
Nodes can be plugged into the chassis while the chassis is powered. The node can then be powered on. Power the node off before removal.
78
IBM PureFlex System and IBM Flex System Products and Technology
a. The first feature code listed is for configurations that are ordered through System x sales channels (HVEC) that use x-config. The second feature code is for configurations that are ordered through the IBM Power Systems channel (AAS) that use e-config.
79
Table 4-7 shows the Feature Codes that are used when you are ordering through the Power Systems channel route (AAS) via e-config.
Table 4-7 Power Supply feature codes AAS (Power Brand) Description 2100 Wa 2500 W Feature Code for base power supplies (quantity must be 2) 9036 9059 Feature code for additional power supplies (quantity must be 0, 2, or 4) 3666 3590
For power supply population, Table 4-11 on page 93 lists details of the support for compute nodes supported based on type and number of power supplies that are installed in the chassis and the power policy enabled (N+N or N+1). Both the 2500W and 2100W power supplies are 80 PLUS Platinum certified. The 80 PLUS certification is a performance specification for power supplies that are used within servers and computers. The standard has several ratings, such as Bronze, Silver, Gold, Platinum. To meet the 80 PLUS Platinum standard, the power supply must have a power factor (PF) of 0.95 or greater at 50% rated load and efficiency equal to or greater than the following values: 90% at 20% of rated load 94% at 50% of rated load 91% at 100% of rated load For more information about 80 PLUS certification, see this website: http://www.plugloadsolutions.com Table 4-8 lists the efficiency of the 2500W Enterprise Chassis power supplies at various percentage loads at different input voltages.
Table 4-8 2500W power supply efficiency at different loads for 200 - 208 VAC and 220 - 240 VAC Load Input voltage Output power Efficiency 10% load 200- 208V 250 W 93.2% 220- 240V 275 W 93.5% 20% load 200- 208V 500 W 94.2% 220- 240V 550 W 94.4% 50% load 200- 208V 1250 W 94.5% 220- 240V 1375 W 92.2% 100% load 200- 208V 2500W 91.8% 220- 240V 2750 W 91.4%
Table 4-9 lists the efficiency of the 2100W Enterprise Chassis power supplies at various percentage loads at 230 VAC nominal voltage.
Table 4-9 2100W power supply efficiency at different loads for 230 VAC Load @ 230 VAC Output Power Efficiency 10% load 210 W 92.8% 20% load 420 W 94.1% 50% load 1050 W 94.2% 100% load 2100 W 91.8%
Figure 4-8 on page 81 shows the location of the power supplies within the enterprise chassis where two power supplies are installed into bay 4 and bay 1. Four power supply bays are shown with fillers that must be removed to install power supplies into the bays. Similar to the fan bay fillers, there are blue touch point and finger hold apertures (circular) that are located below the blue touch points to make the filler removal process easy and intuitive.
80
IBM PureFlex System and IBM Flex System Products and Technology
Population information for the 2100W and 2500W power supplies can be found in Table 4-11 on page 93, which describes the number of power supplies that are required dependent on the nodes being deployed.
Power supply bay 6
With 2500W power supplies, the chassis allows power configurations to be N+N redundancy with most node types. Table 4-11 on page 93 shows the support matrix. Alternatively, a chassis can operate in N+1, where N can equal 3, 4, or 5. All power supply supplies are combined into a single 12.2v DC power domain within the chassis. This combination distributes power to each of the compute nodes, I/O modules, and ancillary components through the Enterprise Chassis midplane. The midplane is a highly reliable design with no active components. Each power supply is designed to provide fault isolation and is hot swappable. Power monitoring of the DC and AC signals allows the CMM to accurately monitor the power supplies. The integral power supply fans are not dependent upon the power supply being functional because they operate and are powered independently from the chassis midplane. Power supplies are added as required to meet the load requirements of the Enterprise Chassis configuration. There is no need to over provision a chassis and power supplies can be added as the nodes are installed. For more information about power-supply unit planning, see Table 4-11 on page 93. Figure 4-9 on page 82 shows the power supply rear view and highlights the LEDs. There is a handle for removal and insertion of the power supply and a removal latch operated by thumb, so the PSU can easily be unlatched and removed with one hand.
81
The rear of the power supply has a C20 inlet socket for connection to power cables. You can use a C19-C20 power cable, which can connect to a suitable IBM DPI rack power distribution unit (PDU). The Power Supply options that are shown in Table 4-6 on page 79 ship with a 2.5m intra-rack power cable (C19 to C20). The rear LEDs indicate the following conditions: AC Power: When lit green, the AC power is being supplied to the PSU inlet. DC Power: When lit green, the DC power is being supplied to the chassis midplane. Fault: When lit amber, there is a fault with the PSU. Before you remove any power supplies, ensure that the remaining power supplies have sufficient capacity to power the Enterprise Chassis. Power usage information can be found in the CMM web interface.
82
IBM PureFlex System and IBM Flex System Products and Technology
Figure 4-10 shows the fan bays in the back of the Enterprise Chassis.
Fan bay 10 Fan bay 5
Fan bay 7
Fan bay 2
Fan bay 6
Fan bay 1
For more information about how to populate the fan modules, see 4.6, Cooling on page 87. Figure 4-11 shows a 40 mm fan module,
Power on LED
Fault LED
The two 40 mm fan modules in fan bays 5 and 10 distribute airflow to the I/O modules and chassis management modules. These modules ship preinstalled in the chassis. Each 40 mm fan module contains two 40 mm counter rotating fan pairs, side-by-side.
Chapter 4. Chassis and infrastructure configuration
83
The 80 mm fan modules distribute airflow to the compute nodes through the chassis from front to rear. Each 80 mm fan module contains two 80 mm fan modules, back-to-back within the module, which are counter rotating. Both fan modules have an electromagnetic compatibility (EMC) mesh screen on the rear internal face of the module. This design also provides a laminar flow through the screen. Laminar flow is a smooth flow of air, sometimes called streamline flow. This flow reduces turbulence of the exhaust air and improves the efficiency of the overall fan assembly. The following factors combine to form a highly efficient fan design that provides the best cooling for lowest energy input: Design of the entire fan assembly Fan blade design Distance between and size of the fan modules EMC mesh screen Figure 4-12 shows an 80 mm fan module.
Power on LED
Fault LED
The minimum number of 80 mm fan modules is four. The maximum number of individual 80 mm fan modules that can be installed is eight. Both fan modules have two LED indicators that consist of a green power-on indicator and an amber fault indicator. The power indicator lights when the fan module has power, and flashes when the module is in the power save state. Table 4-10 lists the specifications of the 80 mm Fan Module Pair option. Pairs and singles: When the modules are ordered as an option, they are supplied as a pair. When the modules are configured by using feature codes, they are single fans.
Table 4-10 80 mm Fan Module Pair option part number Part number 43W9078 (two fans) Feature codea A0UA / 7805 (one fan) Description IBM Flex System Enterprise Chassis 80 mm Fan Module
a. The first feature code listed is for configurations that are ordered through System x sales channels (HVEC) by using x-config. The second feature code is for configurations that are ordered through the IBM Power Systems channel (AAS) by using e-config.
84
IBM PureFlex System and IBM Flex System Products and Technology
For more information about airflow and cooling, see 4.6, Cooling on page 87.
Fan logic modules are multiplexers for the internal I2C bus, which is used for communication between hardware components within the chassis. Each fan pack is accessed through a dedicated I2C bus, switched by the Fan Mux card, from each CMM. The fan logic module switches the I2C bus to each individual fan pack. This module can be used by the Chassis Management Module to determine multiple parameters, such as fan RPM. There is a fan logic module for the left and right side of the chassis. The left fan logic module access the left fan modules, and the right fan logic module accesses the right fan modules. Fan presence indication for each fan pack is read by the fan logic module. Power and fault LEDs are also controlled by the fan logic module.
85
As shown in Figure 4-14, there are two LEDs on the fan logic module. The power-on LED is green when the fan logic module is powered. The amber fault LED flashes to indicate a faulty fan logic module. Fan logic modules are hot swappable. For more information about airflow and cooling, see 4.6, Cooling on page 87
!
White backlit IBM logo Identify LED Check log LED Fault LED
The following items are shown on the front information panel: White Backlit IBM Logo: When lit, this logo indicates that the chassis is powered. Locate LED: When lit (blue) solid, this LED indicates the location of the chassis. When the LED is flashing, this LED indicates that a condition occurred that caused the CMM to indicate that the chassis needs attention. Check Error Log LED: When lit (amber), this LED indicates that a noncritical event occurred. This event might be an incorrect I/O module that is inserted into a bay, or a power requirement that exceeds the capacity of the installed power modules. Fault LED: When lit (amber), this LED indicates that a critical system error occurred. This error can be an error in a power module or a system error in a node.
86
IBM PureFlex System and IBM Flex System Products and Technology
Figure 4-16 shows the LEDs that are on the rear of the chassis.
Identify LED
Fault LED
Figure 4-16 Chassis LEDs on the rear of the unit (lower right)
4.6 Cooling
This section describes the Enterprise Chassis cooling system. The flow of air within the Enterprise Chassis follows a front-to-back cooling path. Cool air is drawn in at the front of the chassis and warm air is exhausted to the rear. Air is drawn in through the front node bays and the front airflow inlet apertures at the top and bottom of the chassis. There are two cooling zones for the nodes: a left zone and a right zone. The cooling process can be scaled up as required, based on which node bays are populated. For more information about the number of fan modules that are required for nodes, see 4.8, Fan module population on page 99. When a node is removed from a bay, an airflow damper closes in the midplane. Therefore, no air is drawn in through an unpopulated bay. When a node is inserted into a bay, the damper is opened by the node insertion, which allows for cooling of the node in that bay.
87
Figure 4-17 shows the upper and lower cooling apertures. Upper cooling apertures
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa aaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aaaa aaaa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aaaa aa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa aa aa aaaa aaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aa aa aa aa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aaaa aaaa aaaa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaa aaaa aaaa aa a aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaa aaaa aa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaa aa aa aaaa aa aa aaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aaaaaaaaaaaaaaaaaa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a a a a a a a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a a a a a a a a a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaa aaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa aa aaaaaaaaaaaaaaaaaaaaaaaaaa a aaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaa aa a a a a a a a a a a a a a aaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaa aaaa aa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaa aa a a aa aaaaaaaaaaaa aa aa aa aaaaaaaaaaaaaaaaaaaa aa aa aa aa aa aa aa aa aa aa aa aa aaaa aaaa a a a a a a a a a a a a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa a aaaaaaaaaaaaaaaaaaa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a aaaaaaaaaaaaaaaaaaaaaaa a a a a a a a a a a a a a a a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaa a a a a a a a a a a a a a a a a a a a a a a aaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa aaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa a a a a a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aaaaaaaaaaaa aaaaaa aa aa a a a a a a a a a a a a a aa aa aa aa aa aaaaaaaaaaaa aaaaaa aa aa aa aa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aaaaaaaaaaaa aaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aaaaaaaaaaaa aaaa aa aa aa aa aa aa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aaaaaaaaaaaa aaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a a a aaaaaaaaaaaa aa aa aa aa aa aa aa aa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a aaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaa aaaa aaaa aaaaaaaaaaaaaaa aaaaaaaaaaaaaa aa aaaa aa aaaaaaaaaaaaa aa aa aaaaaaaaaaaaaa aa aa aa aa aa aaaaaaaaaaa aa aa aa aa aa aa aaaaaaaaaaaaa aa aa aa aa aa aa aa aa aa aa aa aa aaaaaaaaaaaaa aa aa aa aa aa aa aa aa aa a aa aa aa aaaaaaaaaaaaa aa aa aa aa aa aa aa aa aa a aa aa aa aaaaaaaaaaaaa aa aa aa aa aa aa aa aa aa a a a a aaaaaaaaaaaaa a a a a a a a a a a aaaaaaaaaaaaaaaaaaa a aaaaaaaaaaaaaaaaa aaaa aaaaaaaaaaaaaaa aaaa aa aaaaaaaaaaaaa aa aa aaaaaaaaaaa aa aa aa aa aa aa a a a a a aa aa aa aa aa aaaaaaaaaaa aa aa aa aa aaaaaaaaaaa aaaaaa aa aaaaaaaaaaa aa aaaaaa aaaaaaaaaaa a a a a aaaaaaaaaaa a aaaaaaaaaaaaaaaaa aaaa aaaaaaaaaaaaaaa aaaa aa aaaaaaaaaaaaa aa aa aaaaaaaaaaa aa aa aa aa aa aa a a a a a aa aa aa aa aa aaaaaaaaaaa aa aa aa aa aaaaaaaaaaa aaaa aa aa aaaaaaaaaaa aa aaaa aa aaaaaaaaaaa a a a a aaaaaaaaaaa aaaaaaaaaaaaaaaa aaaa aaaaaaaaaaaaaa aa aa aa aa aa aaaaaaaaaaaaaaa aa aa aaaaaaaaaaaaaaa aa aa aaaaaaaaaaaaaaa aa aa aaaaaaaaaaaaaaa aa aa aaaaaaaaaaaaaaa a a aaaaaaaaaaaaaaa aaaaaaaaaaaaaaaa aaaa aaaaaaaaaaaaaa aa aa aa aa aa aaaaaaaaaaaaaaa aa aa aaaaaaaaaaaaaaa aa aa aaaaaaaaaaaaaaa aa aa aaaaaaaaaaaaaaa a a aaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaa a aaaaaaaaaaaaaaaaa aaaa aaaaaaaaaaaaaaa aaaa aa aaaaaaaaaaaaa aa aa a a a a a a aa aa aa aaaaaaaaaaa aa aa aa aa aa aa aaaaaaaaaaaaa aa aa aa aaaaaaaaaaaaa aa aa aa aaaaaaaaaaaaa a a a aaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaa aaaaaaaaaaaa aa aa aa aa aa aaaa a aaaaaaaa aa aa aaaaaa aa aaaa aaaa aa aaaa aaaa aaaa aaaa aaaa aaa aa aaaa aaaa aaaa aaaaa aaaa aaaa aaaa a aaaaa aa aa aa aa aa aaaaaaaaa a a a a a a a aaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a aaaa aa aaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aaaa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaa aaaa aa aa aa aaaaaaaa aaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aaaaaaaa aa aa aaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a a a a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a a a a a a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaa aaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aaaaaaaaaaaaaaaaaaaa aaaa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa aaaaaaaaaaaaaaaaaa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a aaaa aa aaaaaaaaaaaaaaaaaa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaa aa a a a a a a a a a a aa aa aaaaaaaaaaaaaaaaaaaaaa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aaaa aaaa aaaa aaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaa aa aa aa aaaa aaaa aaaa aa aaaaaa aa aa aaaaaa aa aa aa aa aa aa aa aaaa aaaa aaaa aaaaaaaaaaaaaaaaaaaaa a a a a a a a a a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa aaaa a aaaaaaaaaaaaaaaaaaaaa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa a aaaaaaaaaaaaaaaaaaaaaaaaa a a a a a a a a a a a a a a a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a aaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaa aaaa aaaa aaaaaaaaaaaaaaa aaaaaaaaaaaaaa aa aaaa aa aaaaaaaaaaaaa aa aa aa aa aaaaaaaaaaaaaaa aa aa aaaaaaaaaaa aa aa aa aa aa aaaaaaaaaaaaaaa aa aa aa aa aa aa aa aa aa aa aa aaaaaaaaaaaaaaa aa aa aa aa aa aa aa aa aa a aa aa aaaaaaaaaaaaaaa aa aa aa aa aa aa aa aa aa a a a aaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaa aa aaaaaaaaaaaaaaaaa a a a a a a a a a a
a aaaaaaaaaaaaaaaa a a a a a a a a a a a a a a a a a a a a a aaaa aaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaa aa aaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa a a a a a a a a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aa aa aa aaaaaaaaaaaa aaaaaa aa aa aa aa aa aa aa aaaaaaaaaaaa aaaaaa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa a aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aaaaaaaaaaaa aaaaaa aa aa aa aa aa aa aa aaaaaaaaaaaa aaaa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa a a a aaaaaaaaaaaa aaaa aa aa aa a aa aa aa aa aa aa aa aa aa aa aa aa aaaaaaaaaaaaaaaaaa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa a a a a a a a a a a a a a a a a a a a a aa
Various fan modules are present in the chassis to assist with efficient cooling. Fan modules consist of 40 mm and 80 mm types, and are contained within hot pluggable fan modules. The power supplies also have two integrated, independently powered 40 mm fan modules. The cooling path for the nodes begins when air is drawn in from the front of the chassis. The airflow intensity is controlled by the 80 mm fan modules in the rear. Air passes from the front of the chassis, through the node, through openings in the Midplane, and then into a plenum chamber. Each plenum is isolated from the other, providing separate left and right cooling zones. The 80 mm fan packs on each zone then move the warm air from the plenum to the rear of the chassis. In a two-bay wide node, the air flow within the node is not segregated because it spans both airflow zones.
88
IBM PureFlex System and IBM Flex System Products and Technology
Figure 4-18 shows a chassis with the outer casing removed for clarity to show airflow path through the chassis. There is no airflow through the chassis midplane where a node is not installed. The air damper is opened only when a node is inserted in that bay. Node installed in Bay 14 Cool airflow in 80 mm fan pack
Cool airflow in
Figure 4-18 Airflow into chassis through the nodes and exhaust through the 80 mm fan packs
89
Figure 4-19 shows the path of air from the upper and lower airflow inlet apertures to the power supplies. Nodes Power Supply Cool airflow in
90
IBM PureFlex System and IBM Flex System Products and Technology
Figure 4-20 shows the airflow from the lower inlet aperture to the 40 mm fan modules. This airflow provides cooling for the switch modules and CMM installed in the rear of the chassis.
Airflow
I/O modules
CMM
The right-side 40 mm fan module cools the right switches, while the left 40 mm fan module cools the left pair of switches. Each 40 mm fan module has a pair of counter rotating fans for redundancy. Cool air flows in from the lower inlet aperture at the front of the chassis. It is drawn into the lower openings in the CMM and I/O Modules where it provides cooling for these components. It passes through and is drawn out the top of the CMM and I/O modules. The warm air is expelled to the rear of the chassis by the 40 mm fan assembly. This expulsion is shown by the red airflow arrows in Figure 4-20. The removal of the fan pack exposes an opening in the bay to the 80 mm fan packs that are located below. A back flow damper within the fan bay then closes. The backflow damper prevents hot air from reentering the system from the rear of the chassis. The 80 mm fan packs cool the switch modules and the CMM while the fan pack is being replaced. Chassis cooling is implemented as a function of the following components: Node configurations Power Monitor circuits Component temperatures Ambient temperature
91
This results in lower airflow volume (measured in cubic feet per minute or CFM) and lower cooling energy that is spent at a chassis level. This system also maximizes the temperature difference across the chassis (known generally as the Delta T) for more efficient room integration. Monitored Chassis level airflow usage is displayed to enable airflow planning and monitoring for hot air recirculation. Five Acoustic Optimization states can be selected. Use the one that best balances performance requirements with the noise level of the fans. Chassis level CFM usage is available to you for planning purposes. In addition, ambient health awareness can detect potential hot air recirculation to the chassis.
92
IBM PureFlex System and IBM Flex System Products and Technology
In Table 4-11, the colors of the cells have the following meanings: Supported with no limitations as to the number of compute nodes that can be installed Supported but with limitations on the number of compute nodes that can be installed. As you can see, a full complement of any compute nodes at all TDP ratings are supported if all six power supplies are installed and an N+1 power policy is selected.
Table 4-11 Specific number of compute nodes supported based on installed power supplies
Compute node CPU TDP rating 50 W 60 W 70 W 80 W 95 W x222 50 W 60 W 70 W 80 W 95 W x240 60 W 70 W 80 W 95 W 115 W 130 W 135 W x440 95 W 115 W 130 W p24L p260 p270 p460 FSM V7000 All All All All 95 W N/A 2100W power supplies N+1, N=5 6 total 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 7 7 7 14 14 14 7 2 3 N+1, N=4 5 total 14 14 14 14 14 14 14 14 14 13 14 14 14 14 14 14 14 7 7 7 12 12 12 6 2 3 N+1, N=3 4 total 14 14 14 14 14 13 12 11 10 9 14 13 13 12 11 11 10 6 5 5 9 9 9 4 2 3 N+N, N=3 6 total 14 14 14 14 14 14 13 12 11 10 14 14 13 12 12 11 11 6 6 5 10 10 9 5 2 3 N+1, N=5 6 total 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 7 7 7 14 14 14 7 2 3 2500W power supplies N+1, N=4 5 total 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 7 7 7 14 14 14 7 2 3 N+1, N=3 4 total 14 14 14 14 14 14 14 14 13 12 14 14 14 14 14 13 13 7 7 6 12 12 12 6 2 3 N+N, N=3 6 total 14 14 14 14 14 14 14 14 14 13 14 14 14 14 14 14 14 7 7 7 13 13 12 6 2 3
x220
93
The following assumptions are made: All Compute Nodes are fully configured. Throttling and oversubscription are enabled. Tip: For more information about exact configuration support, see the Power configurator at this website: http://ibm.com/systems/bladecenter/resources/powerconfig.html
94
IBM PureFlex System and IBM Flex System Products and Technology
14 13 12 11 10 9 8 7 6 5 4 3 2 1
These configuratio ns are not applicable. It is not physically possible to install more than seven p460s into a chassis. NSb 6 6 6 4 4 2 5 4 4 4 3 3 2
a. Number of power supplies is based on x86 compute nodes with processors of the highest TDP rating. b. Not supported. The number of nodes exceeds the capacity of the power supplies.
95
Table 4-13 Number of 2100W power supplies required for each node type Nodes x220 at 95Wa N+N 6 6 6 6 6 4 4 4 4 4 4 4 2 2 N+1 4 4 4 4 4 3 3 3 3 3 3 3 2 2 x222 at 95Wa N+N NSb NSb NS
b
N+1 6 5 5 5 5 4 4 4 4 3 3 3 3 2
N+1 5 5 5 5 4 4 4 4 3 3 3 3 3 2
N+1 6 6 5 5 5 4 4 4 4 3 3 3 3 2
14 13 12 11 10 9 8 7 6 5 4 3 2 1
NSb 6 6 6 6 6 4 4 4 4 2
Not applicable. It is not physically possible to install more than seven x440s into a chassis.
NSb NS
b
NSb 6 6 6
NSb NSb 6 6 6 6 4 4 4 4 2
These configuratio ns are not applicable. It is not physically possible to install more than seven p460s into a chassis. NSb NSb 6 6 6 4 4 6 5 5 4 4 3 3
NSb NSb 6 6 4 4 4
5 5 4 4 3 3 3
6 6 4 4 4 4 2
a. Number of power supplies is based on x86 compute nodes with processors of the highest TDP rating. b. Not supported. The number of nodes exceeds the capacity of the power supplies.
Tip: For more information about the exact configuration, see the Power configurator at this website: http://ibm.com/systems/bladecenter/resources/powerconfig.html
96
IBM PureFlex System and IBM Flex System Products and Technology
The chassis ships with Power supply bay 1 and 4 preinstalled, as shown in Figure 4-21. In the case of N+N, this can power four x220 nodes as shown, with 2500W power supplies according to Table 4-12 on page 95.
Figure 4-21 Two power supplies installed with four x220 nodes in N+N
Eight x220 nodes with 2500W N+N configuration is shown in Figure 4-22 where another pair of power supplies in bays 2 and 5 were installed into the enterprise chassis.
Figure 4-22 Four power supplies installed with eight x220 nodes in N+N
97
Figure 4-23 shows the full six power supplies installed with 14 x220 nodes that use 2500W power supplies. This is a supported N+N configuration according to Table 4-12 on page 95.
Figure 4-23 Six power configuration with fourteen x220 nodes in N+N.
Figure 4-24 Two power supplies installed with four x220 nodes in N+1
98
IBM PureFlex System and IBM Flex System Products and Technology
When eight x220 nodes are installed and N+1 with 2500W power supplies is required, checking Table 4-12 on page 95 shows support with three power supplies, as shown in Figure 4-25.
Figure 4-25 Eight x220 nodes with 3 2500W power supplies in N+1 configuration
When 14 x220 nodes are required and N+1 is wanted that uses 2500W power supplies, then four 2500W power supplies are required according to Table 4-12 on page 95. Figure 4-26 shows this redundancy configuration of N+1 where in this case N=3.
Figure 4-26 Fourteen x220 nodes with 4 2500W power supplies in N+1 configuration
99
The minimum configuration of 80 mm fan modules is four, which provides cooling for a maximum of four nodes. This configuration is shown in Figure 4-27 and is the base configuration.
Rear View
Figure 4-27 Four 80 mm fan modules allow a maximum of four nodes installed
Installing six 80 mm fan modules allows another four nodes to be supported within the chassis. Therefore, the maximum is eight, as shown in Figure 4-28.
Rear View
Figure 4-28 Six 80 mm fan modules allow for a maximum of eight nodes
100
IBM PureFlex System and IBM Flex System Products and Technology
To cool more than eight nodes, all fan modules must be installed as shown in Figure 4-29.
14 14 12 12 10 10 8 8 6 6 4 4 2 2 9 8 4 3
7 6 Cooling zone
2 1 Cooling zone
Rear View
If there are insufficient fan modules for the number of nodes that are installed, the nodes might be throttled.
a. The first feature code listed is for configurations that are ordered through System x sales channels (HVEC) using x-config. The second feature code is for configurations that are ordered through the IBM Power Systems channel (AAS) by using e-config.
101
Figure 4-30 shows the location of the CMM bays on the back of the Enterprise Chassis.
The CMM provides the following functions: Power control Fan management Chassis and compute node initialization Switch management Diagnostics Resource discovery and inventory management Resource alerts and monitoring management Chassis and compute node power management Network management The CMM includes the following connectors: USB connection: Can be used for insertion of a USB media key for tasks such as firmware updates. 10/100/1000 Mbps RJ45 Ethernet connection: For connection to a management network. The CMM can be managed through this Ethernet port. Serial port (mini-USB): For local serial (CLI) access to the CMM. Use the cable kit that is listed in Table 4-15 for connectivity.
Table 4-15 Serial cable specifications Part number 90Y9338 Feature codea A2RR Description IBM Flex System Management Serial Access Cable Contains two cables: Mini-USB-to-RJ45 serial cable Mini-USB-to-DB9 serial cable
a. The feature code listed is for both the System x sales channel (HVEC) using x-config and the Power Systems sales channel (AAS) using e-config.
102
IBM PureFlex System and IBM Flex System Products and Technology
The CMM includes the following LEDs that provide status information: Power-on LED Activity LED Error LED Ethernet port link and port activity LEDs Figure 4-31 shows the CMM connectors and LEDs.
The CMM also incorporates a reset button, which features the following functions (depending upon how long the button is held in): When pressed for less than 5 seconds, the CMM restarts. When pressed for more than 5 seconds (for example 10 - 15 seconds), the CMM configuration is reset to manufacturing defaults and then restarts. For more information about how the CMM integrates into the Systems Management architecture, see 3.2, Chassis Management Module on page 43.
103
Figure 4-32 Rear view that shows the I/O Module bays 1 - 4
If a node has a two-port integrated LAN on Motherboard (LOM) as standard, modules 1 and 2 are connected to this LOM. If an I/O adapter is installed in the nodes I/O expansion slot 1, modules 1 and 2 are connected to this adapter. Modules 3 and 4 connect to the I/O adapter that is installed within I/O expansion bay 2 on the node. These I/O modules provide external connectivity, and connect internally to each of the nodes within the chassis. They can be Switch or Pass-thru modules, with a potential to support other types in the future.
104
IBM PureFlex System and IBM Flex System Products and Technology
Figure 4-33 shows the connections from the nodes to the switch modules.
LOM connector (remove when I/O expansion adapter is installed)
I/O module 1
I/O module 3
I/O module 4
Node bay 14
The node in bay 1 in Figure 4-33 shows that when shipped with an LOM, the LOM connector provides the link from the node system board to the midplane. Some nodes do not ship with LOM. If required, this LOM connector can be removed and an I/O expansion adapter can be installed in its place. This configuration is shown on the node in bay 2 in Figure 4-33
105
Figure 4-34 shows the electrical connections from the LOM and I/O adapters to the I/O modules, which all takes place across the chassis midplane.
Node M1 1 M2
Switch . . . 1
Node M1 2 M2
Switch . . . 2
Node M1 3 M2
Switch . . . 3
Node M1 14 M2
Switch . . . 4
A total of two I/O expansion adapters (designated M1 and M2 in Figure 4-34) can be plugged into a half-wide node. Up to four I/O adapters can be plugged into a full-wide node. Each I/O adapter has two connectors. One connects to the compute nodes system board (PCI Express connection). The second connector is a high-speed interface to the midplane that mates to the midplane when the node is installed into a bay within the chassis. As shown in Figure 4-34, each of the links to the midplane from the I/O adapter (shown in red) are four links wide. Exactly how many links are used on each I/O adapter is dependent on the design of the adapter and the number of ports that are wired. Therefore, a half-wide node can have a maximum of 16 I/O links and a full wide node can have 32 links.
106
IBM PureFlex System and IBM Flex System Products and Technology
Each of these individual I/O links or lanes can be wired for 1 Gb or 10 Gb Ethernet, or 8 Gbps or 16 Gbps Fibre Channel. The application-specific integrated circuit (ASIC) type on the I/O Expansion adapter dictates the number of links that are enabled. Some ASICs are two-port and some are four port and some I/O expansion adapters contain two ASICs. For a two-port ASIC, one port can go to one switch and one port to the other. This configuration is shown in Figure 4-36 on page 108. In the future, other combinations can be implemented. In an Ethernet I/O adapter, the wiring of the links is to the IEEE 802.3ap standard, which is also known as the Backplane Ethernet standard. The Backplane Ethernet standard has different implementations at 10 Gbps, being 10GBASE-KX4 and 10GBASE-KR. The I/O architecture of the Enterprise Chassis supports the KX4 and KR. The 10GBASE-KX4 uses the same physical layer coding (IEEE 802.3 clause 48) as 10GBASE-CX4, where each individual lane (SERDES = Serializer/DeSerializer) carries 3.125 Gbaud of signaling bandwidth. The 10GBASE-KR uses the same coding (IEEE 802.3 clause 49) as 10GBASE-LR/ER/SR, where the SERDES lane operates at 10.3125 Gbps. Each of the links between I/O expansion adapter and I/O module can be 4x 3.125 Lanes/port (KX-4) or 4x 10 Gbps Lanes (KR). This choice is dependent on the expansion adapter and I/O Module implementation.
107
Figure 4-36 shows how the integrated two-port 10 Gb LOM connects through a LOM connector to switch 1. This implementation provides a pair of 10 Gb lanes. Each lane connects to a 10 Gb switch or 10 Gb pass-through module that is installed in I/O module bays in the rear of the chassis. The LOM connector is sometimes referred to as a periscope connector because of its shape.
10 Gbps KR lane
P1 LOM Connector P1 P2 LOM Half-wide node P1 2-Port I/O adapter in slot 1 I/O adapter in slot 2
P2
Figure 4-36 LOM implementation: Emulex 10 Gb Virtual Fabric onboard LOM to I/O Module
A half-wide compute node with two standard I/O adapter sockets and an I/O adapter with two ports is shown in Figure 4-37. Port 1 connects to one switch in the chassis and Port 2 connects to another switch in the chassis. With 14 compute nodes of this configuration installed in the chassis, each switch requires 14 internal ports for connectivity to the compute nodes.
P1 P3 P5 P7 P2 P4 P6 P8
x1 Ports
P2
x1 Ports
P1 P3 P5 P7 P2 P4 P6 P8 I/O modules
108
IBM PureFlex System and IBM Flex System Products and Technology
Another possible implementation of the I/O adapter is the four-port. Figure 4-38 shows the interconnection to the I/O module bays for such I/O adapters that uses a single four-port ASIC.
P1 P3 P5 P7 P2 P4 P6 P8
x1 Ports x1 Ports
P1 P3 P5 P7 P2 P4 P6 P8 I/O modules
In this case, with each node having a four-port I/O adapter in I/O adapter slot 1, each I/O module requires 28 internal ports enabled. This configuration highlights another key feature of the I/O architecture: scalable on-demand port enablement. Sets of ports are enabled by using IBM Features on Demand (FoD) activation licenses to allow a greater number of connections between nodes and a switch. With two lanes per node to each switch and 14 nodes requiring four ports that are connected, each switch must have 28 internal ports enabled. You also need sufficient uplink ports enabled to support the wanted bandwidth. FoD feature upgrades enable these ports. Finally, Figure 4-39 on page 110 shows an eight-port I/O adapter that is using two, four-port ASICs.
109
Half-wide node
P0 P1 P4 P5 P2 P3 P6 P7
P1 P3 P5 P7 P2 P4 P6 P8
ASIC 4-Port
P1 P3 P5 P7 P2 P4 P6 P8 I/O modules
Six ports active: In the case of the CN4058 8-port 10Gb Converged Adapter, although this is a eight port adapter, the currently available switches only support up to six of those ports (three ports to each of two installed switches). With these switches, three of the four lanes per module can be enabled.
110
IBM PureFlex System and IBM Flex System Products and Technology
The architecture allows for a total of eight lanes per I/O adapter, as shown in Figure 4-40. Therefore, a total of 16 I/O lanes per half wide node is possible. Each I/O module requires the matching number of internal ports to be enabled.
Node A1 bay 1 A2
. .. . Switch . . .. .. .. . bay 1 . .
Node A1 bay 2 A2
. .. . Switch . . .. .. .. . bay 3 . .
. .. . Switch . . .. .. .. . bay 2 . .
. Switch . . .. . .. . bay 4 . .. .. .
For more information about port enablement by using FoD, see 4.11, I/O modules on page 112. For more information about I/O expansion adapters that install on the nodes, see 5.8.1, Overview on page 335.
111
OK
Identify
Switch error
112
IBM PureFlex System and IBM Flex System Products and Technology
The LEDs indicate the following conditions: OK (power) When this LED is lit, it indicates that the switch is on. When it is not lit and the amber switch error LED is lit, it indicates a critical alert. If the amber LED is also not lit, it indicates that the switch is off. Identify You can physically identify a switch by making this blue LED light up by using the management software. Switch Error When this LED is lit, it indicates a POST failure or critical alert. When this LED is lit, the system-error LED on the chassis is also lit. When this LED is not lit and the green LED is lit, it indicates that the switch is working correctly. If the green LED is also not lit, it indicates that the switch is off
a. The feature code listed is for both the System x sales channel (HVEC) using x-config and the Power Systems sales channel (AAS) using e-config.
The part number 90Y9338 includes the following cables: Mini-USB-to-RJ45 serial cable Mini-USB-to-DB9 serial cable
113
EN2092
Fabric Type: EN = Ethernet FC = Fibre Channel CN = Converged Network IB = InfiniBand SI = System Interconnect Series: 2 for 1 Gb 3 for 8 Gb 4 for 10 Gb 5 for 16 Gb 6 for 56 Gb & 40 Gb Vendor name where A=01: 02 = Brocade 09 = IBM 13 = Mellanox 17 = QLogic Maximum number of ports available to each node: 1 = One 2 = Two 3 = Three
114
IBM PureFlex System and IBM Flex System Products and Technology
a. The first feature code that is listed is for configurations that are ordered through System x sales channels (XCC by using x-config). The second feature code is for configurations that are ordered through the IBM Power Systems channel (AAS by using e-config) b. 1 Gb is supported on the CN4093s two external 10 Gb SFP+ ports only. The 12 external Omni Ports do not support 1 GbE speeds. c. Either Upgrade 1 or Upgrade 2 is required to enable enough internal switch ports to connect to both servers in the x222. d. Only two of the ports of this adapter are connected when used with the EN4091 10Gb Pass-thru. e. Only four of the eight ports of CN4058 adapter are connected with the EN2092 switch. f. Only six of the eight ports of the CN4058 adapter are connected with the CN4093, EN4093, EN4093R switches.
115
FC5022 16Gb 24-port 00Y3324 A3DP / ESW5 Yes Yes Yes Yes Yes Yes Yes
FC5022 16Gb 24-port ESB 90Y9356 A2RQ / 3771 Yes Yes Yes Yes Yes Yes Yes
Feature codes (XCC / AAS)a A1BM / 1764 A2N5 / EC25 A1BP / EC2B A45R / EC23 A45S / EC2E A1BQ / A1BQ A3HU / A3HU
88Y6374 A1EH / 3770 Yes Yes Yes Yes Yes Yes Yes
a. The first feature code that is listed is for configurations that are ordered through System x sales channels (XCC by using x-config). The second feature code is for configurations that are ordered through the IBM Power Systems channel (AAS by using e-config)
a. The first feature code that is listed is for configurations that are ordered through System x sales channels (XCC by using x-config). The second feature code is for configurations that are ordered through the IBM Power Systems channel (AAS by using e-config) b. To operate at FDR speeds, the IB6131 switch needs the FDR upgrade. For more information, see Table 4-44 on page 160.
116
IBM PureFlex System and IBM Flex System Products and Technology
This 40 Gb Ethernet solution can deploy more workloads per server without running into I/O bottlenecks. If there are failures or server maintenance, clients can also move their virtual machines much faster by using 40 Gb interconnects within the chassis. The 40 GbE switch and adapter are designed for low latency, high bandwidth, and computing efficiency for performance-driven server and storage clustering applications. They provide extreme scalability for low-latency clustered solutions with reduced packet hops. The IBM Flex System 40 GbE solution offers the highest bandwidth without adding any significant power impact to the chassis. It can also help increase the system usage and decrease the number of network ports for further cost savings.
Switch LEDs
Figure 4-45 External ports of the IBM Flex System EN6131 40Gb Ethernet Switch
The front panel contains the following components: LEDs that shows the following statuses of the module and the network: Green power LED indicates that the module passed the power-on self-test (POST) with no critical faults and is operational. Identify LED: This blue LED can be used to identify the module physically by illuminating it through the management software. The fault LED (switch error) indicates that the module failed the POST or detected an operational fault. Eighteen external QSFP+ ports for 10 Gbps, 20 Gbps, or 40 Gbps connections to the external network devices. An Ethernet physical link LED and an Ethernet Tx/Rx LED for each external port on the module.
117
One mini-USB RS-232 console port that provides another means to configure the switch module. This mini-USB-style connector enables the connection of a special serial cable (the cable is optional and it is not included with the switch). For more information, see Table 4-21. Table 4-20 shows the part number and feature codes that are used to order the EN6131 40Gb Ethernet Switch.
Table 4-20 Part number and feature code for ordering Description IBM Flex System EN6131 40Gb Ethernet Switch Part number 90Y9346 Feature code (x-config / e-config) A3HJ / ESW6
QSFP+ Transceivers ordering: No QSFP+ (quad small form-factor pluggable plus) transceivers or cables are included with the switch. They must be ordered separately. The switch does not include a serial management cable. However, IBM Flex System Management Serial Access Cable 90Y9338 is supported and contains two cables, a mini-USB-to-RJ45 serial cable and a mini-USB-to-DB9 serial cable, either of which can be used to connect to the switch module locally for configuration tasks and firmware updates. Table 4-21 lists the supported cables and transceivers.
Table 4-21 Supported transceivers and direct attach cables Description Serial console cables IBM Flex System Management Serial Access Cable Kit QSFP+ transceiver and optical cables - 40 GbE IBM QSFP+ 40GBASE-SR Transceiver (Requires either cable 90Y3519 or cable 90Y3521) 10m IBM MTP Fiber Optical Cable (requires transceiver 49Y7884) 30m IBM MTP Fiber Optical Cable (requires transceiver 49Y7884) QSFP+ direct-attach cables - 40 GbE 3m FDR InfiniBand Cable 3m IBM QSFP+ to QSFP+ Cable 5m IBM QSFP+ to QSFP+ Cable 7m IBM QSFP+ to QSFP+ Cable 90Y3470 49Y7891 00D5810 00D5813 A227 / None A1DQ / EB2H A2X8 / ECBN A2X9 / ECBP 49Y7884 90Y3519 90Y3521 A1DR / EB27 A1MM / EB2J A1MN / EB2K 90Y9338 A2RR / A2RR Part number Feature code (x-config / e-config)
The EN6131 40Gb Ethernet Switch has the following features and specifications: MLNX-OS operating system Internal ports: A total of 14 internal full-duplex 40 Gigabit ports (10, 20, or 40 Gbps auto-negotiation). One internal full-duplex 1 GbE port that is connected to the chassis management module. 118
IBM PureFlex System and IBM Flex System Products and Technology
External ports: A total of 18 ports for 40 Gb Ethernet QSFP+ transceivers or QSFP+ DACs (10, 20, or 40 Gbps auto-negotiation). QSFP+ modules and DACs are not included and must be purchased separately. One external 1 GbE port with RJ-45 connector for switch configuration and management. One RS-232 serial port (mini-USB connector) that provides another means to configure the switch module. Scalability and performance: 40 Gb Ethernet ports for extreme bandwidth and performance. Non-blocking architecture with wire-speed forwarding of traffic and an aggregated throughput of 1.44 Tbps. Support for up to 48,000 unicast and up to 16,000 multicast media access control (MAC) addresses per subnet. Static and LACP (IEEE 802.3ad) link aggregation, up to 720 Gb of total uplink bandwidth per switch, up to 36 link aggregation groups (LAGs), and up to 16 ports per LAG. Support for jumbo frames (up to 9,216 bytes). Broadcast/multicast storm control. IGMP snooping to limit flooding of IP multicast traffic. Fast port forwarding and fast uplink convergence for rapid STP convergence. Availability and redundancy: IEEE 802.1D STP for providing L2 redundancy. IEEE 802.1w Rapid STP (RSTP) provides rapid STP convergence for critical delay-sensitive traffic such as voice or video. VLAN support: Up to 4094 VLANs are supported per switch, with VLAN numbers 1 - 4094. 802.1Q VLAN tagging support on all ports. Security: Up to 24,000 rules with VLAN-based, MAC-based, protocol-based, and IP-based access control lists (ACLs). User access control (multiple user IDs and passwords). RADIUS, TACACS+, and LDAP authentication and authorization. Quality of service (QoS): Support for IEEE 802.1p traffic processing. Traffic shaping that is based on defined policies. Four Weighted Round Robin (WRR) priority queues per port for processing qualified traffic. Priority-Based Flow Control (PFC) (IEEE 802.1Qbb) extends 802.3x standard flow control to allow the switch to pause traffic based on the 802.1p priority value in each packets VLAN tag. Enhanced Transmission Selection (ETS) (IEEE 802.1Qaz) provides a method for allocating link bandwidth based on the 802.1p priority value in each packets VLAN tag.
119
Manageability: IPv4 and IPv6 host management. Simple Network Management Protocol (SNMP V1, V2, and V3). Web-based GUI. Industry standard CLI (IS-CLI) through Telnet, SSH, and serial port. Link Layer Discovery Protocol (LLDP) to advertise the device's identity, capabilities, and neighbors. Firmware image update (TFTP, FTP, and SCP). Network Time Protocol (NTP) for clock synchronization. Monitoring: Switch LEDs for external port status and switch module status indication. Port mirroring for analyzing network traffic passing through the switch. Change tracking and remote logging with the syslog feature. Support for sFLOW agent for monitoring traffic in data networks (separate sFLOW collector/analyzer is required elsewhere). POST diagnostic tests. The switch supports the following Ethernet standards: IEEE 802.1AB Link Layer Discovery Protocol IEEE 802.1D Spanning Tree Protocol (STP) IEEE 802.1p Class of Service (CoS) prioritization IEEE 802.1Q Tagged VLAN (frame tagging on all ports when VLANs are enabled) IEEE 802.1Qbb Priority-Based Flow Control (PFC) IEEE 802.1Qaz Enhanced Transmission Selection (ETS) IEEE 802.1w Rapid STP (RSTP) IEEE 802.3ab 1000BASE-T copper twisted pair Gigabit Ethernet IEEE 802.3ad Link Aggregation Control Protocol IEEE 802.3ba 40GBASE-SR4 short range fiber optics 40 Gb Ethernet IEEE 802.3ba 40GBASE-CR4 copper 40 Gb Ethernet IEEE 802.3u 100BASE-TX Fast Ethernet IEEE 802.3x Full-duplex Flow Control The EN6131 40Gb Ethernet Switch can be installed in bays 1, 2, 3, and 4 of the Enterprise Chassis. A supported Ethernet adapter must be installed in the corresponding slot of the compute node (slot A1 when I/O modules are installed in bays 1 and 2 or slot A2 when I/O modules are installed in bays 3 and 4). If a four-port 10 GbE adapter is used, only up to two adapter ports can be used with the EN6131 40Gb Ethernet Switch (one port per switch). For more information including example configurations, see the IBM Redbooks Product Guide IBM Flex System EN6131 40Gb Ethernet Switch, TIPS0911, which is available at this website: http://www.redbooks.ibm.com/abstracts/tips0911.html?Open
120
IBM PureFlex System and IBM Flex System Products and Technology
4.11.6 IBM Flex System Fabric CN4093 10Gb Converged Scalable Switch
The IBM Flex System Fabric CN4093 10Gb Converged Scalable Switch provides unmatched scalability, performance, convergence, and network virtualization. It also delivers innovations to help address a number of networking concerns and provides capabilities that help you prepare for the future. The switch offers full Layer 2/3 switching and FCoE Full Fabric and Fibre Channel NPV Gateway operations to deliver a converged and integrated solution. It is installed within the I/O module bays of the IBM Flex System Enterprise Chassis. The switch can help you migrate to a 10 Gb or 40 Gb converged Ethernet infrastructure and offers virtualization features such as Virtual Fabric and IBM VMready, and the ability to work with IBM Distributed Virtual Switch 5000V. Figure 4-46 shows the IBM Flex System Fabric CN4093 10Gb Converged Scalable Switch.
Figure 4-46 IBM Flex System Fabric CN4093 10 Gb Converged Scalable Switch
The CN4093 switch is initially licensed for 14 10-GbE internal ports, two external 10-GbE SFP+ ports, and six external Omni Ports enabled. The following ports can be enabled: A total of 14 more internal ports and two external 40 GbE QSFP+ uplink ports with Upgrade 1. A total of 14 more internal ports and six more external Omni Ports with the Upgrade 2 license options. Upgrade 1 and Upgrade 2 can be applied on the switch independently from each other or in combination for full feature capability. Table 4-22 shows the part numbers for ordering the switches and the upgrades.
Table 4-22 Part numbers and feature codes for ordering Description Switch module IBM Flex System Fabric CN4093 10Gb Converged Scalable Switch Features on Demand upgrades IBM Flex System Fabric CN4093 Converged Scalable Switch (Upgrade 1) IBM Flex System Fabric CN4093 Converged Scalable Switch (Upgrade 2) 00D5845 00D5847 A3HL / ESU1 A3HM / ESU2 00D5823 A3HH / ESW2 Part number Feature code (x-config / e-config)
121
Description Management cable IBM Flex System Management Serial Access Cable
Part number
90Y9338
Neither QSFP+ or SFP+ transceivers or cables are included with the switch. They must be ordered separately (see Table 4-24 on page 124). The switch does not include a serial management cable. However, IBM Flex System Management Serial Access Cable 90Y9338 is supported and contains two cables, a mini-USB-to-RJ45 serial cable and a mini-USB-to-DB9 serial cable, either of which can be used to connect to the switch locally for configuration tasks and firmware updates. The following base switch and upgrades are available: Part number 00D5823 is for the physical device, which comes with 14 internal 10 GbE ports enabled (one to each node bay), two external 10 GbE SFP+ ports that are enabled to connect to a Top of Rack switch or other devices identified as EXT1 and EXT2, and six Omni Ports that are enabled to connect to Ethernet or Fibre Channel networking infrastructure, depending on the SFP+ cable or transceiver that is used. The six Omni ports are from the 12 labeled on the switch as EXT11 through EXT22. Part number 00D5845 (Upgrade 1) can be applied on the base switch when you need more uplink bandwidth with two 40 GbE QSFP+ ports that can be converted into 4x 10 GbE SFP+ DAC links with the optional break-out cables. These are labeled EXT3, EXT7 or EXT3-EXT6, EXT7-EXT10 if converted. This upgrade also enables 14 more internal ports, for a total of 28 ports, to provide more bandwidth to the compute nodes using 4-port expansion cards. Part number 00D5847 (Upgrade 2) can be applied on the base switch when you need more external Omni Ports on the switch or if you want more internal bandwidth to the node bays. The upgrade enables the remaining six external Omni Ports from range EXT11 through EXT22, plus 14 internal 10 Gb ports, for a total of 28 internal ports, to provide more bandwidth to the compute nodes using four-port expansion cards. Both 00D5845 (Upgrade 1) and 00D5847 (Upgrade 2) can be applied on the switch at the same time so that you can use six ports on an eight-port expansion card, and use all the external ports on the switch. Table 4-23 shows the switch upgrades and the ports they enable.
Table 4-23 CN4093 10 Gb Converged Scalable Switch part numbers and port upgrades Part number Feature codea Description Internal 10Gb Base switch (no upgrades) Add Upgrade 1 Add Upgrade 2 Add both Upgrade 1 and Upgrade 2 14 28 28 42 Total ports that are enabled External 10Gb SFP+ 2 2 2 2 External 10Gb Omni 6 6 12 12 External 40Gb QSFP+ 0 2 0 2
A3HH / ESW2 A3HL / ESU1 A3HM / ESU2 A3HL / ESU1 A3HM / ESU2
a. The first feature code listed is for configurations ordered through System x sales channels (HVEC) using x-config. The second feature code is for configurations ordered through the IBM Power Systems channel (AAS) using e-config.
122
IBM PureFlex System and IBM Flex System Products and Technology
Each upgrade license enables more internal ports. To make full use of those ports, each compute node needs the following appropriate I/O adapter installed: The base switch requires a two-port Ethernet adapter (one port of the adapter goes to each of two switches). Adding Upgrade 1 or Upgrade 2 requires a four-port Ethernet adapter (two ports of the adapter to each switch) to use all the internal ports. Adding both Upgrade 1 and Upgrade 2 requires a six-port Ethernet adapter (three ports to each switch) to use all the internal ports.
Front panel
Figure 4-47 shows the main components of the CN4093 switch.
2x 10 Gb ports 2x 40 Gb uplink ports (standard) (enabled with Upgrade 1) 12x Omni Ports (6 standard, 6 with Upgrade 2)
SFP+ ports
QSFP+ ports
SFP+ ports
Management ports
Switch LEDs
Figure 4-47 IBM Flex System Fabric CN4093 10 Gb Converged Scalable Switch
The front panel contains the following components: LEDs that shows the status of the switch module and the network: The OK LED indicates that the switch module passed the power-on self-test (POST) with no critical faults and is operational. Identify: You can use this blue LED to identify the switch physically by illuminating it through the management software. The error LED (switch module error) indicates that the switch module failed the POST or detected an operational fault. One mini-USB RS-232 console port that provides another means to configure the switch module. This mini-USB-style connector enables connection of a special serial cable. (The cable is optional and it is not included with the switch. For more information, see Table 4-24 on page 124. Two external SFP+ ports for 1 Gb or 10 Gb connections to external Ethernet devices. Twelve external SFP+ Omni Ports for 10 Gb connections to the external Ethernet devices or 4/8 Gb FC connections to the external SAN devices. Omni Ports support: 1 Gb is not supported on Omni Ports. Two external QSFP+ port connectors to attach QSFP+ modules or cables for a single 40 Gb uplink per port or splitting of a single port into 4x 10 Gb connections to external Ethernet devices. A link OK LED and a Tx/Rx LED for each external port on the switch module. A mode LED for each pair of Omni Ports indicating the operating mode. (OFF indicates that the port pair is configured for Ethernet operation, and ON indicates that the port pair is configured for Fibre Channel operation.)
123
124
IBM PureFlex System and IBM Flex System Products and Technology
125
Availability and redundancy: Virtual Router Redundancy Protocol (VRRP) for Layer 3 router redundancy. IEEE 802.1D STP for providing L2 redundancy. IEEE 802.1s Multiple STP (MSTP) for topology optimization. Up to 32 STP instances are supported by a single switch. IEEE 802.1w Rapid STP (RSTP) provides rapid STP convergence for critical delay-sensitive traffic, such as voice or video. Per-VLAN Rapid STP (PVRST) enhancements. Layer 2 Trunk Failover to support active/standby configurations of network adapter teaming on compute nodes. Hot Links provides basic link redundancy with fast recovery for network topologies that require Spanning Tree to be turned off. VLAN support: Up to 1024 VLANs supported per switch, with VLAN numbers from 1 - 4095 (4095 is used for management modules connection only). 802.1Q VLAN tagging support on all ports. Private VLANs. Security: VLAN-based, MAC-based, and IP-based access control lists (ACLs). 802.1x port-based authentication. Multiple user IDs and passwords. User access control. Radius, TACACS+, and LDAP authentication and authorization. Quality of service (QoS): Support for IEEE 802.1p, IP ToS/DSCP, and ACL-based (MAC/IP source and destination addresses, VLANs) traffic classification and processing. Traffic shaping and re-marking based on defined policies. Eight Weighted Round Robin (WRR) priority queues per port for processing qualified traffic. IP v4 Layer 3 functions: Host management. IP forwarding. IP filtering with ACLs, with up to 896 ACLs supported. VRRP for router redundancy. Support for up to 128 static routes. Routing protocol support (RIP v1, RIP v2, OSPF v2, and BGP-4), for up to 2048 entries in a routing table. Support for DHCP Relay. Support for IGMP snooping and IGMP relay. Support for Protocol Independent Multicast (PIM) in Sparse Mode (PIM-SM) and Dense Mode (PIM-DM).
126
IBM PureFlex System and IBM Flex System Products and Technology
IP v6 Layer 3 functions: IPv6 host management (except for a default switch management IP address). IPv6 forwarding. Up to 128 static routes. Support for OSPF v3 routing protocol. IPv6 filtering with ACLs. Virtualization: Virtual NICs (vNICs): Ethernet, iSCSI, or FCoE traffic is supported on vNICs. Unified fabric ports (UFPs): Ethernet or FCoE traffic is supported on UFPs 802.1Qbg Edge Virtual Bridging (EVB) is an emerging IEEE standard for allowing networks to become virtual machine (VM)-aware: Virtual Ethernet Bridging (VEB) and Virtual Ethernet Port Aggregator (VEPA) are mechanisms for switching between VMs on the same hypervisor. Edge Control Protocol (ECP) is a transport protocol that operates between two peers over an IEEE 802 LAN providing reliable and in-order delivery of upper layer protocol data units. Virtual Station Interface (VSI) Discovery and Configuration Protocol (VDP) allows centralized configuration of network policies that persists with the VM, independent of its location. EVB Type-Length-Value (TLV) is used to discover and configure VEPA, ECP, and VDP.
VMready. Converged Enhanced Ethernet: Priority-Based Flow Control (PFC) (IEEE 802.1Qbb) extends 802.3x standard flow control to allow the switch to pause traffic that is based on the 802.1p priority value in each packets VLAN tag. Enhanced Transmission Selection (ETS) (IEEE 802.1Qaz) provides a method for allocating link bandwidth that is based on the 802.1p priority value in each packets VLAN tag. Data center Bridging Capability Exchange Protocol (DCBX) (IEEE 802.1AB) allows neighboring network devices to exchange information about their capabilities. Fibre Channel over Ethernet (FCoE): FC-BB5 FCoE specification compliant. Native FC Forwarder switch operations. End-to-end FCoE support (initiator to target). FCoE Initialization Protocol (FIP) support.
Fibre Channel: Omni Ports support 4/8 Gb FC when FC SFPs+ are installed in these ports. Full Fabric mode for end-to-end FCoE or NPV Gateway mode for external FC SAN attachments (support for IBM B-type, Brocade, and Cisco MDS external SANs). Fabric services in Full Fabric mode: Name Server Registered State Change Notification (RSCN) Login services Zoning
Chapter 4. Chassis and infrastructure configuration
127
Stacking: Hybrid stacking support (from two to six EN4093/EN4093R switches with two CN4093 switches) FCoE support vNIC support 802.1Qbg support Manageability Simple Network Management Protocol (SNMP V1, V2, and V3) HTTP browser GUI Telnet interface for CLI SSH Secure FTP (sFTP) Service Location Protocol (SLP) Serial interface for CLI Scriptable CLI Firmware image update (TFTP and FTP) Network Time Protocol (NTP) for switch clock synchronization
Monitoring Switch LEDs for external port status and switch module status indication. Remote Monitoring (RMON) agent to collect statistics and proactively monitor switch performance. Port mirroring for analyzing network traffic that passes through a switch. Change tracking and remote logging with syslog feature. Support for sFLOW agent for monitoring traffic in data networks (separate sFLOW analyzer is required elsewhere). POST diagnostic tests. The following features are not supported by IPv6: Default switch management IP address SNMP trap host destination IP address Bootstrap Protocol (BOOTP) and DHCP RADIUS, TACACS+, and LDAP QoS metering and re-marking ACLs for out-profile traffic VMware Virtual Center (vCenter) for VMready Routing Information Protocol (RIP) Internet Group Management Protocol (IGMP) Border Gateway Protocol (BGP) Virtual Router Redundancy Protocol (VRRP) sFLOW
Standards supported
The switches support the following standards: IEEE 802.1AB data center Bridging Capability Exchange Protocol (DCBX) IEEE 802.1D Spanning Tree Protocol (STP) IEEE 802.1p Class of Service (CoS) prioritization IEEE 802.1s Multiple STP (MSTP) IEEE 802.1Q Tagged VLAN (frame tagging on all ports when VLANs are enabled) IEEE 802.1Qbg Edge Virtual Bridging IEEE 802.1Qbb Priority-Based Flow Control (PFC) 128
IBM PureFlex System and IBM Flex System Products and Technology
IEEE 802.1Qaz Enhanced Transmission Selection (ETS) IEEE 802.1x port-based authentication IEEE 802.1w Rapid STP (RSTP) IEEE 802.2 Logical Link Control IEEE 802.3 10BASE-T Ethernet IEEE 802.3ab 1000BASE-T copper twisted pair Gigabit Ethernet IEEE 802.3ad Link Aggregation Control Protocol IEEE 802.3ae 10GBASE-SR short range fiber optics 10 Gb Ethernet IEEE 802.3ae 10GBASE-LR long range fiber optics 10 Gb Ethernet IEEE 802.3ba 40GBASE-SR4 short range fiber optics 40 Gb Ethernet IEEE 802.3ba 40GBASE-CR4 copper 40 Gb Ethernet IEEE 802.3u 100BASE-TX Fast Ethernet IEEE 802.3x Full-duplex Flow Control IEEE 802.3z 1000BASE-SX short range fiber optics Gigabit Ethernet IEEE 802.3z 1000BASE-LX long range fiber optics Gigabit Ethernet SFF-8431 10GSFP+Cu SFP+ Direct Attach Cable FC-BB-5 FCoE For more information, see the IBM Redbooks Product Guide IBM Flex System Fabric CN4093 10Gb Converged Scalable Switch, TIPS0910, which is available at this website: http://www.redbooks.ibm.com/abstracts/tips0910.html?Open
4.11.7 IBM Flex System Fabric EN4093 and EN4093R 10Gb Scalable Switch
The IBM Flex System EN4093 and IBM Flex System 4093R 10Gb Scalable Switches are 10 Gb 64-port upgradeable midrange to high-end switch modules. They offer Layer 2/3 switching designed for installation within the I/O module bays of the Enterprise Chassis. The latest EN4093R switch adds more capabilities to the EN4093, that is, Virtual NIC (Stacking), Unified fabric port (Stacking), Edge virtual bridging (Stacking), and CEE/FCoE (Stacking), and so it is ideal for clients that are looking to implement a converged infrastructure with NAS, iSCSI, or FCoE. For FCoE implementations, the EN4093R acts as a transit switch that forwards FCoE traffic upstream to another devices, such as the Brocade VDX or Cisco Nexus 5548/5596, where the FC traffic is broken out. For a detailed function comparison, see Table 4-27 on page 135. Each switch contains the following ports: Up to 42 internal 10 Gb ports Up to 14 external 10 Gb uplink ports (enhanced small form-factor pluggable (SFP+) connectors) Up to 2 external 40 Gb uplink ports (quad small form-factor pluggable (QSFP+) connectors) These switches are considered suitable for clients with the following requirements: Building a 10 Gb infrastructure Implementing a virtualized environment Requiring investment protection for 40 Gb uplinks Wanting to reduce total cost of ownership (TCO) and improve performance while maintaining high levels of availability and security Wanting to avoid oversubscription (traffic from multiple internal ports that attempt to pass through a lower quantity of external ports, leading to congestion and performance impact)
Chapter 4. Chassis and infrastructure configuration
129
The EN4093 and 4093R 10Gb Scalable Switches are shown in Figure 4-48.
As listed in Table 4-25, the switch is initially licensed with 14 10-Gb internal ports that are enabled and 10 10-Gb external uplink ports enabled. Further ports can be enabled, including the two 40 Gb external uplink ports with the Upgrade 1 and four more SFP+ 10Gb ports with Upgrade 2 license options. Upgrade 1 must be applied before Upgrade 2 can be applied.
Table 4-25 IBM Flex System Fabric EN4093 10Gb Scalable Switch part numbers and port upgrades Part number 49Y4270 Feature codea A0TB / 3593 Product description Total ports that are enabled Internal IBM Flex System Fabric EN4093 10Gb Scalable Switch: 10x external 10 Gb uplinks 14x internal 10 Gb ports IBM Flex System Fabric EN4093R 10Gb Scalable Switch: 10x external 10 Gb uplinks 14x internal 10 Gb ports IBM Flex System Fabric EN4093 10Gb Scalable Switch (Upgrade 1): Adds 2x external 40 Gb uplinks Adds 14x internal 10 Gb ports IBM Flex System Fabric EN4093 10Gb Scalable Switch (Upgrade 2) (requires Upgrade 1): Adds 4x external 10 Gb uplinks Add 14x internal 10 Gb ports 14 10 Gb uplink 10 40 Gb uplink 0
05Y3309
A3J6 / ESW7
14
10
49Y4798
A1EL / 3596
28
10
88Y6037
A1EM / 3597
42
14
a. The first feature code listed is for configurations ordered through System x sales channels (HVEC) using x-config. The second feature code is for configurations ordered through the IBM Power Systems channel (AAS) using e-config.
130
IBM PureFlex System and IBM Flex System Products and Technology
The key components on the front of the switch are shown in Figure 4-49.
14x 10 Gb uplink ports (10 standard, 4 with Upgrade 2) 2x 40 Gb uplink ports (enabled with Upgrade 1)
SFP+ ports
QSFP+ ports
Management ports
Switch LEDs
Each upgrade license enables more internal ports. To make full use of those ports, each compute node needs the following appropriate I/O adapter installed: The base switch requires a two-port Ethernet adapter (one port of the adapter goes to each of two switches) Upgrade 1 requires a four-port Ethernet adapter (two ports of the adapter to each switch) Upgrade 2 requires a six-port Ethernet adapter (three ports to each switch) Consideration: Adding Upgrade 2 enables another 14 internal ports, for a total of 42 internal ports, with three ports that are connected to each of the 14 compute nodes in the chassis. To make full use of all 42 internal ports, a six-port adapter is required, such as the CN4058 Adapter. Upgrade 2 still provides a benefit with a four-port adapter because this upgrade enables an extra four external 10 Gb uplink as well. The rear of the switch has 14 SPF+ module ports and two QSFP+ module ports. The QSFP+ ports can be used to provide two 40 Gb uplinks or eight 10 Gb ports. Use one of the supported QSFP+ to 4x 10 Gb SFP+ cables that are listed in Table 4-26. This cable splits a single 40 Gb QSPFP port into 4 SFP+ 10 Gb ports. The switch is designed to function with nodes that contain a 1Gb LOM, such as the IBM Flex System x220 Compute Node. To manage the switch, a mini USB port and an Ethernet management port are provided. The supported SFP+ and QSFP+ modules and cables for the switch are listed in Table 4-26.
Table 4-26 Supported SFP+ modules and cables Part number Feature codea Description
Serial console cables 90Y9338 A2RR / A2RR IBM Flex System Management Serial Access Cable Kit
Small form-factor pluggable (SFP) transceivers - 1 GbE 81Y1618 81Y1622 90Y9424 3268 / EB29 3269 / EB2A A1PN / ECB8 IBM SFP RJ-45 Transceiver (does not support 10/100 Mbps) IBM SFP SX Transceiver IBM SFP LX Transceiver
131
Part number
Feature codea
Description
SFP+ transceivers - 10 GbE 46C3447 90Y9412 44W4408 5053 / None A1PM / ECB9 4942 / 3382 IBM SFP+ SR Transceiver IBM SFP+ LR Transceiver 10GBase-SR SFP+ (MMFiber) transceiver
SFP+ Direct Attach Copper (DAC) cables - 10 GbE 90Y9427 90Y9430 90Y9433 A1PH / ECB4 A1PJ / ECB5 A1PK / ECB6 1m IBM Passive DAC SFP+ 3m IBM Passive DAC SFP+ 5m IBM Passive DAC SFP+
QSFP+ transceiver and cables - 40 GbE 49Y7884 90Y3519 90Y3521 A1DR / EB27 A1MM / EB2J A1MN / EB2K IBM QSFP+ 40GBASE-SR Transceiver (Requires either cable 90Y3519 or cable 90Y3521) 10m IBM MTP Fiberoptic Cable (requires transceiver 49Y7884) 30m IBM MTP Fiberoptic Cable (requires transceiver 49Y7884)
QSFP+ breakout cables - 40 GbE to 4x10 GbE 49Y7886 49Y7887 49Y7888 A1DL / EB24 A1DM / EB25 A1DN / EB26 1m 40 Gb QSFP+ to 4 x 10 Gb SFP+ Cable 3m 40 Gb QSFP+ to 4 x 10 Gb SFP+ Cable 5m 40 Gb QSFP+ to 4 x 10 Gb SFP+ Cable
QSFP+ Direct Attach Copper (DAC) cables - 40 GbE 49Y7890 49Y7891 A1DP / EB2B A1DQ / EB2H 1m QSFP+ to QSFP+ DAC 3m QSFP+ to QSFP+ DAC
a. The first feature code listed is for configurations ordered through System x sales channels (HVEC) using x-config. The second feature code is for configurations ordered through the IBM Power Systems channel (AAS) using e-config.
The EN4093/4093R 10Gb Scalable Switch has the following features and specifications: Internal ports: A total of 42 internal full-duplex 10 Gigabit ports (14 ports are enabled by default). Optional FoD licenses are required to activate the remaining 28 ports. Two internal full-duplex 1 GbE ports that are connected to the chassis management module. External ports: A total of 14 ports for 1 Gb or 10 Gb Ethernet SFP+ transceivers (support for 1000BASE-SX, 1000BASE-LX, 1000BASE-T, 10GBASE-SR, or 10GBASE-LR) or SFP+ DAC cables. A total of 10 ports are enabled by default. An optional FoD license is required to activate the remaining four ports. SFP+ modules and DAC cables are not included and must be purchased separately. Two ports for 40 Gb Ethernet QSFP+ transceivers or QSFP+ DACs (ports are disabled by default; an optional FoD license is required to activate them). QSFP+ modules and DAC cables are not included and must be purchased separately.
132
IBM PureFlex System and IBM Flex System Products and Technology
One RS-232 serial port (mini-USB connector) that provides another means to configure the switch module. Scalability and performance: 40 Gb Ethernet ports for extreme uplink bandwidth and performance. Fixed-speed external 10 Gb Ethernet ports to take advantage of 10 Gb core infrastructure. Autosensing 10/1000/1000 external Gigabit Ethernet ports for bandwidth optimization. Non-blocking architecture with wire-speed forwarding of traffic and aggregated throughput of 1.28 Tbps. Media Access Control (MAC) address learning: Automatic update, support of up to 128,000 MAC addresses. Up to 128 IP interfaces per switch. Static and Link Aggregation Control Protocol (LACP) (IEEE 802.3ad) link aggregation: Up to 220 Gb of total uplink bandwidth per switch, up to 64 trunk groups, up to 16 ports per group. Support for jumbo frames (up to 9,216 bytes). Broadcast/multicast storm control. Internet Group Management Protocol (IGMP) snooping to limit flooding of IP multicast traffic. IGMP filtering to control multicast traffic for hosts that participate in multicast groups. Configurable traffic distribution schemes over trunk links that are based on source/destination IP or MAC addresses, or both. Fast port forwarding and fast uplink convergence for rapid STP convergence. Availability and redundancy: Virtual Router Redundancy Protocol (VRRP) for Layer 3 router redundancy. IEEE 802.1D Spanning Tree Protocol (STP) for providing L2 redundancy. IEEE 802.1s Multiple STP (MSTP) for topology optimization, up to 32 STP instances are supported by single switch. IEEE 802.1w Rapid STP (RSTP) provides rapid STP convergence for critical delay-sensitive traffic like voice or video. Rapid Per-VLAN STP (RPVST) enhancements. Layer 2 Trunk Failover to support active/standby configurations of network adapter that team on compute nodes. Hot Links provides basic link redundancy with fast recovery for network topologies that require Spanning Tree to be turned off. Virtual local area network (VLAN) support: Up to 1024 VLANs supported per switch, with VLAN numbers that range from 1 to 4095 (4095 is used for the management modules connection only). 802.1Q VLAN tagging support on all ports. Private VLANs. Security: VLAN-based, MAC-based, and IP-based access control lists (ACLs) 802.1x port-based authentication Multiple user IDs and passwords
Chapter 4. Chassis and infrastructure configuration
133
User access control Radius, TACACS+ and LDAP authentication and authorization Quality of service (QoS): Support for IEEE 802.1p, IP ToS/DSCP, and ACL-based (MAC/IP source and destination addresses, VLANs) traffic classification and processing. Traffic shaping and remarking based on defined policies. Eight weighted round robin (WRR) priority queues per port for processing qualified traffic. IP v4 Layer 3 functions: Host management IP forwarding IP filtering with ACLs, up to 896 ACLs supported VRRP for router redundancy Support for up to 128 static routes Routing protocol support (RIP v1, RIP v2, OSPF v2, BGP-4), up to 2048 entries in a routing table Support for Dynamic Host Configuration Protocol (DHCP) Relay Support for IGMP snooping and IGMP relay Support for Protocol Independent Multicast (PIM) in Sparse Mode (PIM-SM) and Dense Mode (PIM-DM) 802.1Qbg support IP v6 Layer 3 functions: IPv6 host management (except default switch management IP address) IPv6 forwarding Up to 128 static routes Support for OSPF v3 routing protocol IPv6 filtering with ACLs
Virtualization: Virtual Fabric with virtual network interface card (vNIC) 802.1Qbg Edge Virtual Bridging (EVB) IBM VMready Converged Enhanced Ethernet: Priority-based Flow Control (PFC) (IEEE 802.1Qbb) extends 802.3x standard flow control to allow the switch to pause traffic. This function is based on the 802.1p priority value in each packets VLAN tag. Enhanced Transmission Selection (ETS) (IEEE 802.1Qaz) provides a method for allocating link bandwidth that is based on the 802.1p priority value in each packets VLAN tag. Data center Bridging Capability Exchange Protocol (DCBX) (IEEE 802.1AB) allows neighboring network devices to exchange information about their capabilities. Manageability: Simple Network Management Protocol (SNMP V1, V2, and V3) HTTP browser GUI Telnet interface for CLI 134
IBM PureFlex System and IBM Flex System Products and Technology
Secure Shell (SSH) Serial interface for CLI Scriptable CLI Firmware image update: Trivial File Transfer Protocol (TFTP) and File Transfer Protocol (FTP) Network Time Protocol (NTP) for switch clock synchronization Monitoring: Switch LEDs for external port status and switch module status indication. Remote monitoring (RMON) agent to collect statistics and proactively monitor switch performance. Port mirroring for analyzing network traffic that passes through the switch. Change tracking and remote logging with syslog feature. Support for sFLOW agent for monitoring traffic in data networks (separate sFLOW analyzer is required elsewhere). POST diagnostic procedures. Stacking: Up to eight switches in a stack FCoE support (EN4093R only) vNIC support (support for FCoE on vNICs) Table 4-27 compares the EN4093 to the EN4093R.
Table 4-27 EN4093 and EN4093R supported features Feature Layer 2 switching Layer 3 switching Switch Stacking Virtual NIC (stand-alone) Virtual NIC (stacking) Unified Fabric Port (stand-alone) Unified Fabric Port (stacking) Edge virtual bridging (stand-alone) Edge virtual bridging (stacking) CEE/FCoE (stand-alone) CEE/FCoE (stacking) EN4093 Yes Yes Yes Yes Yes Yes No Yes Yes Yes No EN4093R Yes Yes Yes Yes Yes Yes No Yes Yes Yes Yes
Both the EN4093 and EN4093R support vNIC+ FCoE and 802.1Qbg + FCoE stand-alone (without stacking). The EN4093R supports vNIC + FCOE with stacking or 802.1Qbg + FCoE with stacking. For more information, see the IBM Redbooks Product Guide IBM Flex System Fabric EN4093 and EN4093R 10Gb Scalable Switches, TIPS0864, which is available at this website: http://www.redbooks.ibm.com/abstracts/tips0864.html?Open
Chapter 4. Chassis and infrastructure configuration
135
Figure 4-50 IBM Flex System Fabric SI4093 System Interconnect Module
The SI4093 System Interconnect Module is initially licensed for 14 10-Gb internal ports enabled and 10 10-Gb external uplink ports enabled. Further ports can be enabled, including 14 internal ports and two 40 Gb external uplink ports with Upgrade 1, and 14 internal ports and four SFP+ 10 Gb external ports with Upgrade 2 license options. Upgrade 1 must be applied before Upgrade 2 can be applied. The key components on the front of the switch are shown in Figure 4-49 on page 131.
14x 10 Gb uplink ports (10 standard, 4 with Upgrade 2) 2x 40 Gb uplink ports (enabled with Upgrade 1)
SFP+ ports
QSFP+ ports
Management ports
Switch LEDs
Figure 4-51 IBMIBM Flex System Fabric SI4093 System Interconnect Module
136
IBM PureFlex System and IBM Flex System Products and Technology
Table 4-28 shows the part numbers for ordering the switches and the upgrades.
Table 4-28 Ordering information Description Interconnect module IBM Flex System Fabric SI4093 System Interconnect Module Features on Demand upgrades SI4093 System Interconnect Module (Upgrade 1) SI4093 System Interconnect Module (Upgrade 2) 95Y3318 95Y3320 A45U / ESW8 A45V / ESW9 95Y3313 A45T / ESWA Part number Feature code (x-config / e-config)
Important: SFP and SFP+ (small form-factor pluggable plus) transceivers or cables are not included with the switch. They must be ordered separately. For more information, see Table 4-29 on page 138. The following base switch and upgrades are available: Part number 95Y3313 is for the physical device, and it includes 14 other internal 10-Gb ports enabled (one to each node bay) and 10 external 10-Gb ports enabled for connectivity to an upstream network, plus external servers and storage. All external 10 Gb ports are SFP+ based connections. Part number 95Y3318 (Upgrade 1) can be applied on the base interconnect module to make full use of four-port adapters that are installed in each compute node. This upgrade enables 14 other internal ports, for a total of 28 ports. The upgrade also enables two 40 Gb uplinks with QSFP+ connectors. These QSFP+ ports can also be converted to four 10 Gb SFP+ DAC connections by using the appropriate fan-out cable. This upgrade requires the base interconnect module. Part number 95Y3320 (Upgrade 2) can be applied on top of Upgrade 1 when you want more uplink bandwidth on the interconnect module or if you want more internal bandwidth to the compute nodes with the adapters capable of supporting six ports (such as CN4058). The upgrade enables the remaining four external 10 Gb uplinks with SFP+ connectors, plus 14 other internal 10 Gb ports, for a total of 42 ports (three to each compute node).
137
Table 4-29 lists the supported port combinations on the interconnect module and the required upgrades.
Table 4-29 Supported port combinations Quantity required Supported port combinations 14x internal 10 GbE 10x external 10 GbE 28x internal 10 GbE 10x external 10 GbE 2x external 40 GbE 42x internal 10 GbEa 14x external 10 GbE 2x external 40 GbE Base switch, 95Y3313 1 1 Upgrade 1, 95Y3318 0 1 Upgrade 2, 95Y3320 0 0
a. This configuration uses six of the eight ports on the CN4058 adapter that are available for IBM Power Systems compute nodes.
138
IBM PureFlex System and IBM Flex System Products and Technology
Description QSFP+ transceiver and cables - 40 GbE IBM QSFP+ 40GBASE-SR Transceiver (Requires either cable 90Y3519 or cable 90Y3521) 10m IBM MTP Fiber Optical Cable (requires transceiver 49Y7884) 30m IBM MTP Fiber Optical Cable (requires transceiver 49Y7884) QSFP+ breakout cables - 40 GbE to 4x10 GbE 1m 40Gb QSFP+ to 4 x 10Gb SFP+ Cable 3m 40Gb QSFP+ to 4 x 10Gb SFP+ Cable 5m 40Gb QSFP+ to 4 x 10Gb SFP+ Cable QSFP+ direct-attach cables - 40 GbE 1m QSFP+ to QSFP+ DAC 3m QSFP+ to QSFP+ DAC
Part number
49Y7890 49Y7891
With the flexibility of the interconnect module, you can make full use of the technologies that are required for the following environments: For 1 GbE links, you can use SFP transceivers plus RJ-45 cables or LC-to-LC fiber cables, depending on the transceiver. For 10 GbE, you can use direct-attached cables (DAC, also known as Twinax), which come in lengths 1 - 5 m. These DACs are a cost-effective and low-power alternative to transceivers, and are ideal for all 10 Gb Ethernet connectivity within the rack, or even connecting to an adjacent rack. For longer distances, there is a choice of SFP+ transceivers (SR or LR) plus LC-to-LC fiber optic cables. For 40 Gb links, you can use QSFP+ to QSFP+ cables up to 3 m, or QSFP+ transceivers and MTP cables for longer distances. You also can break out the 40 Gb ports into four 10 GbE SFP+ DAC connections by using break-out cables.
139
Internal ports: A total of 42 internal full-duplex 10 Gigabit ports (14 ports are enabled by default; optional FoD licenses are required to activate the remaining 28 ports). Two internal full-duplex 1 GbE ports are connected to the chassis management module. External ports: A total of 14 ports for 1 Gb or 10 Gb Ethernet SFP+ transceivers (support for 1000BASE-SX, 1000BASE-LX, 1000BASE-T, 10GBASE-SR, or 10GBASE-LR) or SFP+ copper direct-attach cables (DAC). A total of 10 ports are enabled by default. An optional FoD license is required to activate the remaining four ports. SFP+ modules and DACs are not included and must be purchased separately. Two ports for 40 Gb Ethernet QSFP+ transceivers or QSFP+ DACs. (Ports are disabled by default. An optional FoD license is required to activate them.) QSFP+ modules and DACs are not included and must be purchased separately. One RS-232 serial port (mini-USB connector) that provides an additional means to configure the switch module. Scalability and performance: 40 Gb Ethernet ports for extreme uplink bandwidth and performance. External 10 Gb Ethernet ports to use 10 Gb upstream infrastructure. Non-blocking architecture with wire-speed forwarding of traffic and aggregated throughput of 1.28 Tbps. Media access control (MAC) address learning: automatic update, support for up to 128,000 MAC addresses. Static and LACP (IEEE 802.3ad) link aggregation, up to 220 Gb of total uplink bandwidth per interconnect module. Support for jumbo frames (up to 9,216 bytes). Availability and redundancy: Layer 2 Trunk Failover to support active/standby configurations of network adapter teaming on compute nodes. Built in link redundancy with loop prevention without a need for Spanning Tree protocol. VLAN support Up to 32 VLANs supported per interconnect module SPAR partition, with VLAN numbers 1 - 4095 (4095 is used for management modules connection only). 802.1Q VLAN tagging support on all ports. Security: VLAN-based access control lists (ACLs) (VLAN-aware mode). Multiple user IDs and passwords. User access control. Radius, TACACS+, and LDAP authentication and authorization.
Quality of service (QoS) Support for IEEE 802.1p traffic classification and processing. Virtualization Switch Independent Virtual NIC (vNIC2) Ethernet, iSCSI, or FCoE traffic is supported on vNICs
140
IBM PureFlex System and IBM Flex System Products and Technology
Switch partitioning (SPAR): SPAR forms separate virtual switching contexts by segmenting the data plane of the switch. Data plane traffic is not shared between SPARs on the same switch. SPAR operates as a Layer 2 broadcast network. Hosts on the same VLAN attached to a SPAR can communicate with each other and with the upstream switch. Hosts on the same VLAN but attached to different SPARs communicate through the upstream switch. SPAR is implemented as a dedicated VLAN with a set of internal server ports and a single uplink port or link aggregation (LAG). Multiple uplink ports or LAGs are not allowed in SPAR. A port can be a member of only one SPAR.
Converged Enhanced Ethernet: Priority-Based Flow Control (PFC) (IEEE 802.1Qbb) extends 802.3x standard flow control to allow the switch to pause traffic based on the 802.1p priority value in each packets VLAN tag. Enhanced Transmission Selection (ETS) (IEEE 802.1Qaz) provides a method for allocating link bandwidth based on the 802.1p priority value in each packets VLAN tag. Data Center Bridging Capability Exchange Protocol (DCBX) (IEEE 802.1AB) allows neighboring network devices to exchange information about their capabilities. Fibre Channel over Ethernet (FCoE): FC-BB5 FCoE specification compliant. FCoE transit switch operations. FCoE Initialization Protocol (FIP) support. Manageability: IPv4 and IPv6 host management. Simple Network Management Protocol (SNMP V1, V2, and V3). Industry standard command-line interface (IS-CLI) through Telnet, SSH, and serial port. Secure FTP (sFTP). Service Location Protocol (SLP). Firmware image update (TFTP and FTP/sFTP). Network Time Protocol (NTP) for clock synchronization. IBM System Networking Switch Center (SNSC) support. Monitoring: Switch LEDs for external port status and switch module status indication. Change tracking and remote logging with syslog feature. POST diagnostic tests.
Supported standards
The switches support the following standards: IEEE 802.1AB Data Center Bridging Capability Exchange Protocol (DCBX) IEEE 802.1p Class of Service (CoS) prioritization IEEE 802.1Q Tagged VLAN (frame tagging on all ports when VLANs are enabled) IEEE 802.1Qbb Priority-Based Flow Control (PFC) IEEE 802.1Qaz Enhanced Transmission Selection (ETS) IEEE 802.3 10BASE-T Ethernet IEEE 802.3ab 1000BASE-T copper twisted pair Gigabit Ethernet
Chapter 4. Chassis and infrastructure configuration
141
IEEE 802.3ad Link Aggregation Control Protocol IEEE 802.3ae 10GBASE-SR short range fiber optics 10 Gb Ethernet IEEE 802.3ae 10GBASE-LR long range fiber optics 10 Gb Ethernet IEEE 802.3ba 40GBASE-SR4 short range fiber optics 40 Gb Ethernet IEEE 802.3ba 40GBASE-CR4 copper 40 Gb Ethernet IEEE 802.3u 100BASE-TX Fast Ethernet IEEE 802.3x Full-duplex Flow Control IEEE 802.3z 1000BASE-SX short range fiber optics Gigabit Ethernet IEEE 802.3z 1000BASE-LX long range fiber optics Gigabit Ethernet SFF-8431 10GSFP+Cu SFP+ Direct Attach Cable For more information, see the IBM Redbooks Product Guide IBM Flex System Fabric SI4093 System Interconnect Module, which is available at this website: http://www.redbooks.ibm.com/abstracts/tips0864.html?Open
Figure 4-52 IBM Flex System EN4091 10Gb Ethernet Pass-thru Module
The ordering part number and feature codes are listed in Table 4-31.
Table 4-31 EN4091 10Gb Ethernet Pass-thru Module part number and feature codes Part number 88Y6043 Feature codea A1QV / 3700 Product Name IBM Flex System EN4091 10Gb Ethernet Pass-thru
a. The first feature code listed is for configurations ordered through System x sales channels (HVEC) using x-config. The second feature code is for configurations ordered through the IBM Power Systems channel (AAS) using e-config.
142
IBM PureFlex System and IBM Flex System Products and Technology
The EN4091 10Gb Ethernet Pass-thru Module includes the following specifications: Internal ports 14 internal full-duplex Ethernet ports that can operate at 1 Gb or 10 Gb speeds External ports Fourteen ports for 1 Gb or 10 Gb Ethernet SFP+ transceivers (support for 1000BASE-SX, 1000BASE-LX, 1000BASE-T, 10GBASE-SR, or 10GBASE-LR) or SFP+ DAC. SFP+ modules and DAC cables are not included, and must be purchased separately. Unmanaged device that has no internal Ethernet management port. However, it can provide its VPD to the secure management network in the CMM. Supports 10 Gb Ethernet signaling for CEE, FCoE, and other Ethernet-based transport protocols. Allows direct connection from the 10 Gb Ethernet adapters that are installed in compute nodes in a chassis to an externally located Top of Rack switch or other external device. Consideration: The EN4091 10Gb Ethernet Pass-thru Module has only 14 internal ports. As a result, only two ports on each compute node are enabled, one for each of two pass-through modules that are installed in the chassis. If four-port adapters are installed in the compute nodes, ports 3 and 4 on those adapters are not enabled. There are three standard I/O module status LEDs, as shown in Figure 4-42 on page 112. Each port has link and activity LEDs. Table 4-32 lists the supported transceivers and DAC cables.
Table 4-32 IBM Flex System EN4091 10Gb Ethernet Pass-thru Module Part number Feature codesa Description
SFP+ transceivers - 10 GbE 44W4408 46C3447 90Y9412 4942 / 3282 5053 / None A1PM / None 10 GbE 850 nm Fibre Channel SFP+ Transceiver (SR) IBM SFP+ SR Transceiver IBM SFP+ LR Transceiver
SFP transceivers - 1 GbE 81Y1622 81Y1618 90Y9424 3269 / EB2A 3268 / EB29 A1PN / None IBM SFP SX Transceiver IBM SFP RJ45 Transceiver IBM SFP LX Transceiver
Direct-attach copper (DAC) cables 81Y8295 81Y8296 81Y8297 95Y0323 95Y0326 95Y0329 A18M / EN01 A18N / EN02 A18P / EN03 A25A / None A25B / None A25C / None 1m 10GE Twinax Act Copper SFP+ DAC (active) 3m 10GE Twinax Act Copper SFP+ DAC (active) 5m 10GE Twinax Act Copper SFP+ DAC (active) 1m IBM Active DAC SFP+ Cable 3m IBM Active DAC SFP+ Cable 5m IBM Active DAC SFP+ Cable
143
a. The first feature code listed is for configurations ordered through System x sales channels (HVEC) using x-config. The second feature code is for configurations ordered through the IBM Power Systems channel (AAS) using e-config.
For more information, see the IBM Redbooks Product Guide IBM Flex System EN4091 10Gb Ethernet Pass-thru Module, TIPS0865, which is available at this website: http://www.redbooks.ibm.com/abstracts/tips0865.html?Open
Figure 4-53 IBM Flex System EN2092 1Gb Ethernet Scalable Switch
As listed in Table 4-33, the switch comes standard with 14 internal and 10 external Gigabit Ethernet ports enabled. Further ports can be enabled, including the four external 10 Gb uplink ports. Upgrade 1 and the 10 Gb Uplinks upgrade can be applied in either order.
Table 4-33 IBM Flex System EN2092 1Gb Ethernet Scalable Switch part numbers and port upgrades Part number 49Y4294 Feature codea A0TF / 3598 Product description IBM Flex System EN2092 1Gb Ethernet Scalable Switch: 14 internal 1 Gb ports 10 external 1 Gb ports IBM Flex System EN2092 1Gb Ethernet Scalable Switch (Upgrade 1): Adds 14 internal 1 Gb ports Adds 10 external 1 Gb ports IBM Flex System EN2092 1Gb Ethernet Scalable Switch (10 Gb Uplinks), which adds 4 external 10 Gb uplinks
90Y3562
A1QW / 3594
49Y4298
A1EN / 3599
a. The first feature code listed is for configurations ordered through System x sales channels (HVEC) using x-config. The second feature code is for configurations ordered through the IBM Power Systems channel (AAS) using e-config.
144
IBM PureFlex System and IBM Flex System Products and Technology
The key components on the front of the switch are shown in Figure 4-54.
20x external 1 Gb ports (10 standard, 10 with Upgrade 1) 4x 10 Gb uplink ports (enabled with Uplinks upgrade)
RJ45 ports
Figure 4-54 IBM Flex System EN2092 1Gb Ethernet Scalable Switch
The standard switch has 14 internal ports, and the Upgrade 1 license enables 14 more internal ports. To make full use of those ports, each compute node needs the following appropriate I/O adapter installed: The base switch requires a two-port Ethernet adapter that is installed in each compute node (one port of the adapter goes to each of two switches). Upgrade 1 requires a four-port Ethernet adapter that is installed in each compute node (two ports of the adapter to each switch). The standard has 10 external ports enabled. More external ports are enabled with the following license upgrades: Upgrade 1 enables 10 more ports for a total of 20 ports Uplinks Upgrade enables the four 10 Gb SFP+ ports. These upgrades can be installed in either order. This switch is considered ideal for clients with the following characteristics: Still use 1 Gb as their networking infrastructure. Are deploying virtualization and require multiple 1 Gb ports. Want investment protection for 10 Gb uplinks. Looking to reduce TCO and improve performance, while maintaining high levels of availability and security. Looking to avoid oversubscription (multiple internal ports that attempt to pass through a lower quantity of external ports, leading to congestion and performance impact). The switch has three switch status LEDs (see Figure 4-42 on page 112) and one mini-USB serial port connector for console management. Uplink Ports 1 - 20 are RJ45, and the 4 x 10 Gb uplink ports are SFP+. The switch supports either SFP+ modules or DAC cables. The supported SFP+ modules and DAC cables for the switch are listed in Table 4-34.
Table 4-34 IBM Flex System EN2092 1Gb Ethernet Scalable Switch SFP+ and DAC cables Part number SFP transceivers 81Y1622 3269 / EB2A IBM SFP SX Transceiver Feature codea Description
145
Part number 81Y1618 90Y9424 SFP+ transceivers 44W4408 46C3447 90Y9412 DAC cables 90Y9427 90Y9430 90Y9433
10 GbE 850 nm Fibre Channel SFP+ Transceiver (SR) IBM SFP+ SR Transceiver IBM SFP+ LR Transceiver
1m IBM Passive DAC SFP+ 3m IBM Passive DAC SFP+ 5m IBM Passive DAC SFP+
a. The first feature code listed is for configurations ordered through System x sales channels (HVEC) using x-config. The second feature code is for configurations ordered through the IBM Power Systems channel (AAS) using e-config.
The EN2092 1 Gb Ethernet Scalable Switch includes the following features and specifications: Internal ports: A total of 28 internal full-duplex Gigabit ports; 14 ports are enabled by default. An optional FoD license is required to activate another 14 ports. Two internal full-duplex 1 GbE ports that are connected to the chassis management module. External ports: Four ports for 1 Gb or 10 Gb Ethernet SFP+ transceivers (support for 1000BASE-SX, 1000BASE-LX, 1000BASE-T, 10GBASE-SR, or 10GBASE-LR) or SFP+ DAC. These ports are disabled by default. An optional FoD license is required to activate them. SFP+ modules are not included and must be purchased separately. A total of 20 external 10/100/1000 1000BASE-T Gigabit Ethernet ports with RJ-45 connectors; 10 ports are enabled by default. An optional FoD license is required to activate another 10 ports. One RS-232 serial port (mini-USB connector) that provides another means to configure the switch module. Scalability and performance: Fixed-speed external 10 Gb Ethernet ports for maximum uplink bandwidth Autosensing 10/1000/1000 external Gigabit Ethernet ports for bandwidth optimization Non-blocking architecture with wire-speed forwarding of traffic MAC address learning: Automatic update, support of up to 32,000 MAC addresses Up to 128 IP interfaces per switch Static and LACP (IEEE 802.3ad) link aggregation, up to 60 Gb of total uplink bandwidth per switch, up to 64 trunk groups, up to 16 ports per group Support for jumbo frames (up to 9,216 bytes)
146
IBM PureFlex System and IBM Flex System Products and Technology
Broadcast/multicast storm control IGMP snooping for limit flooding of IP multicast traffic IGMP filtering to control multicast traffic for hosts that participate in multicast groups Configurable traffic distribution schemes over trunk links that are based on source/destination IP or MAC addresses, or both Fast port forwarding and fast uplink convergence for rapid STP convergence Availability and redundancy: VRRP for Layer 3 router redundancy IEEE 802.1D STP for providing L2 redundancy IEEE 802.1s MSTP for topology optimization, up to 32 STP instances that are supported by a single switch IEEE 802.1w RSTP (provides rapid STP convergence for critical delay-sensitive traffic like voice or video) RPVST enhancements Layer 2 Trunk Failover to support active/standby configurations of network adapter teaming on compute nodes Hot Links provides basic link redundancy with fast recovery for network topologies that require Spanning Tree to be turned off VLAN support: Up to 1024 VLANs supported per switch, with VLAN numbers that range from 1 to 4095 (4095 is used for the management modules connection only) 802.1Q VLAN tagging support on all ports Private VLANs Security: VLAN-based, MAC-based, and IP-based ACLs 802.1x port-based authentication Multiple user IDs and passwords User access control Radius, TACACS+, and Lightweight Directory Access Protocol (LDAP) authentication and authorization QoS: Support for IEEE 802.1p, IP ToS/DSCP, and ACL-based (MAC/IP source and destination addresses, VLANs) traffic classification and processing Traffic shaping and remarking based on defined policies Eight WRR priority queues per port for processing qualified traffic IP v4 Layer 3 functions: Host management IP forwarding IP filtering with ACLs, up to 896 ACLs supported VRRP for router redundancy Support for up to 128 static routes
147
Routing protocol support (RIP v1, RIP v2, OSPF v2, BGP-4), up to 2048 entries in a routing table Support for DHCP Relay Support for IGMP snooping and IGMP relay Support for Protocol Independent Multicast (PIM) in Sparse Mode (PIM-SM) and Dense Mode (PIM-DM). IP v6 Layer 3 functions: IPv6 host management (except default switch management IP address) IPv6 forwarding Up to 128 static routes Support for OSPF v3 routing protocol IPv6 filtering with ACLs
Virtualization: VMready Manageability: Simple Network Management Protocol (SNMP V1, V2, and V3) HTTP browser GUI Telnet interface for CLI SSH Serial interface for CLI Scriptable CLI Firmware image update (TFTP and FTP) NTP for switch clock synchronization
Monitoring: Switch LEDs for external port status and switch module status indication RMON agent to collect statistics and proactively monitor switch performance Port mirroring for analyzing network traffic that passes through the switch Change tracking and remote logging with the syslog feature Support for the sFLOW agent for monitoring traffic in data networks (separate sFLOW analyzer is required elsewhere) POST diagnostic functions For more information, see the IBM Redbooks Product Guide IBM Flex System EN2092 1Gb Ethernet Scalable Switch, TIPS0861, which is available at this website: http://www.redbooks.ibm.com/abstracts/tips0861.html?Open
148
IBM PureFlex System and IBM Flex System Products and Technology
Figure 4-55 shows the IBM Flex System FC5022 16Gb SAN Scalable Switch.
Figure 4-55 IBM Flex System FC5022 16Gb SAN Scalable Switch
Three versions are available, as listed in Table 4-35: 12-port and 24-port switch modules and a 24-port switch with the Enterprise Switch Bundle (ESB) software. The port count can be applied to internal or external ports by using a feature that is called Dynamic Ports on Demand (DPOD). Ports counts can be increased with license upgrades, as described in Port and feature upgrades on page 150.
Table 4-35 IBM Flex System FC5022 16Gb SAN Scalable Switch part numbers Part number 88Y6374 00Y3324 90Y9356 Feature codesa A1EH / 3770 A3DP / ESW5 A1EJ / 3771 Description IBM Flex System FC5022 16Gb SAN Scalable Switch IBM Flex System FC5022 24-port 16Gb SAN Scalable Switch IBM Flex System FC5022 24-port 16Gb ESB SAN Scalable Switch Ports enabled by default 12 24 24
a. The first feature code listed is for configurations ordered through System x sales channels (HVEC) using x-config. The second feature code is for configurations ordered through the IBM Power Systems channel (AAS) using e-config.
Table 4-36 provides a feature comparison between the FC5022 switch models.
Table 4-36 Feature comparison by model Feature FC5022 16Gb 24-port ESB Switch 90Y9356 Number of active ports Number of SFP+ included Full fabric Access Gateway Advanced zoning Enhanced Group Management ISL Trunking Adaptive Networking Advanced Performance Monitoring Fabric Watch 24 None Included Included Included Included Included Included Included Included FC5022 24-port 16Gb SAN Scalable Switch 00Y3324 24 2x 16 Gb SFP+ Included Included Included Included Optional Not available Not available Optional FC5022 16Gb SAN Scalable Switch 88Y6374 12 None Included Included Included Included Not available Not available Not available Not available
149
Feature
FC5022 24-port 16Gb SAN Scalable Switch 00Y3324 Not available Not available
FC5022 16Gb SAN Scalable Switch 88Y6374 Not available Not available
Included Included
The part number for the switch includes the following items: One IBM Flex System FC5022 16Gb SAN Scalable Switch or IBM Flex System FC5022 24-port 16Gb ESB SAN Scalable Switch Important Notices Flyer Warranty Flyer Documentation CD-ROM The switch does not include a serial management cable. However, IBM Flex System Management Serial Access Cable 90Y9338 is supported and contains two cables: a mini-USB-to-RJ45 serial cable and a mini-USB-to-DB9 serial cable. Either cable can be used to connect to the switch locally for configuration tasks and firmware updates.
Feature codesa A1EP / 3772 A1EQ / 3773 A3HN / ESW3 A3HP / ESW4
a. The first feature code listed is for configurations ordered through System x sales channels (HVEC) using x-config. The second feature code is for configurations ordered through the IBM Power Systems channel (AAS) using e-config.
With DPOD, ports are licensed as they come online. With the FC5022 16Gb SAN Scalable Switch, the first 12 ports that report (on a first-come, first-served basis) on boot are assigned licenses. These 12 ports can be any combination of external or internal Fibre Channel ports. After all the licenses are assigned, you can manually move those licenses from one port to another port. Because this process is dynamic, no defined ports are reserved except ports 0 and 29. The FC5022 16Gb ESB Switch has the same behavior. The only difference is the number of ports.
150
IBM PureFlex System and IBM Flex System Products and Technology
Table 4-38 shows the total number of active ports on the switch after you apply compatible port upgrades.
Table 4-38 Total port counts after you apply upgrades Total number of active ports 24-port 16 Gb ESB SAN switch Ports on Demand upgrade Included with base switch Upgrade 1, 88Y6382 (adds 12 ports) Upgrade 2, 88Y6386 (adds 24 ports) 90Y9356 24 Not supported 48 24-port 16 Gb SAN switch 00Y3324 24 Not supported 48 16 Gb SAN switch 88Y6374 12 24 48
Transceivers
The FC5022 12-port and 24-port ESB SAN switches come without SFP+, which must be ordered separately to provide outside connectivity. The FC5022 24-port SAN switch comes standard with two Brocade 16 Gb SFP+ transceivers; more SFP+ can be ordered if required. Table 4-39 lists the supported SFP+ options.
Table 4-39 Supported SFP+ transceivers Part number 88Y6416 88Y6393 Feature codea 5084 / 5370 A22R / 5371 Description Brocade 8 Gb SFP+ Software Optical Transceiver Brocade 16 Gb SFP+ Optical Transceiver
a. The first feature code listed is for configurations ordered through System x sales channels (HVEC) using x-config. The second feature code is for configurations ordered through the IBM Power Systems channel (AAS) using e-config.
Benefits
The switches offer the following key benefits: Exceptional price and performance for growing SAN workloads The FC5022 16Gb SAN Scalable Switch delivers exceptional price and performance for growing SAN workloads. It achieves this through a combination of market-leading 1,600 MBps throughput per port and an affordable high-density form factor. The 48 FC ports produce an aggregate 768 Gbps full-duplex throughput, plus any external eight ports can be trunked for 128 Gbps inter-switch links (ISLs). Because 16 Gbps port technology dramatically reduces the number of ports and associated optics and cabling required through 8/4 Gbps consolidation, the cost savings and simplification benefits are substantial. Accelerating fabric deployment and serviceability with diagnostic ports Diagnostic Ports (D_Ports) are a new port type that is supported by the FC5022 16Gb SAN Scalable Switch. They enable administrators to quickly identify and isolate 16 Gbps optics, port, and cable problems, which reduces fabric deployment and diagnostic times. If the optical media is found to be the source of the problem, it can be transparently replaced because 16 Gbps optics are hot-pluggable.
151
A building block for virtualized, private cloud storage The FC5022 16Gb SAN Scalable Switch supports multi-tenancy in cloud environments through VM-aware end-to-end visibility and monitoring, QoS, and fabric-based advanced zoning features. The FC5022 16Gb SAN Scalable Switch enables secure distance extension to virtual private or hybrid clouds with dark fiber support. They also enable in-flight encryption and data compression. Internal fault-tolerant and enterprise-class reliability, availability, and serviceability (RAS) features help minimize downtime to support mission-critical cloud environments. Simplified and optimized interconnect with Brocade Access Gateway The FC5022 16Gb SAN Scalable Switch can be deployed as a full-fabric switch or as a Brocade Access Gateway. It simplifies fabric topologies and heterogeneous fabric connectivity. Access Gateway mode uses N_Port ID Virtualization (NPIV) switch standards to present physical and virtual servers directly to the core of SAN fabrics. This configuration makes it not apparent to the SAN fabric, which greatly reduces management of the network edge. Maximizing investments To help optimize technology investments, IBM offers a single point of serviceability that is backed by industry-renowned education, support, and training. In addition, the IBM 16/8 Gbps SAN Scalable Switch is in the IBM ServerProven program, which enables compatibility among various IBM and partner products. IBM recognizes that customers deserve the most innovative, expert integrated systems solutions.
152
IBM PureFlex System and IBM Flex System Products and Technology
Brocade Fabric OS delivers distributed intelligence throughout the network and enables a wide range of value-added applications. These applications include Brocade Advanced Web Tools and Brocade Advanced Fabric Services (on certain models). Supports up to 768 Gbps I/O bandwidth. A total of 420 million frames switches per second, 0.7 microseconds latency. 8,192 buffers for up to 3,750 km extended distance at 4 Gbps FC (Extended Fabrics license required). In-flight 64 Gbps Fibre Channel compression and decompression support on up to two external ports (no license required). In-flight 32 Gbps encryption and decryption on up to two external ports (no license required). A total of 48 Virtual Channels per port. Port mirroring to monitor ingress or egress traffic from any port within the switch. Two I2C connections able to interface with redundant management modules. Hot pluggable, up to four hot pluggable switches per chassis. Single fuse circuit. Four temperature sensors. Managed with Brocade Web Tools. Supports a minimum of 128 domains in Native mode and Interoperability mode. Nondisruptive code load in Native mode and Access Gateway mode. 255 N_port logins per physical port. D_port support on external ports. Class 2 and Class 3 frames. SNMP v1 and v3 support. SSH v2 support. Secure Sockets Layer (SSL) support. NTP client support (NTP V3). FTP support for firmware upgrades. SNMP/Management Information Base (MIB) monitoring functionality that is contained within the Ethernet Control MIB-II (RFC1213-MIB). End-to-end optics and link validation. Sends switch events and syslogs to the CMM. Traps identify cold start, warm start, link up/link down and authentication failure events. Support for IPv4 and IPv6 on the management ports. The FC5022 16Gb SAN Scalable Switches come standard with the following software features: Brocade Full Fabric mode: Enables high performance 16 Gb or 8 Gb fabric switching. Brocade Access Gateway mode: Uses NPIV to connect to any fabric without adding switch domains to reduce management complexity. Dynamic Path Selection: Enables exchange-based load balancing across multiple Inter-Switch Links for superior performance.
153
Brocade Advanced Zoning: Segments a SAN into virtual private SANs to increase security and availability. Brocade Enhanced Group Management: Enables centralized and simplified management of Brocade fabrics through IBM Network Advisor.
FC-TAPE INCITS TR-24: 1999 FC-DA INCITS TR-36: 2004, includes the following standards: FC-FLA INCITS TR-20: 1998 FC-PLDA INCIT S TR-19: 1998 FC-MI-2 ANSI/INCITS TR-39-2005 FC-PI INCITS 352: 2002 FC-PI-2 INCITS 404: 2005 FC-PI-4 INCITS 1647-D, revision 7.1 (under development) FC-PI-5 INCITS 479: 2011 FC-FS-2 ANSI/INCITS 424:2006 (includes FC-FS INCITS 373: 2003) FC-LS INCITS 433: 2007 FC-BB-3 INCITS 414: 2006 FC-BB-2 INCITS 372: 2003 FC-SB-3 INCITS 374: 2003 (replaces FC-SB ANSI X3.271: 1996 and FC-SB-2 INCITS 374: 2001) RFC 2625 IP and ARP Over FC RFC 2837 Fabric Element MIB MIB-FA INCITS TR-32: 2003 FCP-2 INCITS 350: 2003 (replaces FCP ANSI X3.269: 1996) SNIA Storage Management Initiative Specification (SMI-S) Version 1.2 and includes the following standards: SNIA Storage Management Initiative Specification (SMI-S) Version 1.03 ISO standard IS24775-2006. (replaces ANSI INCITS 388: 2004) SNIA Storage Management Initiative Specification (SMI-S) Version 1.1.0 SNIA Storage Management Initiative Specification (SMI-S) Version 1.2.0 For more information, see the IBM Redbooks Product Guide IBM Flex System FC5022 16Gb SAN Scalable Switches, TIPS0870, which is available at this website: http://www.redbooks.ibm.com/abstracts/tips0870.html?Open
155
The I/O module has 14 internal ports and 6 external ports. All ports are licensed on the switch because there are no port licensing requirements. Ordering information is listed in Table 4-40.
Table 4-40 FC3171 8Gb SAN Switch Part number 69Y1930 Feature codea A0TD / 3595 Product Name IBM Flex System FC3171 8Gb SAN Switch
a. The first feature code listed is for configurations ordered through System x sales channels (HVEC) using x-config. The second feature code is for configurations ordered through the IBM Power Systems channel (AAS) using e-config.
No SFP modules and cables are supplied as standard. The ones that are listed in Table 4-41 are supported.
Table 4-41 FC3171 8Gb SAN Switch supported SFP modules and cables Part number 44X1964 39R6475 Feature codesa 5075 / 3286 4804 / 3238 Description IBM 8 Gb SFP+ Software Optical Transceiver 4 Gb SFP Transceiver Option
a. The first feature code listed is for configurations ordered through System x sales channels (HVEC) using x-config. The second feature code is for configurations ordered through the IBM Power Systems channel (AAS) using e-config.
You can reconfigure the FC3171 8Gb SAN Switch to become a pass-through module by using the switch GUI or CLI. The module can then be converted back to a full function SAN switch at some future date. The switch requires a reset when you turn on or off transparent mode. The switch can be configured by using the following methods: Command Line Access the switch by using the console port through the CMM or through the Ethernet port. This method requires a basic understanding of the CLI commands. QuickTools Requires a current version of the Java runtime environment on your workstation before you point a web browser to the switchs IP address. The IP address of the switch must be configured. QuickTools does not require a license and code is included. On this switch when in Full Fabric mode, access to all of the Fibre Channel Security features is provided. Security includes additional services of SSL and SSH, which are available. In addition, RADIUS servers can be used for device and user authentication. After SSL or SSH is enabled, the security features are available to be configured. Configuring security features allows the SAN administrator to configure which devices are allowed to log on to the Full Fabric Switch module. This process is done by creating security sets with security groups. These sets are configured on a per switch basis. The security features are not available when in pass-through mode. The FC3171 8Gb SAN Switch includes the following specifications and standards: Fibre Channel standards: 156 C-PH version 4.3 FC-PH-2 FC-PH-3 FC-AL version 4.5
IBM PureFlex System and IBM Flex System Products and Technology
FC-AL-2 Rev 7.0 FC-FLA FC-GS-3 FC-FG FC-PLDA FC-Tape FC-VI FC-SW-2 Fibre Channel Element MIB RFC 2837 Fibre Alliance MIB version 4.0
Fibre Channel protocols: Fibre Channel service classes: Class 2 and class 3 Operation modes: Fibre Channel class 2 and class 3, connectionless External port type: Full fabric mode: Generic loop port Transparent mode: Transparent fabric port Internal port type: Full fabric mode: F_port Transparent mode: Transparent host port/NPIV mode Support for up to 44 host NPIV logins Port characteristics: External ports are automatically detected and self-configuring Port LEDs illuminate at startup Number of Fibre Channel ports: 6 external ports and 14 internal ports Scalability: Up to 239 switches maximum depending on your configuration Buffer credits: 16 buffer credits per port Maximum frame size: 2148 bytes (2112 byte payload) Standards-based FC, FC-SW2 Interoperability Support for up to a 255 to 1 port-mapping ratio Media type: SFP+ module
2 Gb specifications: 2 Gb fabric port speed: 1.0625 or 2.125 Gbps (gigabits per second) 2 Gb fabric latency: Less than 0.4 msec 2 Gb fabric aggregate bandwidth: 80 Gbps at full duplex 4 Gb specifications: 4 Gb switch speed: 4.250 Gbps 4 Gb switch fabric point-to-point: 4 Gbps at full duplex 4 Gb switch fabric aggregate bandwidth: 160 Gbps at full duplex 8 Gb specifications: 8 Gb switch speed: 8.5 Gbps 8 Gb switch fabric point-to-point: 8 Gbps at full duplex 8 Gb switch fabric aggregate bandwidth: 320 Gbps at full duplex Nonblocking architecture to prevent latency System processor: IBM PowerPC For more information, see the IBM Redbooks Product Guide IBM Flex System FC3171 8Gb SAN Switch and Pass-thru, TIPS0866, which is available at: http://www.redbooks.ibm.com/abstracts/tips0866.html?Open
157
a. The first feature code listed is for configurations ordered through System x sales channels (HVEC) using x-config. The second feature code is for configurations ordered through the IBM Power Systems channel (AAS) using e-config.
Exception: If you must enable full fabric capability later, do not purchase this switch. Instead, purchase the FC3171 8Gb SAN Switch. There are no SFPs supplied with the switch and must be ordered separately. Supported transceivers and fiber optic cables are listed in Table 4-43.
Table 4-43 FC3171 8Gb SAN Pass-thru supported modules and cables Part number 44X1964 39R6475 Feature code 5075 / 3286 4804 / 3238 Description IBM 8 Gb SFP+ Software Optical Transceiver 4 Gb SFP Transceiver Option
The FC3171 8Gb SAN Pass-thru can be configured by using the following methods: Command Line Access the module by using the console port through the Chassis Management Module or through the Ethernet port. This method requires a basic understanding of the CLI commands. QuickTools Requires a current version of the JRE on your workstation before you point a web browser to the modules IP address. The IP address of the module must be configured. QuickTools does not require a license and the code is included.
158
IBM PureFlex System and IBM Flex System Products and Technology
The pass-through module supports the following standards: Fibre Channel standards: C-PH version 4.3 FC-PH-2 FC-PH-3 FC-AL version 4.5 FC-AL-2 Rev 7.0 FC-FLA FC-GS-3 FC-FG FC-PLDA FC-Tape FC-VI FC-SW-2 Fibre Channel Element MIB RFC 2837 Fibre Alliance MIB version 4.0
Fibre Channel protocols: Fibre Channel service classes: Class 2 and class 3 Operation modes: Fibre Channel class 2 and class 3, connectionless External port type: Transparent fabric port Internal port type: Transparent host port/NPIV mode Support for up to 44 host NPIV logins Port characteristics: External ports are automatically detected and self- configuring Port LEDs illuminate at startup Number of Fibre Channel ports: 6 external ports and 14 internal ports Scalability: Up to 239 switches maximum depending on your configuration Buffer credits: 16 buffer credits per port Maximum frame size: 2148 bytes (2112 byte payload) Standards-based FC, FC-SW2 Interoperability Support for up to a 255 to 1 port-mapping ratio Media type: SFP+ module
Fabric point-to-point bandwidth: 2 Gbps or 8 Gbps at full duplex 2 Gb Specifications: 2 Gb fabric port speed: 1.0625 or 2.125 Gbps (gigabits per second) 2 Gb fabric latency: Less than 0.4 msec 2 Gb fabric aggregate bandwidth: 80 Gbps at full duplex 4 Gb Specifications: 4 Gb switch speed: 4.250 Gbps 4 Gb switch fabric point-to-point: 4 Gbps at full duplex 4 Gb switch fabric aggregate bandwidth: 160 Gbps at full duplex 8 Gb Specifications: 8 Gb switch speed: 8.5 Gbps 8 Gb switch fabric point-to-point: 8 Gbps at full duplex 8 Gb switch fabric aggregate bandwidth: 320 Gbps at full duplex System processor: PowerPC Maximum frame size: 2148 bytes (2112 byte payload)
159
Nonblocking architecture to prevent latency For more information, see the IBM Redbooks Product Guide IBM Flex System FC3171 8Gb SAN Switch and Pass-thru, TIPS0866, which is available at this website: http://www.redbooks.ibm.com/abstracts/tips0866.html?Open
90Y3450
90Y3462
A1QX / ESW1
a. The first feature code listed is for configurations ordered through System x sales channels (HVEC) using x-config. The second feature code is for configurations ordered through the IBM Power Systems channel (AAS) using e-config.
Running the MLNX-OS, this switch has one external 1 Gb management port and a mini USB Serial port for updating software and debug use. These ports are in addition to InfiniBand internal and external ports. The switch has 14 internal QDR links and 18 CX4 uplink ports. All ports are enabled. The switch can be upgraded to FDR speed (56 Gbps) by using the FOD process with part number 90Y3462 as listed in Table 4-44. There are no InfiniBand cables that are shipped as standard with this switch and these must be purchased separately. Supported cables are listed in Table 4-45.
Table 4-45 IB6131 InfiniBand Switch InfiniBand supported cables Part number 49Y9980 90Y3470 Feature codesa 3866 / 3249 A227 / ECB1 Description IB QDR 3m QSFP Cable Option (passive) 3m FDR InfiniBand Cable (passive)
a. The first feature code listed is for configurations ordered through System x sales channels (HVEC) using x-config. The second feature code is for configurations ordered through the IBM Power Systems channel (AAS) using e-config.
160
IBM PureFlex System and IBM Flex System Products and Technology
The switch includes the following specifications: IBTA 1.3 and 1.21 compliance Congestion control Adaptive routing Port mirroring Auto-Negotiation of 10 Gbps, 20 Gbps, 40 Gbps, or 56 Gbps Measured node-to-node latency of less than 170 nanoseconds Mellanox QoS: 9 InfiniBand virtual lanes for all ports, eight data transport lanes, and one management lane High switching performance: Simultaneous wire-speed any port to any port Addressing: 48K Unicast Addresses maximum per Subnet, 16K Multicast Addresses per Subnet Switch throughput capability of 1.8 Tb/s For more information, see the IBM Redbooks Product Guide IBM Flex System IB6131 InfiniBand Switch, TIPS0871, which is available at this website: http://www.redbooks.ibm.com/abstracts/tips0871.html?Open
161
162
IBM PureFlex System and IBM Flex System Products and Technology
The chassis power system is designed for efficiency by using data center power that consists of 3-phase, 60A Delta 200 VAC (North America), or 3-phase 32A wye 380-415 VAC (international). The chassis can also be fed from single phase 200-240VAC supplies if required. The power is scaled as required, so as more nodes are added, the power and cooling increases accordingly. For power planning, Table 4-11 on page 93 shows the number of power supplies needed for N+N or N+1, node dependent. This section explains single phase and 3-phase example configurations for North America and worldwide, starting with 3-phase and assumes that you have power budget in your configuration to deliver N+N or N+1, given your particular node configuration. The 2100W power modules have the advantage in North America that they draw a maximum 11.8A as opposed to 13.8A of the 2500W power modules. This means that when you are using a 30A supply, which is derated to 24A with a PDU, up to two 2100W power modules can be connected to the same PDU with 0.4A remaining. With 2500W power modules, only one power module can be connected to a 30A PDU at the maximum (label) rating. Thus, for North America, the 2100W power module is advantageous for 30A supply PDU deployments. Figure 4-59 shows two chassis that were populated with six 2100W power supplies. Six 30A PDUs were configured to supply power to the two chassis.
Each PDU has up to 22.8A (label rating) draw by two 2100W Flex Chassis power supplies (2x 11.8A). 0.4A capacity remaining on a 24A Derated PDU.
11.8A
11.8A
11.8A
11.8A
11.8A
11.8A 11.8A
6x 40K9614 IBM DPI 30A 1ph Cord with NEMA L6-30P connector (71762NX + 40K9614 = FC 6500)
Figure 4-59 2100W power supplies optimized for use with 30A UL Derated PDU
163
L3 N G
L2 L1
L3 N G
L2 L1
= Power cables
The maximum number of Enterprise Chassis that can be installed with a 42U rack is four. Therefore, the chassis requires a total of four 32A 3-phase wye feeds to provide for a redundant N+N configuration.
164
IBM PureFlex System and IBM Flex System Products and Technology
L1 G L2 L3 L2
L1 G L3
165
N G
N G
166
IBM PureFlex System and IBM Flex System Products and Technology
L1 G L2 L3 L2
L1 G L3
40K9615 IBM DPI 60a Cord (IEC 309 2P+G) Building power = 200 VAC, 60 Amp, 1 Phase (48A supplied by PDU after UL derating)
= Cables
For more information about planning your IBM Flex System power infrastructure, see IBM Flex System Enterprise Chassis Power Requirements Guide, WP102111, which is available at this website: http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102111
167
= Cables
In North America, the available UPS at 200-208VAC is the UPS6000. This UPS has two outlets that can be used to power two of the power supplies within the chassis. In a fully loaded chassis, the third pair of power supplies must be connected to another UPS. Figure 4-65 shows this UPS configuration.
= Cables
Figure 4-65 Two UPS 6000 North American (200 - 208 VAC)
For more information, see IBM 11000VA LCD 5U Rack Uninterruptible Power Supply, TIPS0814, which is available at this website: http://www.redbooks.ibm.com/abstracts/tips0814.html?Open
168
IBM PureFlex System and IBM Flex System Products and Technology
a. The feature code listed is for both the System x sales channel (HVEC) using x-config and the Power Systems sales channel (AAS) using e-config.
169
The 12.9kW figure is only a potential maximum, where the most power hungry configuration is chosen and all power envelopes are maximum. For a more realistic figure, use the IBM Power Configurator tool to establish specific power requirements for a configuration, which is available at this website: http://www.ibm.com/systems/x/hardware/configtools.html Data center operation at environmental temperatures above 35 C can generally be operated in a free air cooling environment where outside air is filtered and then used to ventilate the data center. This is the definition of ASHRAE class A3 (and also the A4 class, which raises the upper limit to 45 C). A conventional data center does not normally run with computer room air conditioning (CRAC) units up to 40 C because the risk of failures of CRAC or power to the CRACs failing gives limited time for shutdowns before over-temperature events occur. IBM Flex System Enterprise Chassis is suitable for operation in an ASHRAE class A3 environment that is installed in operating and non-operating mode. Information about ASHRAE 2011 thermal guidelines, data center classes, and white papers can be found at the American Society of Heating, Refrigerating, and Air-Conditioning Engineers (ASHRAE) website: http://www.ashrae.org The chassis can be installed within IBM or non-IBM racks. However, the IBM 42U 1100mm Enterprise V2 Dynamic Rack does offer in North America a single floor tile wide and two tiles deep. More information about this sizing, see 4.13, IBM 42U 1100mm Enterprise V2 Dynamic Rack on page 172. If installed within a non-IBM rack, the vertical rails must have clearances to EIA-310-D. There must be sufficient room in front of the vertical front rack-mounted rail to provide minimum bezel clearance of 70 mm (2.76 inches) depth. The rack must be sufficient to support the weight of the chassis, cables, power supplies, and other items that are installed within. There must be sufficient room behind the rear of the rear rack rails to provide for cable management and routing. Ensure the stability of any non-IBM rack by using stabilization feet or baying kits so that it does not become unstable when it is fully populated. Finally, ensure that sufficient airflow is available to the Enterprise Chassis. Racks with glass fronts do not normally allow sufficient airflow into the chassis, unless they are specialized racks that are specifically designed for forced air cooling. Airflow information in CFM is available from the IBM Power Configurator tool.
170
IBM PureFlex System and IBM Flex System Products and Technology
Table 4-50 lists the IBM Flex System Enterprise Chassis supported in each rack cabinet.
Table 4-50 The chassis that is supported in each rack cabinet Part number 93634PX 93634EX 93634CX 93634DX 93634AX 93634BX 201886X 93072PX 93072RX 93074RX 99564RX 99564XX 93084PX 93084EX 93604PX 93604EX 93614PX 93614EX 93624PX 93624EX 14102RX 14104RX Feature code A1RC A1RD A3GR A3GS A31F A31G 2731 6690 1042 1043 5629 5631 5621 5622 7649 7650 7651 7652 7653 7654 1047 1048 None None None None None Rack cabinet IBM 42U 1100 mm Enterprise V2 Deep Dynamic Rack IBM 42U 1100 mm Dynamic Enterprise V2 Expansion Rack IBM PureFlex System 42U Rack IBM PureFlex System 42U Expansion Rack IBM PureFlex System 42U Rack IBM PureFlex System 42U Expansion Rack IBM 11U Office Enablement Kit IBM S2 25U Static Standard Rack IBM S2 25U Dynamic Standard Rack IBM S2 42U Standard Rack IBM S2 42U Dynamic Standard Rack IBM S2 42U Dynamic Standard Expansion Rack IBM 42U Enterprise Rack IBM 42U Enterprise Expansion Rack IBM 42U 1200 mm Deep Dynamic Rack IBM 42U 1200 mm Deep Dynamic Expansion Rack IBM 42U 1200 mm Deep Static Rack IBM 42U 1200 mm Deep Static Expansion Rack IBM 47U 1200 mm Deep Static Rack IBM 47U 1200 mm Deep Static Expansion Rack IBM eServer Cluster 25U Rack IBM Linux Cluster 42U Rack IBM Netfinity Rack IBM Netfinity Rack IBM Netfinity Enterprise Rack IBM Netfinity Enterprise Rack Expansion Cabinet IBM Netfinity NetBAY 22 Enterprise Chassis Yesa Yesa Yesb Yesb Yesc Yesc Yesd Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No No No No No
a. This rack cabinet is optimized for IBM Flex System Enterprise Chassis, including dedicated front to back cable raceways. For more information, including images, see 4.13, IBM 42U 1100mm Enterprise V2 Dynamic Rack on page 172.
171
b. This rack cabinet is optimized for IBM Flex System Enterprise Chassis, including dedicated front to back cable raceways, and includes a unique PureFlex door. For more information, including images of the door, see 4.14, IBM PureFlex System 42U Rack and 42U Expansion Rack on page 178. c. This rack cabinet is optimized for IBM Flex System Enterprise Chassis, including dedicated front to back cable raceways, and includes the original square blue design of unique PureFlex Logod Door, shipped between Q2 and Q4 2012. d. This Office Enablement kit is specifically designed for the IBM BladeCenter S Chassis. The Flex System Enterprise Chassis can be installed within the 11U office enablement kit with 1U of space remaining; however, the acoustic footprint of a configuration might not be acceptable for office use. We recommend that an evaluation be performed before deployment in an office environment.
Racks that have glass-fronted doors do not allow sufficient airflow for the Enterprise Chassis, such as the Netfinity racks that are shown in Table 4-50 on page 171. In some cases with the older Netfinity racks, the chassis depth is such that the Enterprise Chassis cannot be accommodated within the dimensions of the rack.
This 42U rack conforms to the EIA-310-D industry standard for a 24-inch, type A rack cabinet. The dimensions are listed in Table 4-52.
Table 4-52 Dimensions of IBM 42U 1100mm Enterprise V2 Dynamic Rack, 9363-4PX Dimension Height Width Depth Weight Value 2009 mm (79.1 in.) 600 mm (23.6 in.) 1100 mm (43.3 in.) 174 kg (384 lb), including outriggers
The rack features outriggers (stabilizers) allowing for movement while populated.
172
IBM PureFlex System and IBM Flex System Products and Technology
The IBM 42U 1100mm Enterprise V2 Dynamic Rack includes the following features: A perforated front door allows for improved air flow. Square EIA Rail mount points. Six side-wall compartments support 1U-high PDUs and switches without taking up valuable rack space. Cable management rings are included to help cable management. Easy to install and remove side panels are a standard feature. The front door can be hinged on either side, which provides flexibility to open in either direction. Front and rear doors and side panels include locks and keys to help secure servers. Heavy-duty casters with the use of outriggers (stabilizers) come with the 42U Dynamic racks for added stability, which allows movement of the rack while loaded. Tool-less 0U PDU rear channel mounting reduces installation time and increases accessibility. 1U PDU can be mounted to present power outlets to the rear of the chassis in side pocket openings. Removable top and bottom cable access panels in both front and rear. IBM is a leading vendor with specific ship-loadable designs. These kinds of racks are called dynamic racks. The IBM 42U 1100mm Enterprise V2 Dynamic Rack and IBM 42U 1100mm Enterprise V2 Dynamic Expansion Rack are dynamic racks.
173
A dynamic rack has extra heavy-duty construction and sturdy packaging that can be reused for shipping a fully loaded rack. They also have outrigger casters for secure movement and tilt stability. Dynamic racks also include a heavy-duty shipping pallet that includes a ramp for easy on and off maneuvering. Dynamic racks undergo more shock and vibration testing, and all IBM racks are of welded rather than the more flimsy bolted construction. Figure 4-67 shows the rear view of the 42U 1100mm Flex System Dynamic Rack.
Cable raceway
Outriggers
Figure 4-67 42U 1100mm Flex System Dynamic Rack rear view, with doors and sides panels removed
The IBM 42U 1100mm Enterprise V2 Dynamic Rack rack also provides more space than previous rack designs, for front cable management of SAS cables exiting the V7000 Storage Node and the PCIe Expansion Node. There are four cable raceways on each rack, with two on each side. The raceways allow cables to be routed from the front of the rack, through the raceway, and out to the rear of the rack, which is required when connecting an externally mounted Storwize Expansion unit to an integrated V7000 Storage Node.
174
IBM PureFlex System and IBM Flex System Products and Technology
Cable raceway
Figure 4-68 Cable raceway (as viewed from rear of rack)
Figure 4-69 shows a cable raceway when viewed inside the rack looking down. Cables can enter the side bays of the rack from the raceway, or pass from one side bay to the other, passing vertically through the raceway. These openings are at the front and rear of each raceway.
Cable raceway
The 1U rack PDUs can also be accommodated in the side bays. In these bays, the PDU is mounted vertically in the rear of the side bay and presents its outlets to the rear of the rack. Four 0U PDUs can also be vertically mounted in the rear of the rack.
175
Rear vertical aperture blocked by a PDU: When a PDU is installed in a rear side pocket bay, it is not possible to use the cable raceway vertical apertures at the rear. The rack width is 600 mm (which is a standard width of a floor tile in many locations) to complement current raised floor data center designs. Dimensions of the rack base are shown in Figure 4-70.
600 mm 46 mm 199 mm 65 mm
1100 mm
65 mm 458 mm
Front of Rack
Figure 4-70 Rack dimensions
The rack has square mounting holes that are common in the industry, onto which the Enterprise Chassis and other server and storage products can be mounted. For implementations where the front anti-tip plate is not required, an air baffle/air recirculation prevention plate is supplied with the rack. You might not want to use the plate when an airflow tile must be positioned directly in front of the rack.
176
IBM PureFlex System and IBM Flex System Products and Technology
This air baffle, which is shown in Figure 4-71, can be installed to the lower front of the rack. It helps prevent warm air from the rear of the rack from circulating underneath the rack to the front, which improves the cooling efficiency of the entire rack solution.
177
4.14 IBM PureFlex System 42U Rack and 42U Expansion Rack
The IBM PureFlex System 42U Rack and IBM PureFlex System 42U Expansion Rack are optimized for use with IBM Flex System components, IBM System x servers, and BladeCenter systems. Their robust design allows them to be shipped with equipment already installed. The rack footprint is 600 mm x 1100 mm. The IBM PureFlex System 42U Rack is shown in Figure 4-72.
These racks are usually shipped as standard with a PureFlex system, but they are available for ordering by clients who want to deploy rack solutions with a similar design across their data center. The door design also can be fitted to existing deployed PureFlex System racks that have the original solid blue door design that shipped from Q2 2012 onwards. Table 4-53 shows the available options and associated part numbers for the two PureFlex racks and the PureFlex door.
Table 4-53 PureFlex system racks and rack door Model 9363-4CX / A3GR 9363-4DX / A3GS 44X3132 / EU21 Description IBM PureFlex System 42U Rack IBM PureFlex System 42U Expansion Rack IBM PureFlex System Rack Door Details Primary Rack. Ships with side doors. Expansion Rack. Ships with no side doors, but with a baying kit to join onto a primary rack. Front door for rack that is embellished with PureFlex design.
These racks share the rack frame design of the IBM 42U 1100mm Enterprise V2 Dynamic Rack, but ship with a PureFlex branded door. The door can be ordered separately. These IBM PureFlex System 42U racks are industry-standard 19-inch racks that support IBM PureFlex System and Flex System chassis, IBM System x servers, and BladeCenter chassis.
178
IBM PureFlex System and IBM Flex System Products and Technology
The racks conform to the EIA-310-D industry standard for 19-inch, type A rack cabinets, and have outriggers (stabilizers), which allows for movement of large loads. The optional IBM Rear Door Heat eXchanger can be installed into this rack to provide a superior cooling solution, and the entire cabinet will still fit on a standard data center floor tile (width). For more information, see 4.15, IBM Rear Door Heat eXchanger V2 Type 1756 on page 180. The front door is hinged on one side only. The rear door can be hinged on either side and can be removed for ease of access when cabling or servicing systems within the rack. The front door is a unique PureFlex -branded front door that allows for excellent airflow into the rack. The rack includes the following features: Six side-wall compartments support 1U-high power distribution units (PDUs) and switches without taking up valuable rack space. Cable management slots are provided to route hook-and-loop fasteners around cables. Side panels are a standard feature and are easy to install and remove. Front and rear doors and side panels include locks and keys to help secure servers. Horizontal and vertical cable channels are built into the frame. Heavy-duty casters with outriggers (stabilizers) come with the 42U rack for added stability, which allows for movement of large loads. Tool-less 0U PDU rear channel mounting is provided. A 600 mm standard width to complement current raised-floor data center designs. Increase in depth to from 1,000 mm to 1,100 mm to improve cable management. Increase in door perforation to maximize airflow. Support for tool-less 0U PDU mounting, and 1U PDU easy installation of 1U PDUs. Front-to-back cable raceways for easy routing of cables such as Fibre Channel or SAS. Support for shipping of fully integrated solutions. Vertical cable channels that are built into the frame. Lockable doors and side panels. Heavy-duty casters to help safely move large loads in the rack. Front stabilizer plate. The door can be ordered as a separate part number for attaching to existing PureFlex racks. Rack specifications for the two IBM PureFlex System Racks and the PureFlex Rack door are shown in Table 4-54.
Table 4-54 IBM PureFlex System Rack specifications Description 9363-4CX Description PureFlex System 42U Rack Dimension Height Width Depth Weight Value 2009 mm (79.1 in.) 604 mm (23.8 in.) 1100 mm (43.3 in.) 179 kg (394 lb), including outriggers
179
Description 9363-4DX
Value 2009 mm (79.1 in.) 604 mm (23.8 in.) 1100 mm (43.3 in.) 142 kg (314 lb), including outriggers 1924 mm (75.8 in.) 597 mm (23.5 in.) 90 mm (3.6 in.) 19.5 kg (43 lb)
44X3132
180
IBM PureFlex System and IBM Flex System Products and Technology
Attaching a rear door heat exchanger to the rear of a rack allows up to 100,000 BTU/hr or 30kw of heat to be removed at a rack level. As the warm air passes through the heat exchanger, it is cooled with water and exits the rear of the rack cabinet into the data center. The door is designed to provide an overall air temperature drop of up to 25C measured between air that enters the exchanger and exits the rear. Figure 4-74 shows the internal workings of the IBM Rear Door Heat eXchanger V2.
The supply inlet hose provides an inlet for chilled, conditioned water. A return hose delivers warmed water back to the water pump or chiller in the cool loop. It must meet the water supply requirements for secondary loops.
Table 4-55 Rear door heat exchanger Model 1756-42X Description IBM Rear Door Heat eXchanger V2 for 9363 Racks Details Rear door heat exchanger that can be installed to the rear of the 9363 Rack
181
Figure 4-75 shows the percentage heat that is removed from a 30 KW heat load as a function of water temperature and water flow rate. With 18 Degrees at 10 (gpm), 90% of 30 kW heat is removed by the door.
% heat removal as function of water temperature and flow rate for given rack power, rack inlet temperature, and rack air flow rate 140 130 120 110
Water temperature 12C * 14C * 16C * 18C * 20C * 22C * 24C * Rack Power (W) = 30000 Tinlet, air (C) = 27 Airflow (cfm) = 2500 4 6 8 10 12 14
% heat removal
For efficient cooling, water pressure and water temperature must be delivered in accordance with the specifications listed in Table 4-56. The temperature must be maintained above the dew point to prevent condensation from forming.
Table 4-56 1756 RDHX specifications Rear Door heat exchanger V2 Depth Width Height Empty Weight Filled Weight Temperature Drop Water Temperature Specifications 129 mm (5.0 in) 600 mm (23.6 in) 1950 mm (76.8 in) 39 kg (85 lb) 48 kg (105 lb) Up to 25C (45F) between air exiting and entering RDHX Above Dew Point: 18C 1C (64.4F 1.8F) for ASHRAE Class 1 Environment 22C 1C (71.6F 1.8F) for ASHRAE Class 2 Environment Minimum: 22.7 liters (6 gallons) per minute, Maximum: 56.8 liters (15 gallons) per minute
Required water flow rate (as measured at the supply entrance to the heat exchanger)
182
IBM PureFlex System and IBM Flex System Products and Technology
The installation and planning guide provides lists of suppliers that can provide coolant distribution unit solutions, flexible hose assemblies, and water treatment that meet the suggested water quality requirements. It takes three people to install the rear door heat exchanger. The exchanger requires a non-conductive step ladder to be used for attachment of the upper hinge assembly. Consult the planning and implementation guides before proceeding. The installation and planning guides can be found at this website: http://www.ibm.com/support/entry/portal/
183
184
IBM PureFlex System and IBM Flex System Products and Technology
Chapter 5.
Compute nodes
This chapter describes the IBM Flex System servers or compute nodes. The applications that are installed on the compute nodes can run natively on a dedicated physical server or they can be virtualized in a virtual machine that is managed by a hypervisor layer. The IBM Flex System portfolio of compute nodes includes Intel Xeon processors and IBM POWER7 processors. Depending on the compute node design, nodes can come in one of the following form factors: Half-wide node: Occupies one chassis bay, half the width of the chassis (approximately 215 mm or 8.5 in.). An example is the IBM Flex System x240 Compute Node. Full-wide node: Occupies two chassis bays side-by-side, the full width of the chassis (approximately 435 mm or 17 in.). An example is the IBM Flex System p460 Compute Node. This chapter includes the following topics: 5.1, IBM Flex System Manager on page 186 5.2, IBM Flex System x220 Compute Node on page 186 5.3, IBM Flex System x222 Compute Node on page 216 5.4, IBM Flex System x240 Compute Node on page 234 5.5, IBM Flex System x440 Compute Node on page 275 5.6, IBM Flex System p260 and p24L Compute Nodes on page 298 5.7, IBM Flex System p270 Compute Node on page 318 5.8, IBM Flex System p460 Compute Node on page 335 5.9, IBM Flex System PCIe Expansion Node on page 356 5.10, IBM Flex System Storage Expansion Node on page 363 5.11, I/O adapters on page 370
185
5.2.1 Introduction
The IBM Flex System x220 Compute Node is a high-availability, scalable compute node that is optimized to support the next-generation microprocessor technology. With a balance of cost and system features, the x220 is an ideal platform for general business workloads. This section describes the key features of the server.
186
IBM PureFlex System and IBM Flex System Products and Technology
Figure 5-1 shows the front of the compute node and highlights the location of the controls, LEDs, and connectors.
Two 2.5 HS drive bays
USB port
Power
LED panel
Figure 5-2 shows the internal layout and major components of the x220.
Cover
Hard disk drive cage Hot-swap hard disk drive Right air baffle
DIMM
Figure 5-2 Exploded view of the x220, showing the major components
187
Chipset Memory
Memory maximums
RAID support
Network interfaces
188
IBM PureFlex System and IBM Flex System Products and Technology
Specification Two connectors for I/O adapters; each connector has PCIe x8+x4 interfaces. Includes an Expansion Connector (PCIe 3.0 x16) for future use to connect a compute node expansion unit. Dedicated PCIe 2.0 x4 interface for ServeRAID H1135 adapter only. USB ports: One external and two internal ports for an embedded hypervisor. A console breakout cable port on the front of the server provides local KVM and serial ports (cable standard with chassis; additional cables are optional). UEFI, IBM IMM2 with Renesas SH7757 controller, Predictive Failure Analysis, light path diagnostics panel, automatic server restart, and remote presence. Support for IBM Flex System Manager, IBM Systems Director, and IBM ServerGuide. Power-on password, administrator's password, and Trusted Platform Module V1.2. Matrox G200eR2 video core with 16 MB video memory that is integrated into the IMM2. Maximum resolution is 1600x1200 at 75 Hz with 16 M colors. Three-year customer-replaceable unit and onsite limited warranty with 9x5/NBD. Microsoft Windows Server 2008 R2, Red Hat Enterprise Linux 5 and 6, SUSE Linux Enterprise Server 10 and 11, VMware vSphere. For more information, see 5.2.13, Operating system support on page 215. Optional service upgrades are available through IBM ServicePac offerings: 4-hour or 2-hour response time, 8-hour fix time, 1-year or 2-year warranty extension, and remote technical support for IBM hardware and selected IBM and OEM software. Width: 217 mm (8.6 in.), height: 56 mm (2.2 in.), depth: 492 mm (19.4 in.) Maximum configuration: 6.4 kg (14.11 lb).
Ports
Systems management Security features Video Limited warranty Operating systems supported Service and support
Dimensions Weight
Figure 5-3 shows the components on the system board of the x220.
Hot-swap drive bay backplane Processor 2 and six memory DIMMs USB port 2 Broadcom Ethernet I/O connector 1 Fabric Connector
USB port 1
Expansion Connector
Figure 5-3 Layout of the IBM Flex System x220 Compute Node system board
189
5.2.2 Models
The current x220 models are shown in Table 5-2. All models include 4 GB of memory (one 4 GB DIMM) running at either 1333 MHz or 1066 MHz (depending on model).
Table 5-2 Models of the IBM Flex System x220 Compute Node, type 7906
Model Intel Processor E5-2400: 2 maximum Pentium 1400: 1 maximum Memory RAID adapter Disk baysa Disks Embed 1 GbEb I/O slots (used/ max) 1 / 2b 1 / 2b 1 / 2b 1 / 2b 1 / 2b
7906-A2x
1x Intel Pentium 1403 2C 2.6 GHz 5 MB 1066 MHz 80 W 1x Intel Xeon E5-2430L 6C 2.0 GHz 15 MB 1333 MHz 60 W 1x Intel Xeon E5-2403 4C 1.8 GHz 10 MB 1066 MHz 80 W 1x Intel Xeon E5-2420 6C 1.9 GHz 15 MB 1333 MHz 95 W 1x Intel Xeon E5-2418L 4C 2.0 GHz 10 MB 1333 MHz 50 W 1x Intel Xeon E5-2430 6C 2.2 GHz 15 MB 1333 MHz 95 W 1x Intel Xeon E5-2430 6C 2.2 GHz 15 MB 1333 MHz 95 W 1x Intel Xeon E5-2440 6C 2.4 GHz 15 MB 1333 MHz 95 W 1x Intel Xeon E5-2450 8C 2.1 GHz 20 MB 1600 MHz 95 W 1x Intel Xeon E5-2470 8C 2.3 GHz 20 MB 1600 MHz 95 W
1x 4 GB UDIMM (1066 MHz)c 1x 4 GB UDIMM 1333 MHz 1x 4 GB RDIMM (1066 MHz)c 1x 4 GB RDIMM 1333 MHz 1x4GB RDIMM 1333 MHz 1x 4 GB RDIMM 1333 MHz 1x 4 GB RDIMM 1333 MHz 1x 4 GB RDIMM 1333 MHz 1x 4 GB RDIMM 1333 MHzc 1x 4 GB RDIMM 1333 MHzc
ServeRAID C105 ServeRAID C105 ServeRAID C105 ServeRAID C105 ServeRAID C105 ServeRAID C105 ServeRAID C105 ServeRAID C105 ServeRAID C105 ServeRAID C105
2x 2.5 hot-swap 2x 2.5 hot-swap 2x 2.5 hot-swap 2x 2.5 hot-swap 2x 2.5 hot-swap 2x 2.5 hot-swap 2x 2.5 hot-swap 2x 2.5 hot-swap 2x 2.5 hot-swap 2x 2.5 hot-swap
Open
Standard
7906-B2x
Open
Standard
7906-C2x
Open
Standard
7906-D2x
Open
Standard
7906-F2x
Open
Standard
7906-G2x
Open
No
0/2 1 / 2b 1 / 2b
7906-G4x
Open
Standard
7906-H2x
Open
Standard
7906-J2x
Open
No
0/2
7906-L2x
Open
No
0/2
a. The 2.5-inch drive bays can be replaced and expanded with 1.8-inch bays and a ServeRAID M5115 RAID controller. This configuration supports up to eight 1.8-inch SSDs. b. These models include an embedded 1 Gb Ethernet controller. Connections are routed to the chassis midplane by using a Fabric Connector. Precludes the use of I/O connector 1 (except the ServeRAID M5115). c. For A2x and C2x, the memory operates at 1066 MHz, the memory speed of the processor. For J2x and L2x, memory operates at 1333 MHz to match the installed DIMM, rather than 1600 MHz.
190
IBM PureFlex System and IBM Flex System Products and Technology
Up to 14 x220 Compute Nodes can be installed in the chassis in 10U of rack space. The actual number of x220 systems that can be powered on in a chassis depends on the following factors: The TDP power rating for the processors that are installed in the x220 The number of power supplies installed in the chassis The capacity of the power supplies installed in the chassis (2100 W or 2500 W) The power redundancy policy used in the chassis (N+1 or N+N) Table 4-11 on page 93 provides guidelines about the number of x220 systems that can be powered on in the IBM Flex System Enterprise Chassis, based on the type and number of power supplies installed. The x220 is a half wide compute node and requires that the chassis shelf is installed in the IBM Flex System Enterprise Chassis. Figure 5-4 shows the chassis shelf in the chassis.
Figure 5-4 The IBM Flex System Enterprise Chassis showing the chassis shelf
The shelf is required for half-wide compute nodes. To allow for installation of the full-wide or larger, shelves must be removed from within the chassis. Remove the shelves by sliding the two latches on the shelf towards the center, and then sliding the shelf from the chassis.
191
The x220 also supports an Intel Pentium 1403 or 1407 dual-core processor for entry-level server applications. Only one Pentium processor is supported in the x220. CPU socket 2 must be left unused, and only six DIMM sockets are available. Figure 5-5 shows the system architecture of the x220 system.
(optional)
PCIe 2.0 x4
Front KVM port USB x1 DDR3 DIMMs 3 memory channels 2 DIMMs per channel QPI link (up to 8 GT/s) IMM v2 USB Video & serial Management to midplane PCIe 2.0 x2 PCIe 3.0 x8+x4 I/O connector 1 Intel Xeon Processor 2 PCIe 3.0 x4 PCIe 3.0 x4 PCIe 3.0 x8+x4 I/O connector 2 PCIe 3.0 x16 Sidecar connector
1 GbE LOM
Figure 5-5 IBM Flex System x220 Compute Node system board block diagram
The IBM Flex System x220 Compute Node has the following system architecture features as standard: Two 1356-pin, Socket B2 (LGA-1356) processor sockets An Intel C600 PCH Three memory channels per socket Up to two DIMMs per memory channel 12 DDR3 DIMM sockets Support for UDIMMs and RDIMMs One integrated 1 Gb Ethernet controller (1 GbE LOM in diagram) One LSI 2004 SAS controller Integrated software RAID 0 and 1 with support for the H1135 LSI-based RAID controller One IMM2 Two PCIe 3.0 I/O adapter connectors with one x8 and one x4 host connection each (12 lanes total). One internal and one external USB connector
192
IBM PureFlex System and IBM Flex System Products and Technology
Intel Pentium processors None Noneb A1VZ / None A1W0 / None Intel Pentium 1403 2C 2.6 GHz 5 MB 1066 MHz 80 W Intel Pentium 1407 2C 2.8 GHz 5 MB 1066 MHz 80 W A2x -
Intel Xeon processors Noneb 90Y4801 90Y4800 90Y4799 90Y4797 90Y4796 90Y4795 90Y4793 A3C4 / None A1VY / A1WC A1VX / A1WB A1VW / A1WA A1VU / A1W8 A1VT / A1W7 A1VS / A1W6 A1VQ / A1W4 Intel Xeon E5-1410 4C 2.8 GHz 10 MB 1333 MHz 80 W Intel Xeon E5-2403 4C 1.8 GHz 10 MB 1066 MHz 80 W Intel Xeon E5-2407 4C 2.2 GHz 10 MB 1066 MHz 80 W Intel Xeon E5-2420 6C 1.9 GHz 15 MB 1333 MHz 95 W Intel Xeon E5-2430 6C 2.2 GHz 15 MB 1333 MHz 95 W Intel Xeon E5-2440 6C 2.4 GHz 15 MB 1333 MHz 95 W Intel Xeon E5-2450 8C 2.1 GHz 20 MB 1600 MHz 95 W Intel Xeon E5-2470 8C 2.3 GHz 20 MB 1600 MHz 95 W C2x D2x G2x, G4x H2x J2x L2x
Intel Xeon processors - Low power 00D9528 00D9527 90Y4805 00D9526 90Y4804 A3C7 / A3CA A3C6 / A3C9 A1W2 / A1WE A3C5 / A3C8 A1W1 / A1WD Intel Xeon E5-2418L 4C 2.0 GHz 10 MB 1333 MHz 50 W Intel Xeon E5-2428L 6C 1.8 GHz 15 MB 1333 MHz 60 W Intel Xeon E5-2430L 6C 2.0 GHz 15 MB 1333 MHz 60 W Intel Xeon E5-2448L 8C 1.8 GHz 20 MB 1600 MHz 70 W Intel Xeon E5-2450L 8C 1.8 GHz 20 MB 1600 MHz 70 W F2x B2x -
a. The first feature code is for processor 1 and second feature code is for processor 2. b. The Intel Pentium 1407 and Intel Xeon E5-1410 are available through CTO or special bid only.
193
The x220 supports LP DDR3 memory LRDIMMs, RDIMMs, and UDIMMs. The server supports up to six DIMMs when one processor is installed, and up to 12 DIMMs when two processors are installed. Each processor has three memory channels, with two DIMMs per channel. The following rules apply when you select the memory configuration: Mixing 1.5 V and 1.35 V DIMMs in the same server is supported. In such a case, all DIMMs operate at 1.5 V. The maximum number of ranks that are supported per channel is eight. The maximum quantity of DIMMs that can be installed in the server depends on the number of processors. For more information, see the Maximum quantity row in Table 5-5 and Table 5-6 on page 195. All DIMMs in all processor memory channels operate at the same speed, which is determined as the following lowest values: Memory speed that is supported by a specific processor. Lowest maximum operating speed for the selected memory configuration that depends on rated speed. For more information, see the Maximum operating speed section in Table 5-5 and Table 5-6 on page 195. The shaded cells indicate that the speed that is indicated is the maximum that the DIMM allows. Cells that are highlighted with a gray background indicate when the specific combination of DIMM voltage and number of DIMMs per channel still allows the DIMMs to operate at rated speed.
Table 5-5 Maximum memory speeds (Part 1 - UDIMMs and LRDIMMs) Spec Rank Part numbers Rated speed Rated voltage Operating voltage Maximum quantitya Largest DIMM Max memory capacity Max memory at rated speed Maximum operating speed 1 DIMM per channel 2 DIMMs per channel 1333 MHz 1066 MHz 1333 MHz 1066 MHz 1333 MHz 1066 MHz 1333 MHz 1066 MHz 1066 MHz 1066 MHz 1333 MHz 1066 MHz 1.35 V 12 2 GB 24 GB 12 GB Single rank 49Y1403 (2 GB) 1333 MHz 1.35 V 1.5 V 12 2 GB 24 GB 12 GB 1.35 V 12 4 GB 48 GB 24 GB UDIMMs Dual rank 49Y1404 (4 GB) 1333 MHz 1.35 V 1.5 V 12 4 GB 48 GB 24 GB 1.35 V 12 32 GB 384 GB N/A LRDIMMs Quad rank 90Y3105 (32 GB) 1333 MHz 1.35 V 1.5 V 12 32 GB 384 GB 192 GB
a. The maximum quantity that is supported is shown for two processors installed. When one processor is installed, the maximum quantity that is supported is half of that shown.
194
IBM PureFlex System and IBM Flex System Products and Technology
Table 5-6 Maximum memory speeds (Part 2 - RDIMMs) Spec Rank Part numbers Rated speed Rated voltage Operating voltage Max quantitya Largest DIMM Max memory capacity Max memory at rated speed 1.35 V 12 4 GB 48 GB 48 GB Single rank 49Y1406 (4 GB) 1333 MHz 1.35 V 1.5 V 12 4 GB 48 GB 48 GB 1.35 V 12 8 GB 96 GB 96 GB RDIMMs Dual rank 49Y1407 (4 GB) 49Y1397 (8 GB) 1333 MHz 1.35 V 1.5 V 12 8 GB 96 GB 96 GB 90Y3109 (4 GB) 1600 MHz 1.5 V 1.5 V 12 4 GB 48 GB 48 GB Quad rank 49Y1400 (16 GB) 1066 MHz 1.35 V 1.35 V 12 16 GB 192 GB N/A 1.5 V 12 16 GB 192 GB N/A
Maximum operating speed (MHz) 1 DIMM per channel 2 DIMMs per channel 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1600 MHz 1600 MHz 800 MHz 800 MHz 800 MHz 800 MHz
a. The maximum quantity that is supported is shown for two processors installed. When one processor is installed, the maximum quantity that is supported is half of that shown.
The following memory protection technologies are supported: ECC Chipkill (for x4-based memory DIMMs; look for x4 in the DIMM description) Memory mirroring Memory sparing If memory mirroring is used, DIMMs must be installed in pairs (minimum of one pair per processor). Both DIMMs in a pair must be identical in type and size. If memory rank sparing is used, a minimum of one quad-rank DIMM or two single-rank or dual-rank DIMMs must be installed per populated channel. These DIMMs do not need to be identical. In rank sparing mode, one rank of a DIMM in each populated channel is reserved as spare memory. The size of a rank varies depending on the DIMMs installed. Table 5-7 lists the memory options available for the x220 server. DIMMs can be installed one at a time, but for performance reasons, install them in sets of three (one for each of the three memory channels) if possible.
Table 5-7 Supported memory DIMMs Part number Feature codea Description
Unbuffered DIMM (UDIMM) modules 49Y1403 49Y1404 A0QS 8648 2GB (1x2GB, 1Rx8, 1.35V) PC3L-10600 CL9 ECC DDR3 1333MHz LP UDIMM 4GB (1x4GB, 2Rx8, 1.35V) PC3L-10600 CL9 ECC DDR3 1333MHz LP UDIMM
195
Part number
Feature codea
Description
Registered DIMMs (RDIMMs) - 1333 MHz and 1066 MHz 49Y1406 49Y1407 49Y1397 49Y1563 49Y1400 8941 8942 8923 A1QT 8939 4GB (1x4GB, 1Rx4, 1.35V) PC3L-10600 CL9 ECC DDR3 1333MHz LP RDIMM 4GB (1x4GB, 2Rx8, 1.35V) PC3L-10600 CL9 ECC DDR3 1333MHz LP RDIMM 8GB (1x8GB, 2Rx4, 1.35V) PC3L-10600 CL9 ECC DDR3 1333MHz LP RDIMM 16GB (1x16GB, 2Rx4, 1.35V) PC3L-10600 CL9 ECC DDR3 1333MHz LP RDIMM 16GB (1x16GB, 4Rx4, 1.35V) PC3L-8500 CL7 ECC DDR3 1066MHz LP RDIMM
Registered DIMMs (RDIMMs) - 1600 MHz 49Y1559 90Y3178 90Y3109 00D4968 A28Z A24L A292 A2U5 4GB (1x4GB, 1Rx4, 1.5V) PC3-12800 CL11 ECC DDR3 1600MHz LP RDIMM 4GB (1x4GB, 2Rx8, 1.5V) PC3-12800 CL11 ECC DDR3 1600MHz LP RDIMM 8GB (1x8GB, 2Rx4, 1.5V) PC3-12800 CL11 ECC DDR3 1600MHz LP RDIMM 16GB (1x16GB, 2Rx4, 1.5V) PC3-12800 CL11 ECC DDR3 1600MHz LP RDIMM
Load-reduced DIMMs (LRDIMMs) 90Y3105 A291 32GB (1x32GB, 4Rx4, 1.35V) PC3L-10600 CL9 ECC DDR3 1333MHz LP LRDIMM
a. The feature code listed is for both the System x sales channel (HVEC) using x-config and the Power Systems sales channel (AAS) using e-config.
196
IBM PureFlex System and IBM Flex System Products and Technology
Table 5-8 shows DIMM installation if you have one processor installed.
Table 5-8 Suggested DIMM installation with one processor installed (independent channel mode) Optimal memory configa Processor 1 Channel 1 Channel 2 Channel 3 Channel 1 Processor 2 Channel 2 Channel 3
Number of processors
Number of DIMMs
DIMM 11
DIMM 12
DIMM 10
DIMM 1
DIMM 2
DIMM 3
DIMM 4
DIMM 5
DIMM 6
DIMM 9
DIMM 7 DIMM 7 x x x x x
1 1 x 1 1 1 x 1
1 2 3 4 5 6 x x x x x x x x x x x x
x x x x x x x x x
Table 5-9 shows DIMM installation if you have two processors installed.
Table 5-9 Suggested DIMM installation with two processors installed (independent channel mode) Optimal memory configa Processor 1 Channel 1 Channel 2 Channel 3 Channel 1 Processor 2 Channel 2 Channel 3
Number of processors
Number of DIMMs
DIMM 11
DIMM 12
DIMM 10
DIMM 1
DIMM 2
DIMM 3
DIMM 4
DIMM 5
DIMM 6
DIMM 9
2 2 2 2 x 2 2 2 2 2 2 x 1
2 3 4 5 6 7 8 9 10 11 12 x x x x x x x x x x x x x x x x x x x x x x x x
x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x
DIMM 8 x x x x x x x x x x x
DIMM 8
197
Number of processors
Number of DIMMs
DIMM 11
DIMM 12
DIMM 10
DIMM 1
DIMM 2
DIMM 3
DIMM 4
DIMM 5
DIMM 6
DIMM 9
DIMM 7
1 1 x 1
2 4 6 x x x x x x
x x x
x x x
198
IBM PureFlex System and IBM Flex System Products and Technology
DIMM 8
Table 5-11 shows DIMM installation if you have two processors that are installed with rank sparing, by using dual or single ranked DIMMs.
Table 5-11 Suggested DIMM installation with 2 processors, rank-sparing mode, single or dual ranked Optimal memory configa Processor 1 Channel 1 Channel 2 Channel 3 Channel 1 Processor 2 Channel 2 Channel 3
Number of processors
Number of DIMMs
DIMM 11
DIMM 12
DIMM 10
DIMM 1
DIMM 2
DIMM 3
DIMM 4
DIMM 5
DIMM 6
DIMM 9
DIMM 7 x x x x x
2 2 x 2 2 x 2
4 6 8 10 12 x x x x x x x x x x x x
x x x x x
x x x x x x x x x x x x x
a. The pair of DIMMs must be identical in capacity, type, and rank count.
DIMM 8 x x x x x
199
Table 5-13 and Table 5-14 show the suggested DIMM installation in mirrored channel mode for one or two processors.
Table 5-13 Suggested DIMM installation with one processor - mirrored channel mode Optimal memory configa Processor 1 Channel 1 Channel 2 Channel 3 Channel 1 Processor 2 Channel 2 Channel 3
Number of processors
Number of DIMMsb
DIMM 11
DIMM 12
DIMM 10
DIMM 1
DIMM 2
DIMM 3
DIMM 4
DIMM 5
DIMM 6
DIMM 9
DIMM 7 DIMM 7 x
1 x 1
4 6
x x x
x x x
a. For optimal memory performance, populate all memory channels equally. b. The pair of DIMMs must be identical in capacity, type, and rank count. Table 5-14 Suggested DIMM installation with two processors - mirrored channel mode Optimal memory configa Processor 1 Channel 1 Channel 2 Channel 3 Channel 1 Processor 2 Channel 2 Channel 3
Number of processors
Number of DIMMsb
DIMM 11
DIMM 12
DIMM 10
DIMM 1
DIMM 2
DIMM 3
DIMM 4
DIMM 5
DIMM 6
DIMM 9
2 2
4 6 8
x x x x x
x x x x x x
x x x
a. For optimal memory performance, populate all memory channels equally. b. The pair of DIMMs must be identical in capacity, type, and rank count.
Memory installation considerations for IBM Flex System x220 Compute Node
Use the following general guidelines when you determine the memory configuration of your IBM Flex System x220 Compute Node: All memory installation considerations apply equally to one- and two-processor systems. All DIMMs must be DDR3 DIMMs. Memory of different types (RDIMMs, and UDIMMs) cannot be mixed in the system. If you mix DIMMs with 1.35 V and 1.5 V, the system runs all of them at 1.5 V and you lose the energy advantage. If you mix DIMMs with different memory speeds, all DIMMs in the system run at the lowest speed. You cannot mix non-mirrored channel and mirrored channel modes.
200
IBM PureFlex System and IBM Flex System Products and Technology
DIMM 8 x x x
DIMM 8
Install memory DIMMs in order of their size, with the largest DIMM first. The correct installation order is the DIMM slot farthest from the processor first (DIMM slots 5, 8, 3, 10, 1, and 12). Install memory DIMMs in order of their rank, with the largest DIMM in the DIMM slot farthest from the processor. Start with DIMM slots 5 and 8 and work inwards. Memory DIMMs can be installed one DIMM at a time. However, avoid this configuration because it can affect performance. For maximum memory bandwidth, install one DIMM in each of the three memory channels (three DIMMs at a time). Populate equivalent ranks per channel. Physically, DIMM slots 2, 4, 6, 7, 9, and 11 must be populated (actual DIMM or DIMM filler). DIMM slots 1,3, 5, 8, 10, and 12 do not require a DIMM filler. Different memory modes require a different population order (see Table 5-12 on page 199, Table 5-13 on page 200, and Table 5-14 on page 200).
201
Fixed stripe unit size of 64 KB Support for MegaRAID Storage Manager management software Consideration: There is no native (in-box) driver for Windows and Linux. The drivers must be downloaded separately. In addition, there is no support for VMware, Hyper-V, Xen, or SSDs.
ServeRAID H1135
The x220 also supports an entry level hardware RAID solution with the addition of the ServeRAID H1135 Controller for IBM Flex System and BladeCenter. The H1135 is installed in a dedicated slot, as shown in Figure 5-3 on page 189. When the H1135 adapter is installed, the C105 controller is disabled. The H1135 has the following features: Based on the LSI SAS2004 6 Gbps SAS 4-port controller PCIe 2.0 x4 host interface CIOv form factor (supported in the x220 and BladeCenter HS23E) Support for SAS, SATA, and SSD drives Support for RAID 0, RAID 1, and non-RAID 6 Gbps throughput per port Support for up to two volumes Fixed stripe size of 64 KB Native driver support in Windows, Linux, and VMware S.M.A.R.T. support Support for MegaRAID Storage Manager management software
ServeRAID M5115
The ServeRAID M5115 SAS/SATA Controller (90Y4390) is an advanced RAID controller that supports RAID 0, 1, 10, 5, 50, and optional 6 and 60. It includes 1 GB of cache, which can be backed up to flash memory when it is attached to an optional supercapacitor. The M5115 attaches to the I/O adapter 1 connector. It can be attached even if the Fabric Connector is installed (used to route the embedded Gb Ethernet to chassis bays 1 and 2). The ServeRAID M5115 cannot be installed if an adapter is installed in I/O adapter slot 1. When the M5115 adapter is installed, the C105 controller is disabled. The ServeRAID M5115 supports the following combinations of 2.5-inch drives and 1.8-inch SSDs: Up to two 2.5-inch drives only Up to four 1.8-inch drives only Up to two 2.5-inch drives, plus up to four 1.8-inch SSDs Up to eight 1.8-inch SSDs For more information about these configurations, see ServeRAID M5115 configurations and options on page 203. The ServeRAID M5115 controller has the following specifications: Eight internal 6 Gbps SAS/SATA ports. PCI Express 3.0 x8 host interface. 6 Gbps throughput per port. 800 MHz dual-core IBM PowerPC processor with an LSI SAS2208 6 Gbps ROC controller. Support for RAID levels 0, 1, 10, 5, 50 standard; support for RAID 6 and 60 with optional upgrade using 90Y4411. 202
IBM PureFlex System and IBM Flex System Products and Technology
Optional onboard 1 GB data cache (DDR3 running at 1333 MHz) with optional flash backup (MegaRAID CacheVault technology) as part of the Enablement Kit 90Y4342. Support for SAS and SATA HDDs and SSDs. Support for intermixing SAS and SATA HDDs and SSDs. Mixing different types of drives in the same array (drive group) is not recommended. Support for SEDs with MegaRAID SafeStore. Optional support for SSD performance acceleration with MegaRAID FastPath and SSD caching with MegaRAID CacheCade Pro 2.0 (90Y4447). Support for up to 64 virtual drives, up to 128 drive groups, and up to 16 virtual drives per drive group. Also supports up to 32 physical drives per drive group. Support for LUN sizes up to 64 TB. Configurable stripe size up to 1 MB. Compliant with DDF CoD. S.M.A.R.T. support. MegaRAID Storage Manager management software.
At least one hardware kit is required with the ServeRAID M5115 controller. The following hardware kits enable specific drive support: ServeRAID M5100 Series Enablement Kit for IBM Flex System x220 (90Y4424) enables support for up to two 2.5-inch HDDs or SSDs in the hot-swap bays in the front of the server. It includes a CacheVault unit, which enables MegaRAID CacheVault flash cache protection. This enablement kit replaces the standard two-bay backplane that is attached through the system board to an onboard controller. The new backplane attaches with an included flex cable to the M5115 controller. It also includes an air baffle, which also serves as an attachment for the CacheVault unit. MegaRAID CacheVault flash cache protection uses NAND flash memory that is powered by a supercapacitor to protect data that is stored in the controller cache. This module eliminates the need for the lithium-ion battery that is commonly used to protect DRAM cache memory on PCI RAID controllers.
203
To avoid data loss or corruption during a power or server failure, CacheVault technology transfers the contents of the DRAM cache to NAND flash. This process uses power from the supercapacitor. After power is restored to the RAID controller, the saved data is transferred from the NAND flash back to the DRAM cache. The DRAM cache can then be flushed to disk. Tip: The Enablement Kit is only required if 2.5-inch drives are to be used. This kit is not required if you plan to install four or eight 1.8-inch SSDs only. ServeRAID M5100 Series IBM Flex System Flash Kit for x220 (90Y4425) enables support for up to four 1.8-inch SSDs. This kit replaces the standard two-bay backplane with a four-bay SSD backplane that attaches with an included flex cable to the M5115 controller. Because only SSDs are supported, a CacheVault unit is not required, so this kit does not have a supercapacitor. ServeRAID M5100 Series SSD Expansion Kit for IBM Flex System x220 (90Y4426) enables support for up to four internal 1.8-inch SSDs. This kit includes two air baffles (left and right) that can attach two 1.8-inch SSD attachment locations. It also contains flex cables for attachment to up to four 1.8-inch SSDs. Table 5-17 shows the kits that are required for each combination of drives. For example, if you plan to install eight 1.8-inch SSDs, you need the M5115 controller, the Flash kit, and the SSD Expansion kit.
Table 5-17 ServeRAID M5115 hardware kits Drive support that is required Maximum number of 2.5-inch drives 2 0 2 0 Maximum number of 1.8-inch SSDs 0 4 (front) 4 (internal) 8 (both) => => => => ServeRAID M5115 90Y4390 Required Required Required Required Required Required Components required Enablement Kit 90Y4424 Required Required Required Required Flash Kit 90Y4425 SSD Expansion Kit 90Y4426
204
IBM PureFlex System and IBM Flex System Products and Technology
Figure 5-6 shows how the ServeRAID M5115 and the Enablement Kit are installed in the server to support two 2.5-inch drives with MegaRAID CacheVault flash cache protection (see row 1 of Table 5-17 on page 204).
ServeRAID M5115 controller (90Y4390) with ServeRAID M5100 Series Enablement Kit for x220 (90Y4424)
ServeRAID M5115 controller
Figure 5-6 The ServeRAID M5115 and the Enablement Kit installed
Figure 5-7 shows how the ServeRAID M5115 and Flash and SSD Expansion Kits are installed in the server to support eight 1.8-inch solid-state drives (see row 4 of Table 5-17 on page 204).
ServeRAID M5115 controller (90Y4390) with ServeRAID M5100 Series Flash Kit for x220 (90Y4425) and ServeRAID M5100 Series SSD Expansion Kit for x220 (90Y4426)
ServeRAID M5115 controller
SSD Expansion Kit: Four SSDs on special air baffles above DIMMs (no CacheVault flash protection)
Figure 5-7 ServeRAID M5115 with Flash and SSD Expansion Kits installed
The eight SSDs are installed in the following locations: Four in the front of the system in place of the two 2.5-inch drive bays Two in a tray above the memory banks for processor 1 Two in a tray above the memory banks for processor 2
205
Optional add-ons to the ServeRAID M5115 controller are RAID 6 support, SSD performance accelerator, and SSD caching enabler. The FoD license upgrades are listed in Table 5-18.
Table 5-18 Supported upgrade features Part number 90Y4410 90Y4412 90Y4447 Feature code A2Y1 A2Y2 A36G Description Maximum supporte d 1 1 1
ServeRAID M5100 Series RAID 6 Upgrade for IBM Flex System ServeRAID M5100 Series Performance Accelerator for IBM Flex System (MegaRAID FastPath) ServeRAID M5100 Series SSD Caching Enabler for IBM Flex System (MegaRAID CacheCade Pro 2.0)
The following features are included: RAID 6 Upgrade (90Y4410) Adds support for RAID 6 and RAID 60. This license is an FoD license. Performance Accelerator (90Y4412) The Performance Accelerator for IBM Flex System, which is implemented by using the LSI MegaRAID FastPath software and provides high-performance I/O acceleration for SSD-based virtual drives. It uses a low-latency I/O path to increase the maximum IOPS capability of the controller. This feature boosts the performance of applications with a highly random data storage access pattern, such as transactional databases. Part number 90Y4412 is an FoD license. SSD Caching Enabler for traditional hard disk drives (90Y4447) The SSD Caching Enabler for IBM Flex System, which implemented by using the LSI MegaRAID CacheCade Pro 2.0 and is designed to accelerate the performance of HDD arrays. It can do so with only an incremental investment in SSD technology. The feature enables the SSDs to be configured as a dedicated cache to help maximize the I/O performance for transaction-intensive applications, such as databases and web serving. The feature tracks data storage access patterns and identifies the most frequently accessed data. The hot data is then automatically stored on the SSDs that are assigned as a dedicated cache pool on the ServeRAID controller. Part number 90Y4447 is a FoD license. This feature requires that at least one SSD drive is installed.
206
IBM PureFlex System and IBM Flex System Products and Technology
Description IBM 200GB SATA 1.8" MLC Enterprise SSD IBM 100GB SATA 1.8" MLC Enterprise SSD IBM 50 GB SATA 1.8-inch MLC SSD IBM 512 GB SATA 1.8-inch MLC Enterprise Value SSD IBM 64 GB SATA 1.8-inch MLC Enterprise Value SSD IBM 128GB SATA 1.8" MLC Enterprise Value SSD IBM 256GB SATA 1.8" MLC Enterprise Value SSD
Maximum supported 8 8 8 8 8 8 8
10K SAS hard disk drives 42D0637 49Y2003 81Y9650 00AD075 5599 5433 A282 A48S IBM 300 GB 10K 6 Gbps SAS 2.5-inch SFF Slim-HS HDD IBM 600 GB 10K 6 Gbps SAS 2.5-inch SFF Slim-HS HDD IBM 900 GB 10K 6 Gbps SAS 2.5-inch SFF HS HDD IBM 1.2TB 10K 6Gbps SAS 2.5'' G2HS HDD No No No No Supported Supported Supported Supported Supported Supported Supported Supported
15K SAS hard disk drives 42D0677 90Y8926 81Y9670 5536 A2XB A283 IBM 146 GB 15K 6 Gbps SAS 2.5-inch SFF Slim-HS HDD IBM 146GB 15K 6Gbps SAS 2.5" SFF G2HS HDD IBM 300 GB 15K 6 Gbps SAS 2.5-inch SFF HS HDD No No No Supported Supported Supported Supported Supported Supported
10K and 15K Self-encrypting drives (SED) 90Y8944 90Y8913 90Y8908 81Y9662 00AD085 A2ZK A2XF A3EF A3EG A48T IBM 146GB 15K 6Gbps SAS 2.5" SFF G2HS SED IBM 300GB 10K 6Gbps SAS 2.5" SFF G2HS SED IBM 600GB 10K 6Gbps SAS 2.5" SFF G2HS SED IBM 900GB 10K 6Gbps SAS 2.5" SFF G2HS SED IBM 1.2TB 10K 6Gbps SAS 2.5'' G2HS SED No No No No No Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported
SAS-SSD Hybrid drive 00AD102 A4G7 IBM 600GB 10K 6Gbps SAS 2.5'' G2HS Hybrid No Supported Supported
207
Part number NL SATA 81Y9722 81Y9726 81Y9730 NL SAS 42D0707 90Y8953 81Y9690
Feature code
IBM 250 GB 7.2 K 6 Gbps NL SATA 2.5-inch SFF HS HDD IBM 500 GB 7.2 K 6 Gbps NL SATA 2.5-inch SFF HS HDD IBM 1 TB 7.2 K 6 Gbps NL SATA 2.5-inch SFF HS HDD
IBM 500 GB 7200 6 Gbps NL SAS 2.5-inch SFF Slim-HS HDD IBM 500GB 7.2K 6Gbps NL SAS 2.5" SFF G2HS HDD IBM 1 TB 7.2 K 6 Gbps NL SAS 2.5-inch SFF HS HDD
No No No
Solid-state drives - Enterprise 41Y8331 41Y8336 41Y8341 00W1125 43W7718 49Y6129 49Y6134 49Y6139 49Y6195 A4FL A4FN A4FQ A3HR A2FN A3EW A3EY A3F0 A4GH S3700 200GB SATA 2.5" MLC HS Enterprise SSD S3700 400GB SATA 2.5" MLC HS Enterprise SSD S3700 800GB SATA 2.5" MLC HS Enterprise SSD IBM 100GB SATA 2.5" MLC HS Enterprise SSD IBM 200 GB SATA 2.5-inch MLC HS SSDa IBM 200GB SAS 2.5" MLC HS Enterprise SSD IBM 400GB SAS 2.5" MLC HS Enterprise SSD IBM 800GB SAS 2.5" MLC HS Enterprise SSD IBM 1.6TB SAS 2.5" MLC HS Enterprise SSD No No No No No No No No No Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported
Solid-state drives - Enterprise Value 49Y5839 90Y8648 90Y8643 49Y5844 A3AS A2U4 A2U3 A3AU IBM 64 GB SATA 2.5-inch MLC HS Enterprise Value SSD IBM 128GB SATA 2.5" MLC HS Enterprise Value SSD IBM 256GB SATA 2.5" MLC HS Enterprise Value SSD IBM 512 GB SATA 2.5-inch MLC HS Enterprise Value SSD No No No No Supported Supported Supported Supported Supported Supported Supported Supported
208
IBM PureFlex System and IBM Flex System Products and Technology
209
Figure 5-8 shows the rear of the x240 compute node and the locations of the I/O connectors.
I/O connector 1
I/O connector 2
Figure 5-8 Rear of the x220 compute node showing the locations of the I/O connectors
Table 5-21 lists the I/O adapters that are supported in the x220.
Table 5-21 Supported I/O adapters for the x220 compute node Part number Feature code Ports Description
Ethernet adapters 49Y7900 90Y3466 90Y3554 90Y3482 A10Y A1QY A1R1 A3HK 4 2 4 2 IBM Flex System EN2024 4-port 1Gb Ethernet Adapter IBM Flex System EN4132 2-port 10Gb Ethernet Adapter IBM Flex System CN4054 10Gb Virtual Fabric Adapter IBM Flex System EN6132 2-port 40Gb Ethernet Adapter
Fibre Channel adapters 69Y1938 95Y2375 88Y6370 95Y2386 95Y2391 69Y1942 A1BM A2N5 A1BP A45R A45S A1BQ 2 2 2 2 4 2 IBM Flex System FC3172 2-port 8Gb FC Adapter IBM Flex System FC3052 2-port 8Gb FC Adapter IBM Flex System FC5022 2-port 16Gb FC Adapter IBM Flex System FC5052 2-port 16Gb FC Adapter IBM Flex System FC5054 4-port 16Gb FC Adapter IBM Flex System FC5172 2-port 16Gb FC Adapter
InfiniBand adapters 90Y3454 A1QZ 2 IBM Flex System IB6132 2-port FDR InfiniBand Adapter
Consideration: Any supported I/O adapter can be installed in either I/O connector. However, you must be consistent across the chassis and all compute nodes.
210
IBM PureFlex System and IBM Flex System Products and Technology
a. The Blank USB Memory Key requires the download of the VMware vSphere (ESXi) Hypervisor with IBM Customization image, which is available at this website: http://ibm.com/systems/x/os/vmware/
There are two types of USB keys, preloaded keys or blank keys. Blank keys allow you to download an IBM customized version of ESXi and load it onto the key. The x240 supports one or two keys installed, but only certain combinations: The following combinations are supported: One preload key One blank key One preload key and one blank key Two blank keys Two preload keys is an unsupported combination. Installing two preloaded keys prevents ESXi from booting as described at this website: http://kb.vmware.com/kb/1035107 Having two keys installed provides a backup boot device. Both devices are listed in the boot menu, which allows you to boot from either device or to set one as a backup in case the first one gets corrupted.
211
Figure 5-9 shows the location of the LEDs and controls on the front of the x220.
Hard disk drive activity LED Hard disk drive status LED
USB port
Identify LED
Fault LED
NMI control
Figure 5-9 The front of the x220 with the front panel LEDs and controls shown
Location
Blue
Check error log Fault Hard disk drive activity LED Hard disk drive status LED
212
IBM PureFlex System and IBM Flex System Products and Technology
NMI
Power LED
The status of the power LED of the x220 shows the power status of the compute node. It also indicates the discovery status of the node by the Chassis Management Module. The power LED states are listed in Table 5-25.
Table 5-25 The power LED states of the x220 compute node Power LED state Off On; fast flash mode On; slow flash mode On; solid Status of compute node No power to compute node Compute node has power Chassis Management Module is in discovery mode (handshake) Compute node has power Power in stand-by mode Compute node has power Compute node is operational
Exception: The power button does not operate when the power LED is in fast flash mode.
213
The x220 light path diagnostics panel is visible when you remove the server from the chassis. The panel is on the upper right of the compute node as shown in Figure 5-10.
To illuminate the light path diagnostics LEDs, power off the compute node, slide it out of the chassis, and press the power button. The power button doubles as the light path diagnostics remind button when the server is removed from the chassis. The meaning of each LED in the light path diagnostics panel is listed in Table 5-26.
Table 5-26 Light path panel LED definitions LED LP S BRD MIS NMI TEMP MEM ADJ Color Green Yellow Yellow Yellow Yellow Yellow Yellow Meaning The light path diagnostics panel is operational System board error is detected A mismatch has occurred between the processors, DIMMs, or HDDs within the configuration as reported by POST An NMI has occurred An over-temperature condition has occurred that was critical enough to shut down the server A memory fault has occurred. The corresponding DIMM error LEDs on the system board should also be lit. A fault is detected in the adjacent expansion unit (if installed)
214
IBM PureFlex System and IBM Flex System Products and Technology
Remote access to system fan, voltage, and temperature values Remote IMM and UEFI update UEFI update when the server is powered off Remote console by way of a serial over LAN Remote access to the system event log Predictive failure analysis and integrated alerting features; for example, by using Simple Network Management Protocol (SNMP) Remote presence, including remote control of server by using a Java or Active x client Operating system failure window (blue screen) capture and display through the web interface Virtual media that allows the attachment of a diskette drive, CD/DVD drive, USB flash drive, or disk image to a server Remember: Unlike IBM BladeCenter, the assigned TCP/IP address of the IMM is available on the local network. You can use this address to remotely manage the x220 by connecting directly to the IMM independent of the IBM Flex System Manager or Chassis Management Module. For more information about the IMM, see 3.4.1, Integrated Management Module II on page 47.
215
5.3.1 Introduction
The IBM Flex System x222 Compute Node is a high-density offering that is designed to maximize the computing power that is available in the data center. With a balance between cost and system features, the x222 is an ideal platform for dense workloads, such as virtualization. This section describes the key features of the server. Figure 5-11 shows the front of the x222 Compute Node showing the location of the controls, LEDs, and connectors.
USB port
2.5 SS HDD bay Console breakout (or 2x 1.8 HS) cable port
Power
216
IBM PureFlex System and IBM Flex System Products and Technology
Figure 5-12 shows the internal layout and major components of the x222.
Upper system-board assembly Upper air baffle Fabric connector DIMM I/O expansion adapter
Microprocessor Heat sink filler Lower heat sink Lower air baffle Simple-swap hard disk drive
Figure 5-12 Exploded view of the x222, showing the major components
217
Components Memory
Specification Up to 24 DIMM sockets in a standard (half-width) Flex System form factor. Each separate server: Up to 12 DIMM sockets (six DIMMs per processor) by using Low Profile (LP) DDR3 DIMMs. RDIMMs and LRDIMMs are supported. 1.5 V and low-voltage 1.35 V DIMMs are supported. There is support for up to 1600 MHz memory speed, depending on the processor. There are three memory channels per processor (two DIMMs per channel). Supports two DIMMs per channel operating at 1600 MHz (two DPC @ 1600MHz) with single and dual rank RDIMMs. Each separate server: With LRDIMMs: Up to 384 GB with 12x 32 GB LRDIMMs and two processors With RDIMMs: Up to 192 GB with 12x 16 GB RDIMMs and two processors ECC, Chipkill, optional memory mirroring, and memory rank sparing. Each separate server: One 2.5" simple-swap SATA drive bay supporting SATA and SSD drives. Optional SSD mounting kit to convert a 2.5 simple-swap bay into two 1.8 hot-swap SSD bays. Each separate server: Up to 1 TB using a 2.5 SATA simple-swap drive or up to 512 GB using two 1.8 SSDs and the SSD Expansion Kit. None Each separate server: Two 10 Gb Ethernet ports with Embedded 10Gb Virtual Fabric Ethernet LAN on motherboard (LOM) controller; Emulex BE3 based. Routes to chassis bays 1 and 2 through a Fabric Connector to the midplane. Features on Demand upgrade to FCoE and iSCSI. Usage of both ports on both servers requires two scalable Ethernet switches in the chassis, each upgraded to enable 28 internal switch ports. Each separate server: One connector for an I/O adapter; PCI Express 3.0 x16 interface. Supports special mid-mezzanine I/O cards that are shared by both servers. Only one card is needed to connect both servers. Each separate server: One external, two internal USB ports for an embedded hypervisor. A console breakout cable port on the front of the server provides local KVM and serial ports (one cable is provided as standard with chassis; more cables optional). Each separate server: UEFI, IBM Integrated Management Module II (IMM2) with Renesas SH7757 controller, Predictive Failure Analysis, light path diagnostics panel, automatic server restart, and remote presence. Support for IBM Flex System Manager, IBM Systems Director, and IBM ServerGuide. Power-on password and admin password, Trusted Platform Module (TPM) 1.2. Each separate server: Matrox G200eR2 video core with 16 MB video memory that is integrated into IMM2. The maximum resolution is 1600x1200 at 75 Hz with 16 M colors. Three-year, customer-replaceable unit and onsite limited warranty with 9x5/NBD. Microsoft Windows Server 2008 R2 and 2012, Red Hat Enterprise Linux 5 and 6, SUSE Linux Enterprise Server 10 and 11, and VMware ESXi 4.1, 5.0 and 5.1. For more information, see 5.2.13, Operating system support on page 215. Optional country-specific service upgrades are available through IBM ServicePacs: 6, 4, or 2-hour response time, 8-hour fix time, 1-year or 2-year warranty extension, remote technical support for IBM hardware and selected IBM and OEM software. Width: 217 mm (8.6 in.), height: 56 mm (2.2 in.), depth: 492 mm (19.4 in.) Maximum configuration: 8.2 kg (18 lb)
Memory maximums
Memory protection Disk drive bays Maximum internal storage (Raw) RAID support Network interfaces
Ports
Systems management
Security features Video Limited warranty Operating systems supported Service and support
Dimensions Weight
218
IBM PureFlex System and IBM Flex System Products and Technology
5.3.2 Models
The current x222 models are shown in Table 5-28. All models include 2x 8 GB of memory (one 8 GB DIMM per server).
Table 5-28 Standard models Model Intel Xeon processors (2 max per server, 4 total) 2x E5-2418L 4C 2.0GHz 10MB 1333MHz 50W 2x E5-2430L 6C 2.0GHz 15MB 1333MHz 60W 2x E5-2450L 8C 1.8GHz 20MB 1600MHz 70W 2x E5-2403 4C 1.8GHz 10MB 1066MHz 80W 2x E5-2407 4C 2.2GHz 10MB 1066MHz 80W 2x E5-2420 6C 1.9GHz 15MB 1333MHz 95W 2x E5-2430 6C 2.2GHz 15MB 1333MHz 95W 2x E5-2430 6C 2.2GHz 15MB 1333MHz 95W 2x E5-2440 6C 2.4GHz 15MB 1333MHz 95W 2x E5-2450 8C 2.1GHz 20MB 1600MHz 95W 2x E5-2470 8C 2.3GHz 20MB 1600MHz 95W Disk adapter (1 per server) SATA (non-RAID) SATA (non-RAID) SATA (non-RAID) SATA (non-RAID) SATA (non-RAID) SATA (non-RAID) SATA (non-RAID) SATA (non-RAID) SATA (non-RAID) SATA (non-RAID) SATA (non-RAID) Disk bays (1 per server) 2x 2.5 SS 2x 2.5 SS 2x 2.5 SS 2x 2.5 SS 2x 2.5 SS 2x 2.5 SS 2x 2.5 SS 2x 2.5 SS 2x 2.5 SS 2x 2.5 SS 2x 2.5 SS Disks Networking (2 per server) 4x 10 GbE 4x 10 GbE 4x 10 GbE 4x 10 GbE 4x 10 GbE 4x 10 GbE 4x 10 GbE 4x 10 GbE 2x InfiniBanda 4x 10 GbE 4x 10 GbE 4x 10 GbE I/O slots (used/max) 0/1 0/1 0/1 0/1 0/1 0/1 0/1 1/1 0/1 0/1 0/1
7916-A2x 7916-B2x 7916-C2x 7916-D2x 7916-F2x 7916-G2x 7916-H2x 7916-H6x 7916-J2x 7916-M2x 7916-N2x
Open Open Open Open Open Open Open Open Open Open Open
a. Model H6x includes the IBM Flex System IB6132D 2-port FDR InfiniBand Adapter.
219
Up to 14 x222 Compute Nodes (up to 28x separate servers) can be installed in the chassis in 10U of rack space. The actual number of x222 systems that can be powered on in a chassis depends on the following factors: The TDP power rating for the processors that are installed in the x222. The number of power supplies installed in the chassis. The capacity of the power supplies installed in the chassis (2100 W or 2500 W). The power redundancy policy used in the chassis (N+1 or N+N). Table 4-11 on page 93 provides guidelines about the number of x222 systems that can be powered on in the IBM Flex System Enterprise Chassis, based on the type and number of power supplies that are installed.
I/O connector 2 on shared adapter (InfiniBand or FC) with connections top and bottom to each server
The two servers are located on the top and bottom halves of the x222 node.
I/O connector 1 shared with both servers for 10 GbE (2 ports for each server)
Each server within the IBM Flex System x222 Compute Node has the following system architecture features as standard: Two 1356-pin, Socket B2 (LGA-1356) processor sockets An Intel C600 series Platform Controller Hub Three memory channels per socket 220
IBM PureFlex System and IBM Flex System Products and Technology
Up to two DIMMs per memory channel 12 DDR3 DIMM sockets Support for RDIMMs and LRDIMMs One integrated 10 Gb Ethernet controller (10 GbE LOM in Figure 5-14) One IMM2 One connector for attaching to a mid-mezzanine I/O adapter One SATA connector for one 2.5 simple-swap SAS HDD or SSD (or two 1.8 SSDs with the optional 1.8 enablement kit) Two internal and one external USB connector Figure 5-14 shows the system architecture of the x222 system.
x4 ESI link Intel Xeon Processor 0 Intel C600 PCH USB DDR3 DIMMs 3 memory channels 2 DIMMs per channel QPI link (up to 8 GT/s) x1 USB
2.5 HDD or SSD (2x 1.8 with optional kit) 2x Internal USB Front USB Front KVM port Video & serial Management
IMM2
Chassis Midplane
PCIe 3.0 x8
Server interconnect
Mid-mezz Adapter
Fabric Connector
1Gb Switch
Management Connector
x1
USB USB
Video & serial Front KVM port Front USB 2x Internal USB 2.5 HDD or SSD (2x 1.8 with optional kit)
Figure 5-14 IBM Flex System x222 Compute Node block diagram Chapter 5. Compute nodes
221
Intel Xeon processors 00D1266 00D1265 00D1264 00D1263 00D1262 00D1261 00D1260 A35X / A370 A35W / A36Z A35V / A36Y A35U / A36X A35T / A36W A35S / A36V A35R / A36U Intel Xeon E5-2403 4C 1.8GHz 10MB 1066MHz 80W Intel Xeon E5-2407 4C 2.2GHz 10MB 1066MHz 80W Intel Xeon E5-2420 6C 1.9GHz 15MB 1333MHz 95W Intel Xeon E5-2430 6C 2.2GHz 15MB 1333MHz 95W Intel Xeon E5-2440 6C 2.4GHz 15MB 1333MHz 95W Intel Xeon E5-2450 8C 2.1GHz 20MB 1600MHz 95W Intel Xeon E5-2470 8C 2.3GHz 20MB 1600MHz 95W D2x F2x G2x H2x, H6x J2x M2x N2x
Intel Xeon processors - Low power 00D1269 00D1271 00D1268 00D1270 00D1267 A360 / A373 A362 / A375 A35Z / A372 A361 / A374 A35Y / A371 Intel Xeon E5-2418L 4C 2.0GHz 10MB 1333MHz 50W Intel Xeon E5-2428L 6C 1.8GHz 15MB 1333MHz 60W Intel Xeon E5-2430L 6C 2.0GHz 15MB 1333MHz 60W Intel Xeon E5-2448L 8C 1.8GHz 20MB 1333MHz 70W Intel Xeon E5-2450L 8C 1.8GHz 20MB 1600MHz 70W A2x B2x C2x
a. The first feature code is for processor 1 and the second feature code is for processor 2.
222
IBM PureFlex System and IBM Flex System Products and Technology
223
Table 5-31 Maximum memory speeds: RDIMMs Spec Rank Part numbers Single rank 49Y1406 (4 GB) 49Y1559 (4 GB) RDIMMs Dual rank 49Y1407 (4 GB) 49Y1397 (8 GB) 49Y1563 (16 GB) 1333 MHz 1.35 V 1.5 V 12 4 GB 48 GB 48 GB 1.35 V 12 16 GB 192 GB 129 GB 1.5 V 12 16 GB 192 GB 192 GB 90Y3178 (4 GB) 90Y3109 (8 GB) 00D4968(16GB) 1600 MHz 1.5 V 1.5 V 12 16 GB 192 GB 192 GB
Rated speed Rated voltage Operating voltage Max quantitya Largest DIMM Max memory capacity Max memory at rated speed
Maximum operating speed (MHz) 1 DIMM per channel 2 DIMMs per channel 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1600 MHz 1600 MHz 1333 MHz 1333 MHz 1333 MHz 1333 MHz 1600 MHz 1600 MHz
a. The maximum quantity that is supported is shown for two installed processors. When one processor is installed, the maximum quantity that is supported is half of that shown. Table 5-32 Maximum memory speeds - LRDIMMs Spec Rank Part numbers Rated speed Rated voltage Operating voltage Max quantitya 1.35 V 12 32 GB 384 GB N/A LRDIMMs Quad rank 90Y3105 (32 GB) 1333 MHz 1.35 V 1.5 V 12 32 GB 384 GB 192 GB
Largest DIMM Max memory capacity Max memory at rated speed Maximum operating speed (MHz) 1 DIMM per channel 2 DIMMs per channel
a. The maximum quantity that is supported is shown for two installed processors. When one processor is installed, the maximum quantity that is supported is half of that shown.
224
IBM PureFlex System and IBM Flex System Products and Technology
The following memory protection technologies are supported: ECC Chipkill (for x4-based memory DIMMs; look for x4 in the DIMM description) Memory mirroring Memory sparing If memory mirroring is used, the DIMMs must be installed in pairs (minimum of one pair per processor) and both DIMMs in a pair must be identical in type and size. If memory rank sparing is used, a minimum of one quad-rank DIMM or two single-rank or dual-rank DIMMs must be installed per populated channel (the DIMMs do not need to be identical). In rank sparing mode, one rank of a DIMM in each populated channel is reserved as spare memory. The size of a rank varies depending on the DIMMs that are installed. Table 5-33 lists the memory options that are available for the x222. DIMMs can be installed one at a time in each server, but for performance reasons, install them in sets of three (one for each of the three memory channels).
Table 5-33 Memory options for the x222 Part number Feature code Description Models where used
Registered DIMMs (RDIMMs) - 1333 MHz 49Y1406 49Y1407 49Y1397 49Y1563 8941 8942 8923 A1QT 4GB (1x4GB, 1Rx4, 1.35V) PC3L-10600 CL9 ECC DDR3 1333MHz LP RDIMM 4GB (1x4GB, 2Rx8, 1.35V) PC3L-10600 CL9 ECC DDR3 1333MHz LP RDIMM 8GB (1x8GB, 2Rx4, 1.35V) PC3L-10600 CL9 ECC DDR3 1333MHz LP RDIMM 16GB (1x16GB, 2Rx4, 1.35V) PC3L-10600 CL9 ECC DDR3 1333MHz LP RDIMM -
Registered DIMMs (RDIMMs) - 1600 MHz 49Y1559 90Y3178 90Y3109 00D4968 A28Z A24L A292 A2U5 4GB (1x4GB, 1Rx4, 1.5V) PC3-12800 CL11 ECC DDR3 1600MHz LP RDIMM 4GB (1x4GB, 2Rx8, 1.5V) PC3-12800 CL11 ECC DDR3 1600MHz LP RDIMM 8GB (1x8GB, 2Rx4, 1.5V) PC3-12800 CL11 ECC DDR3 1600MHz LP RDIMM 16GB (1x16GB, 2Rx4, 1.5V) PC3-12800 CL11 ECC DDR3 1600MHz LP RDIMM All -
Load-reduced DIMMs (LRDIMMs) 90Y3105 A291 32GB (1x32GB, 4Rx4, 1.35V) PC3L-10600 CL9 ECC DDR3 1333MHz LP LRDIMM -
225
1.8-inch drives and expansion kit 00W0366 00W1120 49Y6119 A3HV A3HQ A3AN IBM Flex System SSD Expansion Kit (used to convert the 2.5-inch bay in to two 1.8-inch bays) IBM 100GB SATA 1.8" MLC Enterprise SSD IBM 200GB SATA 1.8" MLC Enterprise SSD 1 2 2
2.5-inch drives 90Y8974 90Y8979 90Y8984 90Y8989 90Y8994 A369 A36A A36B A36C A36D IBM 500GB 7.2K 6Gbps SATA 2.5'' G2 SS HDD IBM 1TB 7.2K 6Gbps SATA 2.5'' G2 SS HDD IBM 128GB SATA 2.5 MLC Enterprise Value SSD for Flex System x222 IBM 256GB SATA 2.5 MLC Enterprise Value SSD for Flex System x222 IBM 100GB SATA 2.5 MLC Enterprise SSD for Flex System x222 1 1 1 1 1
a. The quantities that are listed here are for each of the separate servers within the x222 node.
226
IBM PureFlex System and IBM Flex System Products and Technology
Embedded 10 GbE
. . . . . .
. . .
Lower server
. . . . . .
. . .
Upper server traffic routes to Upgrade 1 ports on both switches Lower server traffic routes to Base ports on both switches
The following components are shown in Figure 5-15: The blue lines show that the two Ethernet ports in the upper server route to switches in bay 1 and bay 2. These connections require that the switch have Upgrade 1 enabled so as to enable the second bank of internal ports, ports 15 - 28. The red lines show that the two Ethernet ports in the lower server also route to switches in bay 1 and bay 2. These connections both go to the base ports of the switch, ports 1 - 14. Switch upgrade 1 required: You must have Upgrade 1 enabled in the two switches. Without this feature upgrade, the upper server does not have any Ethernet connectivity. For more information about supported Ethernet switches, see 4.11.4, Switch to adapter compatibility on page 115. The Embedded 10Gb VFA is based on the Emulex BladeEngine 3 (BE3), which is a single-chip, dual-port 10 Gigabit Ethernet (10GbE) Ethernet Controller. The Embedded 10Gb VFA includes the following features: PCI-Express Gen2 x8 host bus interface Supports multiple virtual NIC (vNIC) functions TCP/IP Offload Engine (TOE enabled) SR-IOV capable RDMA over TCP/IP capable iSCSI and FCoE upgrade offering through FoD Table 5-35 on page 228 lists the ordering information for the IBM Flex System Embedded 10Gb Virtual Fabric Upgrade, which enables the iSCSI and FCoE support on the Embedded 10Gb Virtual Fabric adapter. Two licenses required: To enable the FCoE/iSCSI upgrade for both servers in the x222 Compute Node, two licenses are required.
227
Table 5-35 Feature on Demand upgrade for FCoE and iSCSI support Part number 90Y9310 Feature code A2TD Description IBM Virtual Fabric Advanced Software Upgrade (LOM) Maximum supported per servera 1 per server 2 per x222 Compute Node
a. To enable the FCoE/iSCSI upgrade for both servers in the x222 Compute Node, two licenses are required.
Lower server
Midplane interface
Table 5-36 lists the supported adapters. Adapters are shared between the two servers with half of the ports routing to each server.
Table 5-36 Network adapters Part number 90Y3486 95Y2379 Feature code A365 A3HU Description IBM Flex System IB6132D 2-port FDR InfiniBand adapter IBM Flex System FC5024D 4-port 16Gb FC adapter Number of ports 2 4 Maximum supporteda 1 1
a. One adapter is supported per x222 Compute Node. The adapter is shared between the two servers within the x222.
228
IBM PureFlex System and IBM Flex System Products and Technology
A compatible I/O module must be installed in the corresponding I/O bays in the chassis, as shown in the Table 5-37.
Table 5-37 Adapter to I/O bay correspondence Upper or Lower server Adapter With FC5024D 4-port With IB6132D 2-port Corresponding I/O module bay in the chassis Module bay 1 (Upgrade 1a) Module bay 2 (Upgrade 1a ) Module bay 1 (Base) Module bay 2 (Base) Module bay 3 Module bay 4 Module bay 3 Module bay 4
Upper server
Embedded 10 GbE Virtual Fabric adapter Embedded 10 GbE Virtual Fabric adapter Port 1
Port 1 Port 2 Port 1 Port 2 Not used Port 1 Port 1 Not used
Lower server
Upper server
Lower server
I/O Expansion adapter FC5024D 4-port 16Gb FC or IB6132D 2-port FDR InfiniBand
a. Requires a scalable switch with 28 or more internal ports enabled. For the EN2092, EN4093, EN4093R, and SI4093 switches, this means Upgrade 1 is required. For the CN4093, Upgrade 1 or Upgrade 2 is required.
For more information about the supported switches, see 4.11.4, Switch to adapter compatibility on page 115. The FC5024D is a four-port adapter where two ports are routed to each server. Port 1 of each server is connected to the switch in bay 3 and Port 2 of each server is connected to the switch in bay 4. To make full use of all four ports, you must install a supported Fibre Channel switch in both switch bays.
229
Figure 5-17 shows how the FC5024D 4-port 16 Gb FC adapter and the Embedded 10Gb VFAs are connected to the Ethernet and Fibre Channel switches installed in the chassis.
Base switch ports . . . . . . Upgrade 1 switch ports Switch bay 1 Ethernet
Embedded 10 GbE
. . .
Switch bay 3 FC
. . .
Lower server
. . . . . .
. . .
Ethernet traffic upper server Ethernet traffic lower server Fibre Channel traffic PCIe traffic
. . .
Switch bay 4 FC
. . .
Figure 5-17 Logical layout of the interconnects: Ethernet and Fibre Channel
The FC5024D 4-port 16Gb FC Adapter is supported by the following switches: IBM Flex System FC5022 16Gb SAN Scalable Switch IBM Flex System FC5022 24-port 16Gb SAN Scalable Switch IBM Flex System FC5022 24-port 16Gb ESB SAN Scalable Switch Fibre Channel switch ports: The Fibre Channel switches in bays 3 and 4 use Ports on Demand to enable both internal and external ports. You should ensure that enough ports are licensed to activate all internal ports and all needed external ports. For more information, see 4.11.11, IBM Flex System FC5022 16Gb SAN Scalable Switch on page 148. For more information about this adapter, see 5.11.15, IBM Flex System FC5024D 4-port 16Gb FC Adapter on page 394. The IB6132D is a two-port adapter and has one port that is routed to each server. One port of the adapter connects to the InfiniBand switch in switch bay 3 and the other adapter port connects to the InfiniBand switch in switch bay 4 in the chassis. The IB6132D requires that two InfiniBand switches be installed in the chassis.
230
IBM PureFlex System and IBM Flex System Products and Technology
Figure 5-18 shows how the IB6132D 2-port FDR InfiniBand adapter and the four ports of the two Embedded 10 GbE VFAs are connected to the Ethernet and InfiniBand switches that are installed in the chassis.
Base switch ports . . . . . . Upgrade 1 switch ports Switch bay 1 Ethernet
Embedded 10 GbE
. . .
. . .
Lower server
. . . . . .
. . .
Ethernet traffic upper server Ethernet traffic lower server InfiniBand traffic PCIe traffic
. . .
. . .
The IB6132D 2-port FDR InfiniBand Adapter is supported by the IBM Flex System IB6131 InfiniBand Switch. To use the adapter at FDR speeds, the switch needs the FDR upgrade. For more information, see 4.11.14, IBM Flex System IB6131 InfiniBand Switch on page 160. For more information about this adapter, see 5.11.20, IBM Flex System IB6132D 2-port FDR InfiniBand Adapter on page 403.
231
a. The Blank USB Memory Key requires the download of the VMware vSphere (ESXi) Hypervisor with IBM Customization image, which is available at this website: http://ibm.com/systems/x/os/vmware/
There are two types of USB keys: preload keys or blank keys. Blank keys allow you to download an IBM customized version of ESXi and load it onto the key. Each server supports one or two keys to be installed, but only in the following combinations: One preload key (keys that are preloaded at the factory) One blank key (a key that you download the customized image) One preload key and one blank key Two blank keys Two preload keys is an unsupported combination. Installing two preload keys prevents ESXi from booting. This is similar to the error as described at this website: http://kb.vmware.com/kb/1035107 Having two keys that are installed provides a backup boot device. Both devices are listed in the boot menu, which allows you to boot from either device or to set one as a backup in case the first one becomes corrupted.
Remote management
A virtual presence capability comes standard for remote server management. Remote server management is provided through the following industry-standard interfaces: Intelligent Platform Management Interface (IPMI) Version 2.0 SNMP Version 3 Common Information Model (CIM) Web browser The server supports virtual media and remote control features, which provide the following functions: Remotely viewing video with graphics resolutions up to 1600 x 1200 at 75 Hz with up to 23 bits per pixel, regardless of the system state. Remotely accessing the server by using the keyboard and mouse from a remote client. Mapping the CD or DVD drive, diskette drive, and USB flash drive on a remote client, and mapping ISO and diskette image files as virtual drives that are available for use by the server. Uploading a diskette image to the IMM2 memory and mapping it to the server as a virtual drive. Capturing blue-screen errors.
232
IBM PureFlex System and IBM Flex System Products and Technology
To illuminate the light path diagnostics LEDs, power off the compute node, slide it out of the chassis, and press the power button on the specific server showing the error. The power button on each server doubles as the light path diagnostics remind button when the server is removed from the chassis.
233
The meanings of the LEDs in the light path diagnostics panel are listed in Table 5-39.
Table 5-39 Light path diagnostic panel LEDs LED LP S BRD MIS NMI TEMP MEM Meaning The light path diagnostics panel is operational. A system board error is detected. A mismatch has occurred between the processors, DIMMs, or HDDs within the configuration (as reported by POST). A non-maskable interrupt (NMI) has occurred. An over-temperature condition occurs that was critical enough to shut down the server. A memory fault has occurred. The corresponding DIMM error LEDs on the system board are also lit.
234
IBM PureFlex System and IBM Flex System Products and Technology
5.4.4, Chassis support on page 239 5.4.5, System architecture on page 240 5.4.6, Processor on page 242 5.4.7, Memory on page 245 5.4.8, Standard onboard features on page 258 5.4.9, Local storage on page 259 5.4.10, Integrated virtualization on page 266 5.4.11, Embedded 10 Gb Virtual Fabric adapter on page 268 5.4.12, I/O expansion on page 269 5.4.13, Systems management on page 271 5.4.14, Operating system support on page 274
5.4.1 Introduction
The x240 supports the following equipment: Up to two Intel Xeon E5-2600 series multi-core processors Twenty-four memory DIMMs Two hot-swap drives Two PCI Express I/O adapters Two optional internal USB connectors Figure 5-20 shows the x240.
235
Figure 5-21 shows the location of the controls, LEDs, and connectors on the front of the x240.
Hard disk drive activity LED Hard disk drive status LED
USB port
NMI control
LED panel
Figure 5-21 The front of the x240 showing the location of the controls, LEDs, and connectors
Figure 5-22 shows the internal layout and major components of the x240.
Cover Heat sink Microprocessor heat sink filler I/O expansion adapter Air baffle
Microprocessor Hot-swap storage backplane Hot-swap storage cage Hot-swap storage drive
Air baffle
Figure 5-22 Exploded view of the x240 showing the major components
236
IBM PureFlex System and IBM Flex System Products and Technology
Chipset Memory
Memory maximums
RAID support
Network interfaces
237
Specification UEFI, IBM Integrated Management Module II (IMM2) with Renesas SH7757 controller, Predictive Failure Analysis, light path diagnostics panel, automatic server restart, remote presence. Support for IBM Flex System Manager, IBM Systems Director, and IBM ServerGuide. Power-on password, administrator's password, Trusted Platform Module 1.2 Matrox G200eR2 video core with 16 MB video memory that is integrated into the IMM2. Maximum resolution is 1600x1200 at 75 Hz with 16 M colors. 3-year customer-replaceable unit and onsite limited warranty with 9x5/NBD Microsoft Windows Server 2008 R2, Red Hat Enterprise Linux 5 and 6, SUSE Linux Enterprise Server 10 and 11, VMware vSphere. For more information, see 5.4.14, Operating system support on page 274. Optional service upgrades are available through IBM ServicePacs: 4-hour or 2-hour response time, 8 hours fix time, 1-year or 2-year warranty extension, and remote technical support for IBM hardware and selected IBM and OEM software. Width 215 mm (8.5), height 51 mm (2.0), depth 493 mm (19.4) Maximum configuration: 6.98 kg (15.4 lb)
Security features Video Limited warranty Operating systems supported Service and support
Dimensions Weight
Figure 5-23 shows the components on the system board of the x240.
Hot-swap drive bay backplane Processor 2 and 12 memory DIMMs I/O connector 1 Fabric Connector
I/O connector 2
Expansion Connector
238
IBM PureFlex System and IBM Flex System Products and Technology
5.4.3 Models
The current x240 models are shown in Table 5-41. All models include 8 GB of memory (2x 4 GB DIMMs) running at either 1600 MHz or 1333 MHz (depending on the model).
Table 5-41 Models of the x240 type 8737 Modelsa 8737-A1x 8737-D2x 8737-F2x 8737-G2x 8737-H1x 8737-H2x 8737-J1x 8737-L2x 8737-M1x 8737-M2x 8737-N2x 8737-Q2x 8737-R2x Intel processor (model, cores, core speed, L3 cache, memory speed, TDP power) (two max) 1x Xeon E5-2630L 6C 2.0 GHz 15 MB 1333 MHz 60 W 1x Xeon E5-2609 4C 2.40 GHz 10 MB 1066 MHz 80 W 1x Xeon E5-2620 6C 2.0 GHz 15 MB 1333 MHz 95 W 1x Xeon E5-2630 6C 2.3 GHz 15 MB 1333 MHz 95 W 1x Xeon E5-2640 6C 2.5 GHz 15 MB 1333 MHz 95 W 1x Xeon E5-2640 6C 2.5 GHz 15 MB 1333 MHz 95 W 1x Xeon E5-2670 8C 2.6 GHz 20 MB 1600 MHz 115 W 1x Xeon E5-2660 8C 2.2 GHz 20 MB 1600 MHz 95 W 1x Xeon E5-2680 8C 2.7 GHz 20 MB 1600 MHz 130 W 1x Xeon E5-2680 8C 2.7 GHz 20 MB 1600 MHz 130 W 1x Xeon E5-2643 4C 3.3 GHz 10 MB 1600 MHz 130 W 1x Xeon E5-2667 6C 2.9 GHz 15 MB 1600 MHz 130 W 1x Xeon E5-2690 8C 2.9 GHz 20 MB 1600 MHz 135 W Standard memoryb 2x 4 GB 2x 4 GB 2x 4 GB 2x 4 GB 2x 4 GB 2x 4 GB 2x 4 GB 2x 4 GB 2x 4 GB 2x 4 GB 2x 4 GB 2x 4 GB 2x 4 GB Available drive bays Two (open) Two (open) Two (open) Two (open) Two (open) Two (open) Two (open) Two (open) Two (open) Two (open) Two (open) Two (open) Two (open) Available I/O slotsc 2 1 1 1 2 1 2 1 2 1 1 1 1 10 GbE embedd No Yes Yes Yes No Yes No Yes No Yes Yes Yes Yes
a. The model numbers that are provided are worldwide generally available variant (GAV) model numbers that are not orderable as listed. They must be modified by country. The US GAV model numbers use the following nomenclature: xxU. For example, the US orderable part number for 8737-A2x is 8737-A2U. See the product-specific official IBM announcement letter for other country-specific GAV model numbers. b. The maximum system memory capacity is 768 GB when you use 24x 32 GB DIMMs. c. Some models include an Embedded 10 Gb Virtual Fabric Ethernet LOM controller as standard. This embedded controller precludes the use of an I/O adapter in I/O connector 1, as shown in Figure 5-23 on page 238. For more information, see 5.4.11, Embedded 10 Gb Virtual Fabric adapter on page 268. d. Models number in the form x2x (for example, 8737-L2x) include an Embedded 10 Gb Virtual Fabric Ethernet LOM controller as standard. Model numbers in the form x1x (for example 8737-A1x) do not include this embedded controller.
239
Up to 14 x240 Compute Nodes can be installed in the chassis in 10U of rack space. The actual number of x240 systems that can be powered on in a chassis depends on the following factors: The TDP power rating for the processors that are installed in the x240. The number of power supplies installed in the chassis. The capacity of the power supplies installed in the chassis (2100 W or 2500 W). The power redundancy policy used in the chassis (N+1 or N+N). Table 4-11 on page 93 provides guidelines about the number of x240 systems that can be powered on in the IBM Flex System Enterprise Chassis, based on the type and number of power supplies installed. The x240 is a half-wide compute node. The chassis shelf must be installed in the IBM Flex System Enterprise Chassis. Figure 5-24 shows the chassis shelf in the chassis.
Figure 5-24 The IBM Flex System Enterprise Chassis showing the chassis shelf
The shelf is required for half-wide compute nodes. To install the full-wide or larger, shelves must be removed from within the chassis. Slide the two latches on the shelf towards the center and then slide the shelf from the chassis.
240
IBM PureFlex System and IBM Flex System Products and Technology
The Xeon E5-2600 series processor implements the second generation of Intel Core microarchitecture (Sandy Bridge) by using a 32 nm manufacturing process. It requires a new socket type, the LGA-2011, which has 2011 pins that touch contact points on the underside of the processor. The architecture also includes the Intel C600 (Patsburg B) Platform Controller Hub (PCH). Figure 5-25 shows the system architecture of the x240 system.
PCIe x4 G2
USB
DDR3 DIMMs 4 memory channels 3 DIMMs per channel QPI links (8 GT/s)
x1
USB
IMM v2
PCIe x16 G3 I/O connector 1 Intel Xeon Processor 2 PCIe x8 G3 PCIe x8 G3 PCIe x16 G3 I/O connector 2 PCIe x16 G3 Sidecar connector
Figure 5-25 IBM Flex System x240 Compute Node system board block diagram
The IBM Flex System x240 Compute Node has the following system architecture features as standard: Two 2011-pin type R (LGA-2011) processor sockets An Intel C600 PCH Four memory channels per socket Up to three DIMMs per memory channel Twenty-four DDR3 DIMM sockets Support for UDIMMs, RDIMMs, and new LRDIMMs One integrated 10 Gb Virtual Fabric Ethernet controller (10 GbE LOM in diagram) One LSI 2004 SAS controller Integrated HW RAID 0 and 1 One Integrated Management Module II Two PCIe x16 Gen3 I/O adapter connectors Two Trusted Platform Module (TPM) 1.2 controllers One internal USB connector
241
The new architecture allows the sharing of data on-chip through a high-speed ring interconnect between all processor cores, the last level cache (LLC), and the system agent. The system agent houses the memory controller and a PCI Express root complex that provides 40 PCIe 3.0 lanes. This ring interconnect and LLC architecture are shown in Figure 5-26.
Core Core
L1/L2 L1/L2
.
Core L1/L2 LLC
to Chipset
System agent
PCIe 3.0 Root Complex 40 lanes PCIe 3.0 Memory Controller
QPI link
The two Xeon E5-2600 series processors in the x240 are connected through two QuickPath Interconnect (QPI) links. Each QPI link is capable of up to eight giga-transfers per second (GTps) depending on the processor model installed. Table 5-43 shows the QPI bandwidth of the Intel Xeon E5-2600 series processors.
Table 5-43 QuickPath Interconnect bandwidth Intel Xeon E5-2600 series processor Advanced Standard Basic QuickPath Interconnect speed (GTps) 8.0 GTps 7.25 GTps 6.4 GTps QuickPath Interconnect bandwidth (GBps) in each direction 32.0 GBps 29.0 GBps 25.6 GBps
5.4.6 Processor
The Intel Xeon E5-2600 series is available with up to eight cores and 20 MB of last-level cache. It features an enhanced instruction set called Intel Advanced Vector Extension (AVX). This set doubles the operand size for vector instructions (such as floating-point) to 256 bits and boosts selected applications by up to a factor of two. The new architecture also introduces Intel Turbo Boost Technology 2.0 and improved power management capabilities. Turbo Boost automatically turns off unused processor cores and increases the clock speed of the cores in use if thermal requirements are still met. Turbo Boost Technology 2.0 makes use of the new integrated design. It also implements a more granular overclocking in 100 MHz steps instead of 133 MHz steps on former Nehalem-based and Westmere-based microprocessors.
242
IBM PureFlex System and IBM Flex System Products and Technology
As listed in Table 5-41 on page 239, standard models come with one processor that is installed in processor socket 1. In a two processor system, both processors communicate with each other through two QPI links. I/O is served through 40 PCIe Gen2 lanes and through a x4 Direct Media Interface (DMI) link to the Intel C600 PCH. Processor 1 has direct access to 12 DIMM slots. By adding the second processor, you enable access to the remaining 12 DIMM slots. The second processor also enables access to the sidecar connector, which enables the use of mezzanine expansion units. Table 5-44 show a comparison between the features of the Intel Xeon 5600 series processor and the new Intel Xeon E5-2600 series processor that is installed in the x240.
Table 5-44 Comparison of Xeon 5600 series and Xeon E5-2600 series processor features Specification Cores Physical Addressing Cache size Memory channels per socket Max memory speed Virtualization technology New instructions QPI frequency Inter-socket QPI links PCI Express Xeon 5600 Up to six cores / 12 threads 40-bit (Uncorea limited) 12 MB 3 1333 MHz Real Mode support and transition latency reduction AES-NI 6.4 GTps 1 36 Lanes PCIe on chipset Xeon E5-2600 Up to eight cores / 16 threads 46-bit (Core and Uncorea ) Up to 20 MB 4 1600 MHz Adds Large VT pages Adds AVX 8.0 GTps 2 40 Lanes/Socket Integrated PCIe
a. Uncore is an Intel term that is used by Intel to describe the parts of a processor that are not the core.
Table 5-45 lists the features for the different Intel Xeon E5-2600 series processor types.
Table 5-45 Intel Xeon E5-2600 series processor features Processor model Advanced Xeon E5-2650 Xeon E5-2658 Xeon E5-2660 Xeon E5-2665 Xeon E5-2670 Xeon E5-2680 Xeon E5-2690 2.0 GHz 2.1 GHz 2.2 GHz 2.4 GHz 2.6 GHz 2.7 GHz 2.9 GHz Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 20 MB 20 MB 20 MB 20 MB 20 MB 20 MB 20 MB 8 8 8 8 8 8 8 95 W 95 W 95 W 115 W 115 W 130 W 135 W 8 GT/s 8 GT/s 8 GT/s 8 GT/s 8 GT/s 8 GT/s 8 GT/s 1600 MHz 1600 MHz 1600 MHz 1600 MHz 1600 MHz 1600 MHz 1600 MHz Processor frequency Turbo HT L3 cache Cores Power TDP QPI Link speeda Max DDR3 speed
243
Processor model Standard Xeon E5-2620 Xeon E5-2630 Xeon E5-2640 Basic Xeon E5-2603 Xeon E5-2609 Low power Xeon E5-2650L Xeon E5-2648L Xeon E5-2630L Special Purpose Xeon E5-2667 Xeon E5-2643 Xeon E5-2637
Processor frequency
Turbo
HT
L3 cache
Cores
Power TDP
15 MB 15 MB 15 MB
6 6 6
95 W 95 W 95 W
No No
No No
10 MB 10 MB
4 4
80 W 80 W
20 MB 20 MB 15 MB
8 8 6
70 W 70 W 60 W
Yes No No
Yes No No
15 MB 10 MB 5 MB
6 4 2
130 W 130 W 80 W
244
IBM PureFlex System and IBM Flex System Products and Technology
Description Intel Xeon Processor E5-2667 6C 2.9 GHz 15 MB Cache 1600 MHz 130 W Intel Xeon Processor E5-2670 8C 2.6 GHz 20 MB Cache 1600 MHz 115 W Intel Xeon Processor E5-2680 8C 2.7 GHz 20 MB Cache 1600 MHz 130 W Intel Xeon Processor E5-2690 8C 2.9 GHz 20 MB Cache 1600 MHz 135 W
For more information about the Intel Xeon E5-2600 series processors, see this website: http://intel.com/content/www/us/en/processors/xeon/xeon-processor-5000-sequence.html
5.4.7 Memory
The x240 has 12 DIMM sockets per processor (24 DIMMs in total) running at 800, 1066, 1333, or 1600 MHz. It supports 2 GB, 4 GB, 8 GB, 16 GB, and 32 GB memory modules, as shown in Table 5-49 on page 250. The x240 with the Intel Xeon E5-2600 series processors can support up to 768 GB of memory in total when you use 32 GB LRDIMMs with both processors installed. The x240 uses double data rate type 3 (DDR3) LP DIMMs. You can use registered DIMMs (RDIMMs), unbuffered DIMMs (UDIMMs), or load-reduced DIMMs (LRDIMMs). However, the mixing of the different memory DIMM types is not supported. The E5-2600 series processor has four memory channels, and each memory channel can have up to three DIMMs. Figure 5-27 shows the E5-2600 series and the four memory channels.
Channel 2
Channel 3
DIMM 10
Channel 0
Channel 1
DIMM 6
DIMM 3
DIMM 4
Figure 5-27 The Intel Xeon E5-2600 series processor and the four memory channels
DIMM 5
DIMM 2
DIMM 1
DIMM 11
DIMM 12
DIMM 9
DIMM 8
DIMM 7
245
Memory subsystem characteristic Maximum number of DIMMs per channel (DPC) DIMM slot maximum Mixing of memory types (RDIMMS, UDIMMS, LRDIMMs) Mixing of memory speeds Mixing of DIMM voltage ratings Registered DIMM (RDIMM) modules Supported memory sizes Supported memory speeds Maximum system capacity Maximum memory speed
IBM Flex System x240 Compute Node 3 (using 1.5V DIMMs) 2 (using 1.35V DIMMs) One-processor: 12 Two-processor: 24 Not supported in any configuration Supported; lowest common speed for all installed DIMMs Supported; all 1.35 V will run at 1.5 V
32, 16, 8, 4, and 2 GB 1600, 1333, 1066, and 800 MHz 512 GB (16 x 16 GB) 1.35V @ 2DPC: 1333 MHz 1.5V @ 2DPC: 1600 MHz 1.5V @ 3DPC: 1066 MHz 8 One-processor: 12 Two-processor: 24
Maximum ranks per channel (any memory voltage) Maximum number of DIMMs Unbuffered DIMM (UDIMM) modules Supported memory sizes Supported memory speeds Maximum system capacity Maximum memory speed
4 GB 1333 MHz 64 GB (16 x 4 GB) 1.35V @ 2DPC: 1333 MHz 1.5V @ 2DPC: 1333 MHz 1.35V or 1.5V @ 3DPC: Not supported 8 One-processor: 8 Two-processor: 16
Maximum ranks per channel (any memory voltage) Maximum number of DIMMs Load-reduced (LRDIMM) modules Supported sizes Maximum capacity Supported speeds Maximum memory speed
32 and 16 GB 768 GB (24 x 32 GB) 1333 and 1066 MHz 1.35V @ 2DPC: 1066 MHz 1.5V @ 2DPC: 1333 MHz 1.35V or 1.5V @ 3DPC: 1066 MHz 8a
246
IBM PureFlex System and IBM Flex System Products and Technology
a. Because of reduced electrical loading, a 4R (four-rank) LRDIMM has the equivalent load of a two-rank RDIMM. This reduced load allows the x240 to support three 4R LRDIMMs per channel (instead of two as with UDIMMs and RDIMMs). For more information, see on page 247.
Figure 5-28 shows the location of the 24 memory DIMM sockets on the x240 system board and other components.
DIMMs 13-18 Microprocessor 2 DIMMs 1-6 I/O expansion 1 LOM connector (some models only)
I/O expansion 2
DIMMs 19-24
DIMMs 7-12
Microprocessor 1
Table 5-48 lists which DIMM connectors belong to which processor memory channel.
Table 5-48 The DIMM connectors for each processor memory channel Processor Memory channel Channel 0 Channel 1 Processor 1 Channel 2 Channel 3 Channel 0 Channel 1 Processor 2 Channel 2 Channel 3 13, 14, and 15 16, 17, and 18 7, 8, and 9 10, 11, and 12 22, 23, and 24 19, 20, and 21 DIMM connector 4, 5, and 6 1, 2, and 3
247
Memory types
The x240 supports the following types of DIMM memory: RDIMM modules Registered DIMMs are the mainstream module solution for servers or any applications that demand heavy data throughput, high density, and high reliability. RDIMMs use registers to isolate the memory controller address, command, and clock signals from the dynamic random-access memory (DRAM). This process results in a lighter electrical load. Therefore, more DIMMs can be interconnected and larger memory capacity is possible. However, the register often does impose a clock or more of delay, meaning that registered DIMMs often have slightly longer access times than their unbuffered counterparts. In general, RDIMMs have the best balance of capacity, reliability, and workload performance with a maximum performance of 1600 MHz (at 2 DPC). For more information about supported x240 RDIMM memory options, see Table 5-49 on page 250. UDIMM modules In contrast to RDIMMs that use registers to isolate the memory controller from the DRAMs, UDIMMs attach directly to the memory controller. Therefore, they do not introduce a delay, which creates better performance. The disadvantage is limited drive capability. Limited capacity means that the number of DIMMs that can be connected together on the same memory channel remains small because of electrical loading. This leads to less DIMM support, fewer DIMMs per channel (DPC), and overall lower total system memory capacity than RDIMM systems. UDIMMs have the lowest latency and lowest power usage. They also have the lowest overall capacity. For more information about supported x240 UDIMM memory options, see Table 5-49 on page 250. LRDIMM modules Load-reduced DIMMs are similar to RDIMMs. They also use memory buffers to isolate the memory controller address, command, and clock signals from the individual DRAMS on the DIMM. Load-reduced DIMMs take the buffering a step further by buffering the memory controller data lines from the DRAMs as well.
248
IBM PureFlex System and IBM Flex System Products and Technology
Load-reduced DIMM
DRAM DRAM DRAM DRAM
Memory controller
Register
CMD/ADDR/ CLK
Memory controller
DRAM DRAM
DATA
DRAM DRAM
DRAM DRAM
In essence, all signaling between the memory controller and the LRDIMM is now intercepted by the memory buffers on the LRDIMM module. This system allows more ranks to be added to each LRDIMM module without sacrificing signal integrity. It also means that fewer actual ranks are seen by the memory controller (for example, a 4R LRDIMM has the same look as a 2R RDIMM). The added buffering that the LRDIMMs support greatly reduces the electrical load on the system. This reduction allows the system to operate at a higher overall memory speed for a certain capacity. Conversely, it can operate at a higher overall memory capacity at a certain memory speed. LRDIMMs allow maximum system memory capacity and the highest performance for system memory capacities above 384 GB. They are suited for system workloads that require maximum memory such as virtualization and databases. For more information about supported x240 LRDIMM memory options, see Table 5-49 on page 250. The memory type that is installed in the x240 combines with other factors to determine the ultimate performance of the x240 memory subsystem. For a list of rules when populating the memory subsystem, see Memory installation considerations on page 257.
249
Memory options
Table 5-49 lists the memory DIMM options for the x240.
Table 5-49 Memory DIMMs for the x240 Part number FC Description Where used
Registered DIMM (RDIMM) modules - 1066 MHz and 1333 MHz 49Y1405 49Y1406 8940 8941 2 GB (1x2GB, 1Rx8, 1.35V) PC3L-10600 CL9 ECC DDR3 1333 MHz LP RDIMM 4 GB (1x4GB, 1Rx4, 1.35V) PC3L-10600 CL9 ECC DDR3 1333 MHz LP RDIMM H1x, H2x, G2x, F2x, D2x, A1x
4 GB (1x4GB, 2Rx8, 1.35V) PC3L-10600 CL9 ECC DDR3 1333 MHz LP RDIMM 8 GB (1x8GB, 2Rx4, 1.35V) PC3L-10600 CL9 ECC DDR3 1333 MHz LP RDIMM 16 GB (1x16GB, 2Rx4, 1.35V) PC3L-10600 CL9 ECC DDR3 1333 MHz LP RDIMM 16 GB (1x16GB, 4Rx4, 1.35V) PC3L-8500 CL7 ECC DDR3 1066 MHz LP RDIMM
Registered DIMM (RDIMM) modules - 1600 MHz 49Y1559 A28Z 4 GB (1x4GB, 1Rx4, 1.5V) PC3-12800 CL11 ECC DDR3 1600 MHz LP RDIMM R2x, Q2x, N2x, M2x, M1x, L2x, J1x
4 GB (1x4GB, 2Rx8, 1.5V) PC3-12800 CL11 ECC DDR3 1600 MHz LP RDIMM 8 GB (1x8GB, 2Rx4, 1.5V) PC3-12800 CL11 ECC DDR3 1600 MHz LP RDIMM 16 GB (1x16GB, 2Rx4, 1.5V) PC3-12800 CL11 ECC DDR3 1600 MHz LP RDIMM
Unbuffered DIMM (UDIMM) modules 49Y1404 8648 4 GB (1x4GB, 2Rx8, 1.35V) PC3L-10600 CL9 ECC DDR3 1333 MHz LP UDIMM
Load-reduced (LRDIMM) modules 49Y1567 90Y3105 A290 A291 16 GB (1x16GB, 4Rx4, 1.35V) PC3L-10600 CL9 ECC DDR3 1333 MHz LP LRDIMM 32 GB (1x32GB, 4Rx4, 1.35V) PC3L-10600 CL9 ECC DDR3 1333 MHz LP LRDIMM
250
IBM PureFlex System and IBM Flex System Products and Technology
Speed of DDR3 DIMMs installed For maximum performance, the speed rating of each DIMM module must match the maximum memory clock speed of the Xeon E5-2600 processor. Remember the following rules when you match processors and DIMM modules: The processor never over-clocks the memory in any configuration. The processor clocks all the installed memory at either the rated speed of the processor or the speed of the slowest DIMM installed in the system. For example, an Intel Xeon E5-2640 series processor clocks all installed memory at a maximum speed of 1333 MHz. If any 1600 MHz DIMM modules are installed, they are clocked at 1333 MHz. However, if any 1066 MHz or 800 MHz DIMM modules are installed, all installed DIMM modules are clocked at the slowest speed (800 MHz). Number of DIMMs per channel (DPC) Generally, the Xeon E5-2600 processor series clocks up to 2DPC at the maximum rated speed of the processor. However, if any channel is fully populated (3DPC), the processor slows all the installed memory down. For example, an Intel Xeon E5-2690 series processor clocks all installed memory at a maximum speed of 1600 MHz up to 2DPC. However, if any one channel is populated with 3DPC, all memory channels are clocked at 1066 MHz. DIMM voltage rating The Xeon E5-2600 processor series supports both low voltage (1.35 V) and standard voltage (1.5 V) DIMMs. Table 5-49 on page 250 shows that the maximum clock speed for supported low voltage DIMMs is 1333 MHz. The maximum clock speed for supported standard voltage DIMMs is 1600 MHz. Table 5-50 and Table 5-51 on page 252 list the memory DIMM types that are available for the x240 and shows the maximum memory speed, which is based on the number of DIMMs per channel, ranks per DIMM, and DIMM voltage rating.
Table 5-50 Maximum memory speeds (Part 1 - UDIMMs, LRDIMMs and Quad rank RDIMMs) Spec Rank Part numbers Rated speed Rated voltage Operating voltage Maximum quantitya Largest DIMM Max memory capacity Max memory at rated speed Maximum operating speed 1 DIMM per channel 2 DIMMs per channel 3 DIMMs per channel 1333 MHz 1333 MHz NSc 1333 MHz 1333 MHz NSc 1066 MHz 1066 MHz 1066 MHz 1333 MHz 1333 MHz 1066 MHz 800 MHz NSb NSd 1066 MHz 800 MHz NSd 1.35 V 16 4 GB 48 GB 48 GB UDIMMs Dual rank 49Y1404 (4 GB) 1333 MHz 1.35 V 1.5 V 16 4 GB 48 GB 48 GB 1.35 V 24 32 GB 768 GB N/A LRDIMMs Quad rank 49Y1567 (16 GB) 90Y3105 (32 GB) 1333 MHz 1.35 V 1.5 V 24 32 GB 768 GB 512 GB 1.35 V 8 32 GB 256 GB N/A RDIMMs Quad rank 49Y1400 (16 GB) 90Y3102 (32 GB) 1066 MHz 1.35 V 1.5 V 16 32 GB 512 GB 256 GB
251
a. The maximum quantity that is supported is shown for two processors installed. When one processor is installed, the maximum quantity that is supported is half of that shown. b. NS = Not supported at 1.35 V. Will operate at 1.5 V instead c. NS = Not supported. UDIMMs only support up to 2 DIMMs per channel. d. NS = Not supported. RDIMMs support up to 8 ranks per channel. Table 5-51 Maximum memory speeds (Part 2 - Single and Dual rank RDIMMs) Spec Rank Part numbers Single rank 49Y1405 (2GB) 49Y1406 (4GB) 1333 MHz 1.35 V 1.35 V 16 4 GB 64 GB N/A 1.5 V 24 4 GB 96 GB 64 GB 49Y1559 (4GB) RDIMMs Dual rank 49Y1407 (4GB) 49Y1397 (8GB) 49Y1563 (16GB) 1333 MHz 1.35 V 1.35 V 16 16 GB 256 GB N/A 1.5 V 24 16 GB 384 GB 256 GB 90Y3178 (4GB) 90Y3109 (8GB) 00D4968 (16GB) 1600 MHz 1.5 V 1.5 V 24 16 GB 384 GB 256 GB
Maximum operating speed (MHz) 1 DIMM per channel 2 DIMMs per channel 3 DIMMs per channel 1333 MHz 1333 MHz NSb 1333 MHz 1333 MHz 1066 MHz 1600 MHz 1600 MHz 1066 MHz 1333 MHz 1333 MHz NSb 1333 MHz 1333 MHz 1066 MHz 1600 MHz 1600 MHz 1066 MHz
a. The maximum quantity that is supported is shown for two processors installed. When one processor is installed, the maximum quantity that is supported is half of that shown. b. NS = Not supported at 1.35 V. Will operate at 1.5 V instead
Tip: When an unsupported memory configuration is detected, the IMM illuminates the DIMM mismatch light path error LED and the system does not boot. A DIMM mismatch error includes the following examples: Mixing of RDIMMs, UDIMMs, or LRDIMMs in the system Not adhering to the DIMM population rules In some cases, the error log points to the DIMM slots that are mismatched.
Memory modes
The x240 type 8737 supports the following memory modes: Independent channel mode Rank-sparing mode Mirrored-channel mode These modes can be selected in the Unified Extensible Firmware Interface (UEFI) setup. For more information, see 5.4.13, Systems management on page 271.
252
IBM PureFlex System and IBM Flex System Products and Technology
Rank-sparing mode
In rank-sparing mode, one memory DIMM rank serves as a spare of the other ranks on the same channel. The spare rank is held in reserve and is not used as active memory. The spare rank must have an identical or larger memory capacity than all the other active memory ranks on the same channel. After an error threshold is surpassed, the contents of that rank are copied to the spare rank. The failed rank of memory is taken offline, and the spare rank is put online and used as active memory in place of the failed rank. The memory DIMM installation sequence when using rank-sparing mode is identical to independent channel mode, as described in Memory DIMM installation: Independent channel and rank-sparing modes on page 254.
Mirrored-channel mode
In mirrored-channel mode, memory is installed in pairs. Each DIMM in a pair must be identical in capacity, type, and rank count. The channels are grouped in pairs. Each channel in the group receives the same data. One channel is used as a backup of the other, which provides redundancy. The memory contents on channel 0 are duplicated in channel 1, and the memory contents of channel 2 are duplicated in channel 3. The DIMMs in channel 0 and channel 1 must be the same size and type. The DIMMs in channel 2 and channel 3 must be the same size and type. The effective memory that is available to the system is only half of what is installed. Because memory mirroring is handled in hardware, it is operating system-independent. Consideration: In a two processor configuration, memory must be identical across the two processors to enable the memory mirroring feature.
253
Figure 5-30 shows the E5-2600 series processor with the four memory channels and which channels are mirrored when operating in mirrored-channel mode.
Channel 1
Channel 3
DIMM 10
Channel 2
DIMM 7
Mirrored Pair
Figure 5-30 Showing the mirrored channels and DIMM pairs when in mirrored-channel mode
For more information about the memory DIMM installation sequence when mirrored channel mode is used, see Memory DIMM installation: Mirrored-channel on page 257.
254
IBM PureFlex System and IBM Flex System Products and Technology
DIMM 8
DIMM 9
DIMM 11
DIMM 12
DIMM 1
DIMM 2
DIMM 3
Table 5-52 shows DIMM installation if you have one processor installed.
Table 5-52 Suggested DIMM installation for the x240 with one processor installed Optimal memory configa Processor 1
Channel 2 Channel 1 Channel 3 Channel 4 Channel 3
Processor 2
Channel 4 Channel 2 Channel 1
Number of processors
Number of DIMMs
DIMM 10
DIMM 11
DIMM 12
DIMM 13
DIMM 14
DIMM 15
DIMM 16
DIMM 17
DIMM 18
DIMM 19
DIMM 20
DIMM 21
DIMM 22
DIMM 23
1 1 1 1 1 1 1 1 1 1 1 1
1 2 3 4 5 6 7 8 9 10 11 12 x x x x x x x x x x x x x x x x x x x x x
x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x
a. For optimal memory performance, populate all the memory channels equally.
255
DIMM 24
DIMM 1
DIMM 2
DIMM 3
DIMM 4
DIMM 5
DIMM 6
DIMM 7
DIMM 8
DIMM 9
Table 5-53 shows DIMM installation if you have two processors installed.
Table 5-53 Suggested DIMM installation for the x240 with two processors installed Optimal memory configa Processor 1
Channel 2 Channel 1 Channel 3 Channel 4 Channel 3
Processor 2
Channel 4 Channel 2 Channel 1
Number of processors
Number of DIMMs
DIMM 10
DIMM 11
DIMM 12
DIMM 13
DIMM 14
DIMM 15
DIMM 16
DIMM 17
DIMM 18
DIMM 19
DIMM 20
DIMM 21
DIMM 22
DIMM 23 x x x x x x x x x x x
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x
x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x
a. For optimal memory performance, populate all the memory channels equally.
256
IBM PureFlex System and IBM Flex System Products and Technology
DIMM 24
DIMM 1
DIMM 2
DIMM 3
DIMM 4
DIMM 5
DIMM 6
DIMM 7
DIMM 8
DIMM 9
a. The pair of DIMMs must be identical in capacity, type, and rank count.
257
USB ports
The x240 has one external USB port on the front of the compute node. Figure 5-31 shows the location of the external USB connector on the x240.
Figure 5-31 The front USB connector on the x240 Compute Node
The x240 also supports an option that provides two internal USB ports (x240 USB Enablement Kit) that are primarily used for attaching USB hypervisor keys. For more information, see 5.4.10, Integrated virtualization on page 266.
258
IBM PureFlex System and IBM Flex System Products and Technology
Table 5-55 lists the ordering part number and feature code of the console breakout cable. One console breakout cable ships with the IBM Flex System Enterprise Chassis.
Table 5-55 Ordering part number and feature code Part number 81Y5286 Feature code A1NF Description IBM Flex System Console Breakout Cable
Figure 5-33 The LSI2004 SAS controller connections to the HDD interface
259
Figure 5-34 shows the front of the x240, including the two hot-swap drive bays.
Figure 5-34 The x240 showing the front hot-swap disk drive bays
10K SAS hard disk drives 42D0637 90Y8877 49Y2003 90Y8872 81Y9650 00AD075 5599 A2XC 5433 A2XD A282 A48S IBM 300 GB 10K 6 Gbps SAS 2.5" SFF Slim-HS HDD IBM 300GB 10K 6Gbps SAS 2.5" SFF G2HS HDD IBM 600 GB 10K 6 Gbps SAS 2.5" SFF Slim-HS HDD IBM 600GB 10K 6Gbps SAS 2.5" SFF G2HS HDD IBM 900 GB 10K 6 Gbps SAS 2.5" SFF HS HDD IBM 1.2TB 10K 6Gbps SAS 2.5'' G2HS HDD
15K SAS hard disk drives 42D0677 90Y8926 81Y9670 NL SATA 81Y9722 81Y9726 81Y9730 NL SAS 42D0707 90Y8953 81Y9690 5409 A2XE A1P3 IBM 500 GB 7200 6 Gbps NL SAS 2.5" SFF Slim-HS HDD IBM 500GB 7.2K 6Gbps NL SAS 2.5" SFF G2HS HDD IBM 1TB 7.2K 6 Gbps NL SAS 2.5" SFF HS HDD A1NX A1NZ A1AV IBM 250 GB 7.2K 6 Gbps NL SATA 2.5" SFF HS HDD IBM 500 GB 7.2K 6 Gbps NL SATA 2.5" SFF HS HDD IBM 1TB 7.2K 6 Gbps NL SATA 2.5" SFF HS HDD 5536 A2XB A283 IBM 146 GB 15K 6 Gbps SAS 2.5" SFF Slim-HS HDD IBM 146GB 15K 6Gbps SAS 2.5" SFF G2HS HDD IBM 300 GB 15K 6 Gbps SAS 2.5" SFF HS HDD
10K and 15K Self-encrypting drives (SED) 90Y8944 90Y8913 A2ZK A2XF IBM 146GB 15K 6Gbps SAS 2.5" SFF G2HS SED IBM 300GB 10K 6Gbps SAS 2.5" SFF G2HS SED
260
IBM PureFlex System and IBM Flex System Products and Technology
Description IBM 600GB 10K 6Gbps SAS 2.5" SFF G2HS SED IBM 900GB 10K 6Gbps SAS 2.5" SFF G2HS SED IBM 1.2TB 10K 6Gbps SAS 2.5'' G2HS SED
SAS-SSD Hybrid drive 00AD102 A4G7 IBM 600GB 10K 6Gbps SAS 2.5'' G2HS Hybrid
Solid-state drives - Enterprise 41Y8331 41Y8336 41Y8341 00W1125 49Y6129 49Y6134 49Y6139 49Y6195 A4FL A4FN A4FQ A3HR A3EW A3EY A3F0 A4GH S3700 200GB SATA 2.5" MLC HS Enterprise SSD S3700 400GB SATA 2.5" MLC HS Enterprise SSD S3700 800GB SATA 2.5" MLC HS Enterprise SSD IBM 100GB SATA 2.5" MLC HS Enterprise SSD IBM 200GB SAS 2.5" MLC HS Enterprise SSD IBM 400GB SAS 2.5" MLC HS Enterprise SSD IBM 800GB SAS 2.5" MLC HS Enterprise SSD IBM 1.6TB SAS 2.5" MLC HS Enterprise SSD
Solid-state drives - Enterprise Value 49Y5844 49Y5839 90Y8643 90Y8648 A3AU A3AS A2U3 A2U4 IBM 512GB SATA 2.5" MLC HS Enterprise Value SSD IBM 64GB SATA 2.5" MLC HS Enterprise Value SSD IBM 256GB SATA 2.5" MLC HS Enterprise Value SSD IBM 128GB SATA 2.5" MLC HS Enterprise Value SSD
261
The ServeRAID M5115 supports the following combinations of 2.5-inch drives and 1.8-inch solid-state drives: Up to two 2.5-inch drives only Up to four 1.8-inch drives only Up to two 2.5-inch drives, plus up to four 1.8-inch solid-state drives Up to eight 1.8-inch solid-state drives The ServeRAID M5115 SAS/SATA Controller (90Y4390) provides an advanced RAID controller that supports RAID 0, 1, 10, 5, 50, and, optionally, 6 and 60. It includes 1 GB of cache. This cache can be backed up to a flash cache when attached to the supercapacitor included with the optional ServeRAID M5100 Series Enablement Kit (90Y4342). At least one of the following hardware kits is required with the ServeRAID M5115 controller to enable specific drive support: ServeRAID M5100 Series Enablement Kit for IBM Flex System x240 (90Y4342) enables support for up to two 2.5-inch HDDs or SSDs in the hot-swap bays in the front of the server. It includes a CacheVault unit, which enables MegaRAID CacheVault flash cache protection. This enablement kit replaces the standard two-bay backplane (which is attached through the system board to an onboard controller) with a new backplane. The new backplane attaches with an included flex cable to the M5115 controller. It also includes an air baffle, which also serves as an attachment for the CacheVault unit. MegaRAID CacheVault flash cache protection uses NAND flash memory that is powered by a supercapacitor to protect data that is stored in the controller cache. This module eliminates the need for the lithium-ion battery that is commonly used to protect DRAM cache memory on Peripheral Component Interconnect (PCI) RAID controllers. To avoid data loss or corruption during a power or server failure, CacheVault technology transfers the contents of the DRAM cache to NAND flash. This process uses power from the supercapacitor. After the power is restored to the RAID controller, the saved data is transferred from the NAND flash back to the DRAM cache. The DRAM cache can then be flushed to disk. Tip: The Enablement Kit is only required if 2.5-inch drives are used. If you plan to install four or eight 1.8-inch SSDs, this kit is not required. ServeRAID M5100 Series IBM Flex System Flash Kit for x240 (90Y4341) enables support for up to four 1.8-inch SSDs. This kit replaces the standard two-bay backplane with a four-bay SSD backplane that attaches with an included flex cable to the M5115 controller. Because only SSDs are supported, a CacheVault unit is not required, and so this kit does not have a supercap. ServeRAID M5100 Series SSD Expansion Kit for IBM Flex System x240 (90Y4391) enables support for up to four internal 1.8-inch SSDs. This kit includes two air baffles that replace the existing baffles, and each baffle has mounts for two SSDs. Included flexible cables connect the drives to the controller. Table 5-59 on page 263 shows the kits that are required for each combination of drives. For example, if you plan to install eight 1.8-inch SSDs, you need the M5115 controller, the Flash kit, and the SSD Expansion kit. Tip: If the ServeRAID M5100 Series SSD Expansion Kit (90Y4391) is installed, the x240 USB Enablement Kit (49Y8119, which is described in 5.2.11, Integrated virtualization on page 211) cannot be installed. The x240 USB Enablement Kit and the SSD Expansion Kit both include special air baffles that cannot be installed at the same time.
262
IBM PureFlex System and IBM Flex System Products and Technology
Table 5-59 ServeRAID M5115 hardware kits Required drive support Maximum number of 2.5-inch drives 2 0 2 0 Maximum number of 1.8-inch SSDs 0 4 (front) 4 (internal) 8 (both) => => => => Components required ServeRAID M5115 90Y4390 Required Required Required Required Required Required Enablement Kit 90Y4342 Required Required Required Required Flash Kit 90Y4341 SSD Expansion Kit 90Y4391a
a. If you install the SSD Expansion Kit, you cannot install the x240 USB Enablement Kit (49Y8119) at the same time.
Figure 5-35 shows how the ServeRAID M5115 and the Enablement Kit are installed in the server to support two 2.5-inch drives with MegaRAID CacheVault flash cache protection (as shown in row 1 of Table 5-59).
ServeRAID M5115 controller (90Y4390) with ServeRAID M5100 Series Enablement Kit (90Y4342)
ServeRAID M5115 controller
Figure 5-35 The ServeRAID M5115 and the Enablement Kit installed
263
Figure 5-36 shows how the ServeRAID M5115 and Flash and SSD Expansion Kits are installed in the server to support eight 1.8-inch solid-state drives (as shown in row 4 of Table 5-59 on page 263).
ServeRAID M5115 controller (90Y4390) with ServeRAID M5100 Series Flash Kit (90Y4341) and ServeRAID M5100 Series SSD Expansion Kit (90Y4391)
ServeRAID M5115 controller
SSD Expansion Kit: Four SSDs on special air baffles above DIMMs (no CacheVault flash protection)
Figure 5-36 ServeRAID M5115 with Flash and SSD Expansion Kits installed
The eight SSDs are installed in the following locations: Four in the front of the system in place of the two 2.5-inch drive bays Two in a tray above the memory banks for CPU 1 Two in a tray above the memory banks for CPU 2 The ServeRAID M5115 controller has the following specifications: Eight internal 6 Gbps SAS/SATA ports PCI Express 3.0 x8 host interface 6 Gbps throughput per port 800 MHz dual-core IBM PowerPC processor with LSI SAS2208 6 Gbps RAID-on-Chip (ROC) controller Support for RAID levels 0, 1, 10, 5, 50 standard; support for RAID 6 and 60 with optional upgrade using 90Y4411 Optional onboard 1 GB data cache (DDR3 running at 1333 MHz) with optional flash backup (MegaRAID CacheVault technology) as part of the Enablement Kit 90Y4342 Support for SAS and SATA HDDs and SSDs Support for intermixing SAS and SATA HDDs and SSDs; mixing different types of drives in the same array (drive group) is not recommended Support for self-encrypting drives (SEDs) with MegaRAID SafeStore Optional support for SSD performance acceleration with MegaRAID FastPath and SSD caching with MegaRAID CacheCade Pro 2.0 (90Y4447) Support for up to 64 virtual drives, up to 128 drive groups, up to 16 virtual drives per drive group, and up to 32 physical drives per drive group Support for logical unit number (LUN) sizes up to 64 TB 264
IBM PureFlex System and IBM Flex System Products and Technology
Configurable stripe size up to 1 MB Compliant with Disk Data Format (DDF) configuration on disk (CoD) S.M.A.R.T. support MegaRAID Storage Manager management software Optional add-ons to the ServeRAID M5115 controller are RAID 6 support, SSD performance accelerator, and SSD caching enabler. Table 5-60 lists all Feature on Demand (FoD) license upgrades.
Table 5-60 Supported upgrade features Part number 90Y4410 90Y4412 90Y4447 Feature code A2Y1 A2Y2 A36G Description ServeRAID M5100 Series RAID 6 Upgrade for IBM Flex System ServeRAID M5100 Series Performance Accelerator for IBM Flex System (MegaRAID FastPath) ServeRAID M5100 Series SSD Caching Enabler for IBM Flex System (MegaRAID CacheCade Pro 2.0) Maximum supported 1 1 1
These features have the following characteristics: RAID 6 Upgrade (90Y4410) Adds support for RAID 6 and RAID 60. This license is a Feature on Demand license. Performance Accelerator (90Y4412) The Performance Accelerator for IBM Flex System is implemented by using the LSI MegaRAID FastPath software. It provides high-performance I/O acceleration for SSD-based virtual drives by using a low-latency I/O path to increase the maximum input/output operations per second (IOPS) capability of the controller. This feature boosts the performance of applications with a highly random data storage access pattern, such as transactional databases. Part number 90Y4412 is a Feature on Demand license. SSD Caching Enabler for traditional hard disk drives (90Y4447) The SSD Caching Enabler for IBM Flex System is implemented by using the LSI MegaRAID CacheCade Pro 2.0. It is designed to accelerate the performance of HDD arrays with only an incremental investment in SSD technology. The feature enables the SSDs to be configured as a dedicated cache. This configuration helps maximize the I/O performance for transaction-intensive applications, such as databases and web serving. The feature tracks data storage access patterns and identifies the most frequently accessed data. The hot data is then automatically stored on the SSDs that are assigned as a dedicated cache pool on the ServeRAID controller. Part number 90Y4447 is a Feature on Demand license. This feature requires at least one SSD drive be installed.
265
a. The Blank USB Memory Key requires the download of the VMware vSphere (ESXi) Hypervisor with IBM Customization image, which is available at this website: http://ibm.com/systems/x/os/vmware/
The USB memory keys connect to the internal x240 USB Enablement Kit. Table 5-63 lists the ordering information for the internal x240 USB Enablement Kit.
Table 5-63 Internal USB port option Part number 49Y8119 Feature code A3A3 Description x240 USB Enablement Kit
266
IBM PureFlex System and IBM Flex System Products and Technology
The x240 USB Enablement Kit connects to the system board of the server, as shown in Figure 5-37. The kit offers two ports and enables you to install two memory keys. If you install both keys, both devices are listed in the boot menu. With this setup, you can boot from either device, or set one as a backup in case the first one becomes corrupted.
USB flash key USB two-port assembly
Figure 5-37 The x240 compute node showing the location of the internal x240 USB Enablement Kit
There are two types of USB keys, preloaded keys or blank keys. Blank keys allow you to download an IBM customized version of ESXi and load it onto the key. The x240 supports one or two keys installed, but only in the following combinations: One preload key One blank key One preload key and one blank key Two blank keys Two preload keys is an unsupported combination. Installing two preloaded keys prevents ESXi from booting as described at this website: http://kb.vmware.com/kb/1035107 Having two keys installed provides a backup boot device. Both devices are listed in the boot menu, which allows you to boot from either device or to set one as a backup in case the first one gets corrupted. Consideration: The x240 USB Enablement Kit and USB memory keys are not supported if the ServeRAID M5100 Series SSD Expansion Kit (90Y4391) is already installed because these kits occupy the same location in the server.
267
For a complete description of the features and capabilities of VMware ESX Server, see this website: http://www.vmware.com/products/vi/esx/
The Compute Node Fabric Connector enables port 1 on the Embedded 10 Gb Virtual Fabric adapter to be routed to I/O module bay 1. Similarly, port 2 can be routed to I/O module bay 2. The Compute Node Fabric Connector can be unscrewed and removed, if required, to allow the installation of an I/O adapter on I/O connector 1. Consideration: If I/O connector 1 has the Embedded 10 Gb Virtual Fabric adapter installed, only I/O connector 2 is available for the installation of additional I/O adapters. (An exception is that the ServeRAID Controller can coexist in slot 1 with an Embedded adapter.) The Embedded 10 Gb Virtual Fabric adapter is based on the Emulex BladeEngine 3, which is a single-chip, dual-port 10 Gigabit Ethernet (10 GbE) Ethernet Controller. The Embedded 10 Gb Virtual Fabric adapter includes the following features: PCI-Express Gen2 x8 host bus interface Supports multiple Virtual Network Interface Card (vNIC) functions TCP/IP offload Engine (TOE enabled) SRIOV capable RDMA over TCP/IP capable iSCSI and FCoE upgrade offering using FoD
268
IBM PureFlex System and IBM Flex System Products and Technology
Table 5-64 lists the ordering information for the IBM Flex System Embedded 10 Gb Virtual Fabric Upgrade. This upgrade enables the iSCSI and FCoE support on the Embedded 10 Gb Virtual Fabric adapter.
Table 5-64 Feature on Demand upgrade for FCoE and iSCSI support Part number 90Y9310 Feature code A2TD Description IBM Virtual Fabric Advanced Software Upgrade (LOM)
Figure 5-39 shows the x240 and the location of the Compute Node Fabric Connector on the system board.
Captive screws LOM connector
Figure 5-39 The x240 showing the location of the Compute Node Fabric Connector
269
Figure 5-40 shows the rear of the x240 compute node and the locations of the I/O connectors.
I/O connector 1
I/O connector 2
Figure 5-40 Rear of the x240 compute node showing the locations of the I/O connectors
Table 5-65 lists the I/O adapters that are supported in the x240.
Table 5-65 Supported I/O adapters for the x240 compute node Part number Feature code Ports Description
Ethernet adapters 49Y7900 90Y3466 90Y3554 90Y3482 A10Y A1QY A1R1 A3HK 4 2 4 2 IBM Flex System EN2024 4-port 1Gb Ethernet Adapter IBM Flex System EN4132 2-port 10Gb Ethernet Adapter IBM Flex System CN4054 10Gb Virtual Fabric Adapter IBM Flex System EN6132 2-port 40Gb Ethernet Adapter
Fibre Channel adapters 69Y1938 95Y2375 88Y6370 95Y2386 95Y2391 69Y1942 A1BM A2N5 A1BP A45R A45S A1BQ 2 2 2 2 4 2 IBM Flex System FC3172 2-port 8Gb FC Adapter IBM Flex System FC3052 2-port 8Gb FC Adapter IBM Flex System FC5022 2-port 16Gb FC Adapter IBM Flex System FC5052 2-port 16Gb FC Adapter IBM Flex System FC5054 4-port 16Gb FC Adapter IBM Flex System FC5172 2-port 16Gb FC Adapter
InfiniBand adapters 90Y3454 A1QZ 2 IBM Flex System IB6132 2-port FDR InfiniBand Adapter
Requirement: Any supported I/O adapter can be installed in either I/O connector. However, you must be consistent not only across chassis, but across all compute nodes.
270
IBM PureFlex System and IBM Flex System Products and Technology
The x240 also supports adapters that are installed in an attached Flex System PCIe Expansion Node. For more information, see 5.9, IBM Flex System PCIe Expansion Node on page 356.
USB port
Identify LED
Fault LED
NMI control
Figure 5-41 The front of the x240 with the front panel LEDs and controls shown
Location
Blue
Yellow Yellow
271
LED Hard disk drive activity LED Hard disk drive status LED
Description Each hot-swap hard disk drive has an activity LED. When this LED is flashing, it indicates that the drive is in use. When this LED is lit, it indicates that the drive failed. If an optional IBM ServeRAID controller is installed in the server, when this LED is flashing slowly (one flash per second), it indicates that the drive is being rebuilt. When the LED is flashing rapidly (three flashes per second), it indicates that the controller is identifying the drive.
NMI
Power LED
The status of the power LED of the x240 shows the power status of the x240 compute node. It also indicates the discovery status of the node by the Chassis Management Module. The power LED states are listed in Table 5-68.
Table 5-68 The power LED states of the x240 compute node Power LED state Off On; fast flash mode Status of compute node No power to the compute node. The compute node has power. The Chassis Management Module is in discovery mode (handshake). The compute node has power. Power in stand-by mode. The compute node has power. The compute node is operational.
Consideration: The power button does not operate when the power LED is in fast flash mode.
The x240 light path diagnostics panel is visible when you remove the server from the chassis. The panel is on the upper right of the compute node, as shown in Figure 5-42.
To illuminate the light path diagnostics LEDs, power off the compute node, slide it out of the chassis, and press the power button. The power button doubles as the light path diagnostics remind button when the server is removed from the chassis. The meaning of each LED in the light path diagnostics panel is listed in Table 5-69.
Table 5-69 Light path panel LED definitions LED LP S BRD MIS NMI TEMP MEM ADJ Color Green Yellow Yellow Yellow Yellow Yellow Yellow Meaning The light path diagnostics panel is operational. A system board error is detected. A mismatch occurred between the processors, DIMMs, or HDDs within the configuration as reported by POST. A non-maskable interrupt (NMI) occurred. An over-temperature condition occurred that was critical enough to shut down the server. A memory fault occurred. The corresponding DIMM error LEDs on the system board are also lit. A fault is detected in the adjacent expansion unit (if installed).
273
Remote access to system fan, voltage, and temperature values Remote IMM and UEFI update UEFI update when the server is powered off Remote console by way of a serial over LAN Remote access to the system event log Predictive failure analysis and integrated alerting features; for example, by using SNMP Remote presence, including remote control of server by using a Java or Active x client Operating system failure window (blue screen) capture and display through the web interface Virtual media that allow the attachment of a diskette drive, CD/DVD drive, USB flash drive, or disk image to a server Remember: Unlike IBM BladeCenter, the assigned TCP/IP address of the IMM is available on the local network. You can use this address to remotely manage the x240 by connecting directly to the IMM independent of the File System Manager (FSM) or Chassis Management Module (CMM). For more information about the IMM, see 3.4.1, Integrated Management Module II on page 47.
274
IBM PureFlex System and IBM Flex System Products and Technology
5.5.1 Introduction
The IBM Flex System x440 Compute Node is a double-wide compute node that provides scalability to support up to four Intel Xeon E5-4600 processors. The nodes width allows for significant I/O capability. The server is ideal for virtualization, database, and memory-intensive high performance computing environments. Figure 5-43 shows the front of the compute node, which includes the location of the controls, LEDs, and connectors. The light path diagnostic panel is on the upper edge of the front panel bezel, in the same place as on the x220 and x240.
Two 2.5 HS drive bays Light path diagnostics panel
LED panel
275
Figure 5-44 shows the internal layout and major components of the x440.
Cover
Air baffle Air baffle Air baffle Heat sink Microprocessor heat sink filler Backplane assembly I/O expansion adapter
Microprocessor
Figure 5-44 Exploded view of the x440 showing the major components
Chipset Memory
276
IBM PureFlex System and IBM Flex System Products and Technology
Components Memory maximums Memory protection Disk drive bays Maximum internal storage
Specification With LRDIMMs: Up to 1.5 TB with 48x 32 GB LRDIMMs and four processors. With RDIMMs: Up to 768 GB with 48x 16 GB RDIMMs and four processors. ECC, Chipkill (for x4-based memory DIMMs), memory mirroring, and memory rank sparing. Two 2.5-inch hot-swap SAS/SATA drive bays that support SAS, SATA, and SSD drives. Optional Flash Kit support for up to eight 1.8-inch SSDs. With two 2.5-inch hot-swap drives: Up to 2 TB with 1 TB 2.5" NL SAS HDDs, or up to 2.4 TB with 1.2 TB 2.5" SAS HDDs, or up to 2 TB with 1 TB 2.5" NL SATA HDDs, or up to 3.2 TB with 1.6 TB 2.5" SAS SSDs. Intermix of SAS and SATA HDDs and SSDs is supported. With 1.8-inch SSDs and ServeRAID M5115 RAID adapter: Up to 1.6 TB with eight 200 GB 1.8-inch SSDs. RAID 0 and 1 with integrated LSI SAS2004 controller. Optional ServeRAID M5115 RAID controller with RAID 0, 1, 10, 5, and 50 support and 1 GB cache. Supports up to eight 1.8-inch SSDs with expansion kits. Optional flash-backup for cache, RAID 6/60, and SSD performance enabler. x4x models: Four 10 Gb Ethernet ports with two dual-port Embedded 10Gb Virtual Fabric Ethernet LAN-on-motherboard (LOM) controllers; Emulex BE3 based. Upgradeable to FCoE and iSCSI using IBM Feature on Demand license activation. x2x models: None standard; optional 1 Gb or 10 Gb Ethernet adapters. Four I/O connectors for adapters. PCI Express 3.0 x16 interface. USB ports: One external. Two internal for embedded hypervisor. Console breakout cable port that provides local KVM and serial ports (cable standard with chassis; additional cables are optional). UEFI, IBM Integrated Management Module 2 (IMM2) with Renesas SH7757 controller, Predictive Failure Analysis, light path diagnostics panel, automatic server restart, and remote presence. Support for IBM Flex System Manager, IBM Systems Director, and IBM ServerGuide. Power-on password, administrator's password, and Trusted Platform Module V1.2. Matrox G200eR2 video core with 16 MB video memory that is integrated into the IMM2. Maximum resolution is 1600x1200 at 75 Hz with 16 M colors. Three-year customer-replaceable unit and onsite limited warranty with 9x5/NBD. Microsoft Windows Server 2008 R2, Red Hat Enterprise Linux 5 and 6, SUSE Linux Enterprise Server 10 and 11, VMware ESX 4, and vSphere 5. For details, see 5.5.14, Operating systems support on page 297. Optional service upgrades are available through IBM ServicePac offerings: Four-hour or 2-hour response time, 8-hour fix time, 1-year or 2-year warranty extension, and remote technical support for IBM hardware and selected IBM and OEM software. Width: 437 mm (17.2 in.), height 51 mm (2.0 in.), depth 493 mm (19.4 in.) Maximum weight: 12.25 kg (27 lbs).
RAID support
Network interfaces
Systems management Security features Video Limited warranty Operating systems supported Service and support
Dimensions Weight
277
Figure 5-45 shows the components on the system board of the x440.
1
I/O adapters: 1 (top) to 4 (bottom)
2
USB ports
Figure 5-45 Layout of the IBM Flex System x440 Compute Node system board
5.5.2 Models
The current x440 models, with processor, memory, and other embedded options that are shipped as standard with each model type, are shown in Table 5-71.
Table 5-71 Model Standard models of the IBM Flex System x440 Compute Node, type 7917 Intel Processor E5-4800: 4 maximuma Memory RAID adapter Disk bays (used/max)b Disks Embedded 10GbE Virtual Fabric No Standard No Standard No Standard I/O slots (used/ max) 0/4 2 / 4d 0/4 2 / 4d 0/4 2 / 4d
Xeon E5-4603 4C 2.0 GHz 10 MB 1066 MHz 95W Xeon E5-4603 4C 2.0 GHz 10 MB 1066 MHz 95W Xeon E5-4607 6C 2.2 GHz 12 MB 1066 MHz 95W Xeon E5-4607 6C 2.2 GHz 12 MB 1066 MHz 95W Xeon E5-4610 6C 2.4 GHz 15 MB 1333 MHz 95W Xeon E5-4610 6C 2.4 GHz 15 MB 1333 MHz 95W
1x 8 GB 1066 MHzc 1x 8 GB 1066 MHzc 1x 8 GB 1066 MHzc 1x 8 GB 1066 MHzc 1x 8 GB 1333 MHz 1x 8 GB 1333 MHz
SAS/SATA RAID SAS/SATA RAID SAS/SATA RAID SAS/SATA RAID SAS/SATA RAID SAS/SATA RAID
2.5 hot-swap (0 / 2) 2.5 hot-swap (0 / 2) 2.5 hot-swap (0 / 2) 2.5 hot-swap (0 / 2) 2.5 hot-swap (0 / 2) 2.5 hot-swap (0 / 2)
278
IBM PureFlex System and IBM Flex System Products and Technology
Model
Memory
RAID adapter
Disks
Xeon E5-4620 8C 2.2 GHz 16 MB 1333 MHz 95W Xeon E5-4620 8C 2.2 GHz 16 MB 1333 MHz 95W Xeon E5-4650 8C 2.7 GHz 20 MB 1600 MHz 130W Xeon E5-4650 8C 2.7 GHz 20 MB 1600 MHz 130W
a. Processor detail: Processor quantity and model, cores, core speed, L3 cache, memory speed, and power consumption. b. The 2.5-inch drive bays can be replaced and expanded with more internal bays to support up to eight 1.8-inch SSDs. For more information, see 5.5.7, Internal disk storage on page 284. c. For models Axx and Bxx, the standard DIMM is rated at 1333 MHz, but operates at up to 1066 MHz to match the processor memory speed. d. The x4x models include two Embedded 10Gb Virtual Fabric Ethernet controllers. Connections are routed by using a Fabric Connector. The Fabric Connectors preclude the use of an I/O adapter in I/O connectors 1 and 3, except the ServeRAID M5115 controller, which can be installed in slot 1.
Up to seven x440 Compute Nodes can be installed in the chassis in 10U of rack space. The actual number of x440 systems that can be powered on in a chassis depends on the following factors: The TDP power rating for the processors that are installed in the x440. The number of power supplies installed in the chassis. The capacity of the power supplies installed in the chassis (2100 W or 2500 W). The power redundancy policy used in the chassis (N+1 or N+N). Table 4-11 on page 93 provides guidelines about what number of x440 systems can be powered on in the IBM Flex System Enterprise Chassis, based on the type and number of power supplies installed.
279
PCIe Mux
PCIe x4 G2
PCIe x8 G3 Intel Xeon CPU 1 QPI links (8 GT/s) Intel Xeon CPU 2
USB
PCIe x16 G3
The IBM Flex System x440 Compute Node has the following system architecture features as standard: Four 2011-pin type R (LGA-2011) processor sockets. An Intel C600 PCIe Controller Hub (PCH). Four memory channels per socket. Up to three DIMMs per memory channel. A total of 16 DDR3 DIMM sockets. Support for LRDIMMs and RDIMMs. Two dual port integrated 10Gb Virtual Fabric Ethernet controllers that are based on Emulex BE3. Upgradeable to FCoE and iSCSI through IBM Features on Demand (FoD). 280
IBM PureFlex System and IBM Flex System Products and Technology
One LSI 2004 SAS controller with integrated RAID 0 and 1 to the two internal drive bays. Support for ServeRAID M5115 controller for RAID 5 and other levels to up to 1.8-inch bays. Integrated Management Module II (IMMv2) for systems management. Four PCIe 3.0 I/O adapter connectors x16. Two internal and one external USB connectors. Important: A second processor must be installed to use I/O adapter slots 3 and 4 in the x440 compute node. This configuration is necessary because the PCIe lanes that are used to drive I/O slots 3 and 4 are routed to processors 2 and 4, as shown in Figure 5-46 on page 280.
CPU s 1&2
Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No
CPU s 3&4
No Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes
90Y9060 88Y6263 90Y9062 69Y3100 90Y9064 69Y3106 90Y9066 90Y9049 90Y9070 69Y3112 90Y9068 90Y9055 90Y9072 69Y3118 90Y9186 90Y9185
Xeon E5-4603 4C 2.0GHz 10MB 1066MHz 95W Xeon E5-4603 4C 2.0GHz 10MB 1066MHz 95W Xeon E5-4607 6C 2.2GHz 12MB 1066MHz 95W Xeon E5-4607 6C 2.2GHz 12MB 1066MHz 95W Xeon E5-4610 6C 2.4GHz 15MB 1333MHz 95W Xeon E5-4610 6C 2.4GHz 15MB 1333MHz 95W Xeon E5-4617 6C 2.9GHz 15MB 1600MHz 130W Xeon E5-4617 6C 2.9GHz 15MB 1600MHz 130W Xeon E5-4620 8C 2.2GHz 16MB 1333MHz 95W Xeon E5-4620 8C 2.2GHz 16MB 1333MHz 95W Xeon E5-4640 8C 2.4GHz 20MB 1600MHz 95W Xeon E5-4640 8C 2.4GHz 20MB 1600MHz 95W Xeon E5-4650 8C 2.7GHz 20MB 1600MHz 130W Xeon E5-4650 8C 2.7GHz 20MB 1600MHz 130W Xeon E5-4650L 8C 2.6GHz 20MB 1600MHz 115W Xeon E5-4650L 8C 2.6GHz 20MB 1600MHz 115W
281
a. When two feature codes are specified, the first feature code is for CPU 1 and the second feature code is for CPU 2. When only one feature code is specified, this is the feature code that is used for CPU 3 and CPU 4.
Rated speed
282
IBM PureFlex System and IBM Flex System Products and Technology
Specification Rated voltage Maximum quantitya Largest DIMM Max memory capacity Max memory at rated speed
RDIMMs 1.35 V 48 4 GB 192 GB 192 GB 1.5V 48 4 GB 192 GB 128 GB 1.35 V 48 16 GB 768 GB 512 GB 1.5 V 48 16 GB 768 GB 512 GB
Maximum operating speed (MHz) 1 DIMM per channel 2 DIMMs per channel 3 DIMMs per channel 1333 MHz 1333 MHz 1066 MHz (1.5 V) 1600 MHz 1600 MHz 1066 MHz 1333 MHz 1333 MHz 1066 (1.5V) 1600 MHz 1600 MHz 1066 MHz 1333 MHz (1.5 V) 1333 MHz (1.5V) 1066 MHz
a. The maximum quantity that is supported is shown for four processors installed. When two processors are installed, the maximum quantity that is supported is a half of the quantity that is shown. When one processor is installed, the quantity is one quarter of that shown.
The x440 supports the following memory protection technologies: ECC Chipkill (for x4-based memory DIMMs; look for x4 in the DIMM description) Memory mirroring Memory sparing If memory mirroring is used, DIMMs must be installed in pairs (minimum of one pair per processor). Both DIMMs in a pair must be identical in type and size. If memory rank sparing is used, a minimum of one quad-rank DIMM or two single-rank or dual-rank DIMMs must be installed per populated channel. These DIMMs do not need to be identical. In rank sparing mode, one rank of a DIMM in each populated channel is reserved as spare memory. The size of a rank varies depending on the DIMMs installed. Table 5-75 lists the memory options available for the x440 server. DIMMs can be installed one at a time, but for performance reasons, install them in sets of four (one for each of the memory channels). A total of 48 DIMMs are the maximum number supported.
Table 5-75 Memory options for the x440 Part number Feature code Description Models where used
Registered DIMM (RDIMM) modules 49Y1406 49Y1407 49Y1559 8941 8947 A28Z 4 GB (1x 4 GB, 1Rx4, 1.35 V) PC3L-10600 CL9 ECC DDR3 1333 MHz LP RDIMM 4 GB (1x 4 GB, 2Rx8, 1.35 V) PC3L-10600 CL9 ECC DDR3 1333 MHz LP RDIMM 4 GB (1x 4 GB, 1Rx4, 1.5 V) PC3-12800 CL11 ECC DDR3 1600 MHz LP RDIMM -
283
Description 8 GB (1x 8 GB, 2Rx4, 1.5 V) PC3-12800 CL11 ECC DDR3 1600 MHz LP RDIMM 8 GB (1x 8 GB, 2Rx4, 1.35 V) PC3L-10600 CL9 ECC DDR3 1333 MHz LP RDIMM 16 GB (1x 16 GB, 2Rx4, 1.35 V) PC3L-10600 CL9 ECC DDR3 1333 MHz LP RDIMM 16 GB (1x 16 GB, 2Rx4, 1.5 V) PC3-12800 CL11 ECC DDR3 1600 MHz LP RDIMM
Load Reduced DIMM (LRDIMM) modules 49Y1567 90Y3105 A290 A291 16 GB (1x1 6 GB, 4Rx4, 1.35 V) PC3L-10600 CL9 ECC DDR3 1333 MHz LP LRDIMM 32 GB (1x 32 GB, 4Rx4, 1.35 V) PC3L-10600 CL9 ECC DDR3 1333 MHz LP LRDIMM -
10K SAS hard disk drives 00AD075 81Y9650 90Y8872 90Y8877 A48S A282 A2XD A2XC IBM 1.2TB 10K 6Gbps SAS 2.5'' G2HS HDD IBM 900GB 10K 6Gbps SAS 2.5" SFF HS HDD IBM 600GB 10K 6Gbps SAS 2.5" SFF G2HS HDD IBM 300GB 10K 6Gbps SAS 2.5" SFF G2HS HDD 2 2 2 2
10K and 15K SAS self-encrypting drives (SEDs)a 90Y8944 00AD085 81Y9662 A2ZK A48T A3EG IBM 146GB 15K 6Gbps SAS 2.5" SFF G2HS SED IBM 1.2TB 10K 6Gbps SAS 2.5'' G2HS SED IBM 900GB 10K 6Gbps SAS 2.5" SFF G2HS SED 2 2 2
284
IBM PureFlex System and IBM Flex System Products and Technology
Description IBM 600GB 10K 6Gbps SAS 2.5" SFF G2HS SED IBM 300GB 10K 6Gbps SAS 2.5" SFF G2HS SED IBM 300GB 10K 6Gbps SAS 2.5" SFF Slim-HS SED
Maximum supported 2 2 2
15K SAS hard disk drives 81Y9670 90Y8926 A283 A2XB IBM 300GB 15K 6Gbps SAS 2.5" SFF HS HDD IBM 146GB 15K 6Gbps SAS 2.5" SFF G2HS HDD 2 2
NL SATA drives 81Y9730 81Y9726 81Y9722 A1AV A1NZ A1NX IBM 1TB 7.2K 6Gbps NL SATA 2.5" SFF HS HDD IBM 500GB 7.2K 6Gbps NL SATA 2.5" SFF HS HDD IBM 250GB 7.2K 6Gbps NL SATA 2.5" SFF HS HDD 2 2 2
NL SAS drives 81Y9690 90Y8953 A1P3 A2XE IBM 1TB 7.2K 6Gbps NL SAS 2.5" SFF HS HDD IBM 500GB 7.2K 6Gbps NL SAS 2.5" SFF G2HS HDD 2 2
SAS-SSD Hybrid drive 00AD102 A4G7 IBM 600GB 10K 6Gbps SAS 2.5'' G2HS Hybrid 2
Enterprise SSDs 49Y6195 49Y6139 49Y6134 49Y6129 41Y8331 41Y8336 41Y8341 43W7718 00W1125 A4GH A3F0 A3EY A3EW A4FL A4FN A4FQ A2FN A3HR IBM 1.6TB SAS 2.5" MLC HS Enterprise SSD IBM 800GB SAS 2.5" MLC HS Enterprise SSD IBM 400GB SAS 2.5" MLC HS Enterprise SSD IBM 200GB SAS 2.5" MLC HS Enterprise SSD S3700 200GB SATA 2.5" MLC HS Enterprise SSD S3700 400GB SATA 2.5" MLC HS Enterprise SSD S3700 800GB SATA 2.5" MLC HS Enterprise SSD IBM 200GB SATA 2.5" MLC HS SSD IBM 100GB SATA 2.5" MLC HS Enterprise SSD 2 2 2 2 2 2 2 2 2
Enterprise Value SSDs 90Y8643 90Y8648 A2U3 A2U4 IBM 256GB SATA 2.5" MLC HS Enterprise Value SSD IBM 128GB SATA 2.5" MLC HS Enterprise Value SSD 2 2
a. Supports self-encrypting drive (SED) technology. For more information, see Self-Encrypting Drives for IBM System x at this website: http://www.redbooks.ibm.com/abstracts/tips0761.html?Open.
285
The following hardware kits are available: ServeRAID M5100 Series Enablement Kit for IBM Flex System x440 (46C9030) enables support for up to two 2.5-inch HDDs or SSDs in the hot-swap bays in the front of the server. It includes a CacheVault unit, which enables MegaRAID CacheVault flash cache protection. This enablement kit replaces the two standard 1-bay backplanes (which are attached through the system board to an onboard controller) with new 1-bay backplanes that attach to an included flex cable to the M5115 controller. It also includes an air baffle, which also serves as an attachment for the CacheVault unit. MegaRAID CacheVault flash cache protection uses NAND flash memory that is powered by a supercapacitor to protect data that is stored in the controller cache. This module eliminates the need for the lithium-ion battery that is commonly used to protect DRAM cache memory on PCI RAID controllers. To avoid the possibility of data loss or corruption during a power or server failure, CacheVault technology transfers the contents of the DRAM cache to NAND flash memory by using power from the supercapacitor. After the power is restored to the RAID controller, the saved data is transferred from the NAND flash memory back to the DRAM cache, which can then be flushed to disk. Tip: The Enablement Kit is only required if 2.5-inch drives are to be used. This kit is not required if you plan to install four or eight 1.8-inch SSDs only. 286
IBM PureFlex System and IBM Flex System Products and Technology
ServeRAID M5100 Series IBM Flex System Flash Kit for x440 (46C9031) enables support for up to four 1.8-inch SSDs. This kit replaces the two standard 1-bay backplanes with a two 2-bay backplanes that attach to an included flex cable to the M5115 controller. Because only SSDs are supported, a CacheVault unit is not required. Therefore, this kit does not have a supercapacitor. ServeRAID M5100 Series SSD Expansion Kit for IBM Flex System x440 (46C9032) enables support for up to four internal 1.8-inch SSDs. This kit includes two air baffles that can attach two 1.8-inch SSD attachment locations and flex cables for attachment to up to four 1.8-inch SSDs. Product-specific kits: These kits are specific for the x440 and cannot be used with the x240 or x220. Table 5-78 shows the kits that are required for each combination of drives. For example, if you plan to install eight 1.8-inch SSDs, you need the M5115 controller, the Flash Kit, and the SSD Expansion Kit.
Table 5-78 ServeRAID M5115 hardware kits Wanted drive support Maximum number of 2.5-inch drives 2 0 2 0 Maximum number of 1.8-inch SSDs 0 4 (front) 4 (internal) 8 (both) => => => => Components required ServeRAID M5115 90Y4390 Required Required Required Required Required Required Enablement kIt 46C9030 Required Required Required Required Flash Kit 46C9031 SSD Expansion Kit 46c9032
Figure 5-47 shows how the ServeRAID M5115 and the Enablement Kit are installed in the server to support two 2.5-inch drives with MegaRAID CacheVault flash cache protection (as shown in row 1 of Table 5-78).
ServeRAID M5115 controller (90Y4390) with ServeRAID M5100 Series Enablement Kit (46C9030)
ServeRAID M5115 controller
Figure 5-47 The ServeRAID M5115 and the Enablement Kit installed
287
Figure 5-48 shows how the ServeRAID M5115 and Flash and SSD Expansion Kits are installed in the server to support eight 1.8-inch solid-state drives (as shown in row 4 in Table 5-78 on page 287).
ServeRAID M5115 controller (90Y4390) with Flash Kit for x440 (46C9031) and SSD Expansion Kit for x440 (46C9032)
ServeRAID M5115 controller
SSD Expansion Kit: Four SSDs connectors on special air baffles above DIMMs (no CacheVault flash protection)
Figure 5-48 ServeRAID M5115 with Flash and SSD Expansion Kits installed
The eight SSDs are installed in the following locations: Four in the front of the system in place of the two 2.5-inch drive bays Four on trays above the memory banks The ServeRAID M5115 controller, 90Y4390, includes the following specifications: Eight internal 6 Gbps SAS/SATA ports. PCI Express 3.0 x8 host interface. 6 Gbps throughput per port. 800 MHz dual-core IBM PowerPC processor with LSI SAS2208 6 Gbps RAID on Chip (ROC) controller. Support for RAID levels 0, 1, 10, 5, 50 standard; support for RAID 6 and 60 with optional upgrade using 90Y4411. Optional onboard 1 GB data cache (DDR3 running at 1333 MHz) with optional flash backup (MegaRAID CacheVault technology) as part of the Enablement Kit 90Y4342. Support for SAS and SATA HDDs and SSDs. Support for intermixing SAS and SATA HDDs and SSDs. Mixing different types of drives in the same array (drive group) is not recommended. Support for self-encrypting drives (SEDs) with MegaRAID SafeStore. Optional support for SSD performance acceleration with MegaRAID FastPath and SSD caching with MegaRAID CacheCade Pro 2.0 (90Y4447). Support for up to 64 virtual drives, up to 128 drive groups, up to 16 virtual drives per one drive group, and up to 32 physical drives per one drive group. Support for logical unit number (LUN) sizes up to 64 TB. Configurable stripe size up to 1 MB. 288
IBM PureFlex System and IBM Flex System Products and Technology
Compliant with Disk Data Format (DDF) configuration on disk (COD). S.M.A.R.T. support. MegaRAID Storage Manager management software. Optional add-ons to the ServeRAID M5115 controller are RAID 6 support, SSD performance upgrade, and SSD caching enabler. The feature upgrades are as listed in Table 5-79. These upgrades are all Feature on Demand (FoD) license upgrades.
Table 5-79 Supported ServeRAID M5115 upgrade features Part number 90Y4410 90Y4412 90Y4447 Feature code A2Y1 A2Y2 A36G Description Maximum supporte d 1 1 1
ServeRAID M5100 Series RAID 6 Upgrade for IBM Flex System ServeRAID M5100 Series Performance Upgrade for IBM Flex System (MegaRAID FastPath) ServeRAID M5100 Series SSD Caching Enabler for IBM Flex System (MegaRAID CacheCade Pro 2.0)
The following features are available: RAID 6 Upgrade (90Y4410) Adds support for RAID 6 and RAID 60. This is a Feature on Demand license. Performance Upgrade (90Y4412) The Performance Upgrade for IBM Flex System (which is implemented by using the LSI MegaRAID FastPath software) provides high-performance I/O acceleration for SSD-based virtual drives by using a low-latency I/O path to increase the maximum I/O per second (IOPS) capability of the controller. This feature boosts the performance of applications with a highly random data storage access pattern, such as transactional databases. Part number 90Y4412 is a Feature on Demand license. SSD Caching Enabler for traditional hard disk drives (90Y4447) The SSD Caching Enabler for IBM Flex System (which is implemented by using the LSI MegaRAID CacheCade Pro 2.0) is designed to accelerate the performance of HDD arrays with only an incremental investment in SSD technology. The feature enables the SSDs to be configured as a dedicated cache to help maximize the I/O performance for transaction-intensive applications, such as databases and web serving. The feature tracks data storage access patterns and identifies the most frequently accessed data. The hot data is then automatically stored on the SSDs that are assigned as a dedicated cache pool on the ServeRAID controller. Part number 90Y4447 is a Feature on Demand license. This feature requires at least one SSD drive be installed.
289
The 1.8-inch SSDs supported with the ServeRAID M5115 controller are listed in Table 5-80.
Table 5-80 Supported 1.8-inch SSDs Part number 43W7746 43W7726 Feature code 5420 5428 Description IBM 200GB SATA 1.8" MLC SSD IBM 50GB SATA 1.8" MLC SSD Maximum supported 8 8
The Fabric Connector enables port 1 of each embedded 10 Gb controller to be routed to I/O module bay 1 and port 2 of each controller to be routed to I/O module bay 2. The Fabric Connectors can be unscrewed and removed, if required, to allow the installation of an I/O adapter on I/O connector 1 and 3. The Embedded 10Gb controllers are based on the Emulex BladeEngine 3 (BE3), which is a single-chip, dual-port 10 Gigabit Ethernet (10GbE) Ethernet Controller. The Embedded 10Gb controller includes the following features: PCI-Express Gen2 x8 host bus interface Supports multiple virtual NIC (vNIC) functions TCP/IP Offload Engine (TOE enabled) SRIOV capable RDMA over TCP/IP capable iSCSI and FCoE upgrade offering through FoD Table 5-81 on page 291 lists the ordering information for the IBM Flex System Embedded 10Gb Virtual Fabric Upgrade, which enables the iSCSI and FCoE support on the Embedded 10Gb Virtual Fabric controller. To upgrade both controllers, you need two FoD licenses.
290
IBM PureFlex System and IBM Flex System Products and Technology
Table 5-81 Feature on Demand upgrade for FCoE and iSCSI support Part number 90Y9310 Feature code A2TD Description IBM Virtual Fabric Advanced Software Upgrade (LOM) Maximum supported 2
1 2 3 4
Figure 5-50 Location of the I/O adapters in the IBM Flex System x440 Compute Node
Important: A second processor must be installed so that I/O adapter slots 3 and 4 in the x440 compute node can be used because the PCIe lanes that used to drive I/O slots 3 and 4 are routed to processors 2 and 4, as shown in Figure 5-46 on page 280.
291
All I/O adapters are the same shape and can be used in any available slot. A compatible switch or pass-through module must be installed in the corresponding I/O bays in the chassis, as indicated in Table 5-82. Installing two switches means that all ports of the adapter are enabled, which improves performance and network availability.
Table 5-82 Adapter to I/O bay correspondence I/O adapter slot in the x440 Slot 1 Port on the adapter Port 1 Port 2 Port 3 (for 4-port cards) Port 4 (for 4-port cards) Slot 2 Port 1 Port 2 Port 3 (for 4-port cards) Port 4 (for 4-port cards) Slot 3 Port 1 Port 2 Port 3 (for 4-port cards) Port 4(for 4-port cards) Slot 4 Port 1 Port 2 Port 3 (for 4-port cards) Port 4 (for 4-port cards) Corresponding I/O module bay in the chassis Module bay 1 Module bay 2 Module bay 1 Module bay 2 Module bay 3 Module bay 4 Module bay 3 Module bay 4 Module bay 1 Module bay 2 Module bay 1 Module bay 2 Module bay 3 Module bay 4 Module bay 3 Module bay 4
292
IBM PureFlex System and IBM Flex System Products and Technology
Figure 5-51 shows the location of the switch bays in the rear of the Enterprise Chassis.
I/O module bay 1 I/O module bay 3 I/O module bay 2 I/O module bay 4
Figure 5-52 shows how the two port adapters are connected to switches that are installed in the I/O Module bays in an Enterprise Chassis.
. Switch . . bay 1 . . .
. Switch . . bay 3 . . .
. Switch . . bay 2 . . .
. Switch . . bay 4 . . .
Figure 5-52 Logical layout of the interconnects between I/O adapters and I/O module
293
40Gb Ethernet 90Y3482 A3HK IBM Flex System EN6132 2-port 40Gb Ethernet adapter 2 4
10Gb Ethernet 90Y3554 90Y3558 90Y3466 A1R1 A1R0 A1QY IBM Flex System CN4054 10Gb Virtual Fabric adapter IBM Flex System CN4054 Virtual Fabric adapter (Software Upgrade) (Feature on Demand to provide FCoE and iSCSI support) IBM Flex System EN4132 2-port 10Gb Ethernet adapter 4 License 2 4 4 4
1Gb Ethernet 49Y7900 InfiniBand 90Y3454 A1QZ IBM Flex System IB6132 2-port FDR InfiniBand adapter 2 4 A10Y IBM Flex System EN2024 4-port 1Gb Ethernet adapter 4 4
a. For x4x models with two Embedded 10Gb Virtual Fabric controllers standard, the Compute Node Fabric Connectors occupy the same space as the I/O adapters in I/O slots 1 and 3, so you must remove the Fabric Connectors if you plan to install adapters in those I/O slots
294
IBM PureFlex System and IBM Flex System Products and Technology
a. For x4x models with two Embedded 10Gb Virtual Fabric controllers standard, the Compute Node Fabric Connectors occupy the same space as the I/O adapters in I/O slots 1 and 3, so you must remove the Fabric Connectors if you plan to install adapters in those I/O slots
a. The Blank USB Memory Key requires the download of the VMware vSphere (ESXi) Hypervisor with IBM Customization image, which is available at this website: http://ibm.com/systems/x/os/vmware/
There are two types of USB keys: preload keys or blank keys. Blank keys allow you to download an IBM customized version of ESXi and load it onto the key. Each server supports one or two keys to be installed, but only the following combinations: One preload key (keys that are preloaded at the factory) One blank key (a key that you download the customized image) One preload key and one blank key Two blank keys
295
Two preload keys is an unsupported combination.Installing two preload keys prevents ESXi from booting. This is similar to the error as described at this website: http://kb.vmware.com/kb/1035107 Having two keys that are installed provides a backup boot device. Both devices are listed in the boot menu, which allows you to boot from either device or to set one as a backup in case the first one becomes corrupted
To illuminate the light path diagnostics LEDs, power off the compute node, slide it out of the chassis, and press the power button. The power button doubles as the light path diagnostics remind button when the server is removed from the chassis. The meanings of the LEDs in the light path diagnostics panel are listed in Table 5-86.
Table 5-86 Light path diagnostic panel LEDs LED LP S BRD Meaning The light path diagnostics panel is operational. A system board error is detected.
296
IBM PureFlex System and IBM Flex System Products and Technology
Meaning A mismatch occurred between the processors, DIMMs, or HDDs within the configuration reported by POST. A non-maskable interrupt (NMI) occurred. An over-temperature condition occurred that was critical enough to shut down the server. A memory fault occurred. The corresponding DIMM error LEDs on the system board are also lit. A fault is detected in the adjacent expansion unit (if installed).
Remote management
The server contains an IBM Integrated Management Module II (IMMv2), which interfaces with the advanced management module in the chassis. The combination of these two components provides advanced service-processor control, monitoring, and an alerting function. If an environmental condition exceeds a threshold or if a system component fails, LEDs on the system board are lit to help you diagnose the problem, the error is recorded in the event log, and you are alerted to the problem. A virtual presence capability comes standard for remote server management. Remote server management is provided through the following industry-standard interfaces: Intelligent Platform Management Interface (IPMI) Version 2.0 Simple Network Management Protocol (SNMP) Version 3 Common Information Model (CIM) Web browser The server also supports virtual media and remote control features, which provide the following functions: Remotely viewing video with graphics resolutions up to 1600 x 1200 at 75 Hz with up to 23 bits per pixel, regardless of the system state. Remotely accessing the server by using the keyboard and mouse from a remote client. Mapping the CD or DVD drive, diskette drive, and USB flash drive on a remote client, and mapping ISO and diskette image files as virtual drives that are available for use by the server. Uploading a diskette image to the IMM2 memory and mapping it to the server as a virtual drive. Capturing blue-screen errors.
297
Red Hat Enterprise Linux 6 Server x64 Edition SUSE Linux Enterprise Server 10 for AMD64/EM64T SUSE Linux Enterprise Server 11 for AMD64/EM64T SUSE Linux Enterprise Server 11 with Xen for AMD64/EM64T VMware ESX 4.1 VMware ESXi 4.1 VMware vSphere 5 VMware vSphere 5.1 Support by some of these operating system versions is after the date of initial availability. Check the IBM ServerProven website for the latest information about the specific versions and service levels that are supported and any other prerequisites at this website: http://www.ibm.com/systems/info/x86servers/serverproven/compat/us/nos/matrix.shtml
5.6.1 Specifications
The IBM Flex System p260 Compute Node is a half-wide, Power Systems compute node with the following characteristics: Two POWER7 or POWER7+ processor sockets Sixteen memory slots Two I/O adapter slots An option for up to two internal drives for local storage
298
IBM PureFlex System and IBM Flex System Products and Technology
The IBM Flex System p260 Compute Node includes the specifications that are shown in Table 5-87.
Table 5-87 IBM Flex System p260 Compute Node specifications Components Model numbers Form factor Chassis support Processor Specification IBM Flex System p24L Compute Node: 1457-7FL IBM Flex System p260 Compute Node: 7895-22X, 7895-23X and 7895-23A Half-wide compute node. IBM Flex System Enterprise Chassis. p24L: Two IBM POWER7 processors p260: Two IBM POWER7 (model 22X) or POWER7+ (models 23A and 23X) processors. POWER7 processors: Each processor is a single-chip module (SCM) that contains either eight cores (up to 3.55 GHz and 32 MB L3 cache) or four cores (3.3 GHz and 16 MB L3 cache). Each processor has 4 MB L3 cache per core. Integrated memory controller in each processor, each with four memory channels. Each memory channel operates at 6.4 Gbps. There is one GX++ I/O bus connection per processor. Supports SMT4 mode, which enables four instruction threads to run simultaneously per core. Uses 45 nm fabrication technology. POWER7+ processors: Each processor is a single-chip module (SCM) that contains either eight cores (up to 4.1 GHz or 3.6 GHz and 80 MB L3 cache), four cores (4.0 GHz and 40 MB L3 cache) or two cores (4.0 GHz and 20 MB L3 cache). Each processor has 10 MB L3 cache per core. There is an integrated memory controller in each processor, each with four memory channels. Each memory channel operates at 6.4 Gbps. There is one GX++ I/O bus connection per processor. Supports SMT4 mode, which enables four instruction threads to run simultaneously per core. Uses 32 nm fabrication technology. Chipset Memory IBM P7IOC I/O hub. 16 DIMM sockets. RDIMM DDR3 memory supported. Integrated memory controller in each processor, each with four memory channels. Supports IBM Active Memory Expansion with AIX 6.1 or later. All DIMMs operate at 1066 MHz. Both LP (low profile) and VLP (very low profile) DIMMs supported, although only VLP DIMMs are supported if internal HDDs are configured. The use of 1.8-inch solid-state drives allows the use of LP and VLP DIMMs. 512 GB using 16x 32 GB DIMMs. ECC, Chipkill. Two 2.5-inch non-hot-swap drive bays that support 2.5-inch SAS HDD or 1.8-inch SATA SSD drives. If LP DIMMs are installed, only 1.8-inch SSDs are supported. If VLP DIMMs are installed, both HDDs and SSDs are supported. An HDD and an SSD cannot be installed together. 1.8 TB using two 900 GB SAS HDD drives, or 354 GB using two 177 GB SSD drives. RAID support by using the operating system. None standard. Optional 1 Gb or 10 Gb Ethernet adapters.
299
Components PCI Expansion slots Ports Systems management Security features Video Limited warranty Operating systems supported Service and support
Specification Two I/O connectors for adapters. PCI Express 2.0 x16 interface. One external USB port. FSP, Predictive Failure Analysis, light path diagnostics panel, automatic server restart, Serial over LAN support. IPMI compliant. Support for IBM Flex System Manager, and IBM Systems Director. Power-on password, selectable boot sequence. None. Remote management by using Serial over LAN and IBM Flex System Manager. 3-year customer-replaceable unit and onsite limited warranty with 9x5/NBD. IBM AIX, IBM i, and Linux.
Optional service upgrades are available through IBM ServicePacs: 4-hour or 2-hour response time, 8-hour fix time, 1-year or 2-year warranty extension, remote technical support for IBM hardware and selected IBM and OEM software. Width: 215 mm (8.5), height: 51 mm (2.0), depth: 493 mm (19.4). Maximum configuration: 7.0 kg (15.4 lb).
Dimensions Weight
300
IBM PureFlex System and IBM Flex System Products and Technology
16 DIMM slots
(HDDs are mounted on the cover, located over the memory DIMMs)
Figure 5-54 Layout of the IBM Flex System p260 Compute Node
301
Power button
Figure 5-55 Front panel of the IBM Flex System p260 Compute Node
The USB port on the front of the Power Systems compute nodes is useful for various tasks. These tasks include out-of-band diagnostic procedures, hardware RAID setup, operating system access to data on removable media, and local OS installation. It might be helpful to obtain a USB optical (CD or DVD) drive for these purposes, in case the need arises. Tip: There is no optical drive in the IBM Flex System Enterprise Chassis.
302
IBM PureFlex System and IBM Flex System Products and Technology
The power-control button on the front of the server (see Figure 5-55 on page 302) has the following functions: When the system is fully installed in the chassis: Use this button to power the system on and off. When the system is removed from the chassis: Use this button to illuminate the light path diagnostic panel on the top of the front bezel, as shown in Figure 5-56.
The LEDs on the light path panel indicate the status of the following devices: LP: Light Path panel power indicator S BRD: System board LED (might indicate trouble with processor or MEM, too) MGMT: Flexible Support Processor (or management card) LED D BRD: Drive or direct access storage device (DASD) board LED DRV 1: Drive 1 LED (SSD 1 or HDD 1) DRV 2: Drive 2 LED (SSD 2 or HDD 2) If problems occur, the light path diagnostics LEDs help with identifying the subsystem involved. To illuminate the LEDs with the compute node removed, press the power button on the front panel. Pressing this button temporarily illuminates the LEDs of the troubled subsystem to direct troubleshooting efforts. Typically, you can obtain this information from the IBM Flex System Manager or Chassis Management Module before you remove the node. However, having the LEDs helps with repairs and troubleshooting if onsite assistance is needed. For more information about the front panel and LEDs, see IBM Flex System p260 and p460 Compute Node Installation and Service Guide, which is available at this website: http://www.ibm.com/support
303
DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM
SMI
HDDs/SSDs
SMI
SMI
POWER7 Processor 0
USB controller
To front panel
SMI
4 bytes each
SMI
I/O connector 1
SMI
POWER7 Processor 1
SMI
FSP
Phy
Gb Ethernet ports
Figure 5-57 IBM Flex System p260 Compute Node and IBM Flex System p24L Compute Node block diagram
This diagram shows the two CPU slots, with eight memory slots for each processor. Each processor is connected to a P7IOC I/O hub, which connects to the I/O subsystem (I/O adapters, local storage). At the bottom, you can see a representation of the service processor (FSP) architecture.
304
IBM PureFlex System and IBM Flex System Products and Technology
5.6.7 Processor
The IBM POWER7 processor represents a leap forward in technology and associated computing capability. The multi-core architecture of the POWER7 processor is matched with a wide range of related technologies to deliver leading throughput, efficiency, scalability, and reliability, availability, and serviceability (RAS). Although the processor is an important component in servers, many elements and facilities must be balanced across a server to deliver maximum throughput. As with previous generations, the design philosophy for POWER7 processor-based systems is system-wide balance. The POWER7 processor plays an important role in this balancing.
IBM Flex System p260 Compute Node - 7895-23X EPRD EPRB EPRA 4 8 8 2 2 2 8 16 16 4.0 GHz 3.6 GHz 4.1 GHz 40 MB (10 MB per core) 80 MB (10 MB per core) 80 MB (10 MB per core)
IBM Flex System p260 Compute Node - 7895-22X EPR1 EPR3 EPR5 4 8 8 2 2 2 8 16 16 3.3 GHz 3.2 GHz 3.55 GHz 16 MB (4 MB per core) 32 MB (4 MB per core) 32 MB (4 MB per core)
IBM Flex System p260 Compute Node - 7895-23A EPRC 2 2 4 4.0 GHz 20 MB (10 MB per core)
IBM Flex System p24L Compute Node EPR8 EPR9 EPR7 8 8 6 2 2 2 16 16 12 3.2 GHz 3.55 GHz 3.7 GHz 32 MB (4 MB per core) 32 MB (4 MB per core) 24 MB (4 MB per core)
To optimize software licensing, you can deconfigure or disable one or more cores. The feature is listed in Table 5-89.
Table 5-89 Unconfiguration of cores for p260 and p24L Feature code 2319 Description Factory Deconfiguration of 1-core Minimum 0 Maximum 1 less than the total number of cores (For EPR5, the maximum is 7)
305
POWER7 architecture
IBM uses innovative methods to achieve the required levels of throughput and bandwidth. Areas of innovation for the POWER7 processor and POWER7 processor-based systems include (but are not limited to) the following elements: On-chip L3 cache that is implemented in embedded dynamic random-access memory (eDRAM) Cache hierarchy and component innovation Advances in memory subsystem Advances in off-chip signaling The superscalar POWER7 processor design also provides the following capabilities: Binary compatibility with the prior generation of POWER processors Support for PowerVM virtualization capabilities, including PowerVM Live Partition Mobility to and from IBM POWER6 and IBM POWER6+ processor-based systems Figure 5-58 shows the POWER7 processor die layout with major areas identified: Eight POWER7 processor cores, L2 cache, L3 cache and chip power bus interconnect, SMP links, GX++ interface, and integrated memory controller.
GX++ Bridge
Memory Controller
C1 Core L2
C1 Core L2
C1 Core L2
C1 Core L2
SMP
Figure 5-58 POWER7 processor architecture
POWER7+ architecture
The POWER7+ architecture builds on the POWER7 architecture. IBM uses innovative methods to achieve the required levels of throughput and bandwidth. Areas of innovation for the POWER7+ processor and POWER7+ processor-based systems include (but are not limited to) the following elements: On-chip L3 cache implemented in embedded dynamic random access memory (eDRAM) Cache hierarchy and component innovation Advances in memory subsystem Advances in off-chip signaling Advances in RAS features such as power-on reset and L3 cache dynamic column repair
306
IBM PureFlex System and IBM Flex System Products and Technology
Memory Buffers
The superscalar POWER7+ processor design also provides the following capabilities: Binary compatibility with the prior generation of POWER processors Support for PowerVM virtualization capabilities, including PowerVM Live Partition Mobility to and from POWER6, POWER6+, and POWER7 processor-based systems Figure 5-59 shows the POWER7+ processor die layout with the following major areas identified: Eight POWER7+ processor cores L2 cache L3 cache Chip power bus interconnect SMP links GX++ interface Memory controllers I/O links
307
Components Processor cores Max execution threads core/chip L2 cache per core/per chip On-chip L3 cache per core / per chip DDR3 memory controllers Compatibility
5.6.8 Memory
Each POWER7 processor has an integrated memory controller. Industry-standard DDR3 RDIMM technology is used to increase the reliability, speed, and density of the memory subsystems.
Generally, use a minimum of 2 GB of RAM per core. The functional minimum memory configuration for the system is 4 GB (2x2 GB). However, this configuration is not sufficient for reasonable production use of the system.
308
IBM PureFlex System and IBM Flex System Products and Technology
Table 5-92 lists the available memory options for the p260 and p24L.
Table 5-92 Supported memory DIMMs - Power Systems compute nodes Part number 78P1011 78P0501 78P0502 78P1917 78P0639 78P1915 78P1539 Feature code EM04 8196 8199 EEMD 8145 EEME EEMF Description 2x 2 GB DDR3 RDIMM 1066 MHz 2x 4 GB DDR3 RDIMM 1066 MHz 2x 8 GB DDR3 RDIMM 1066 MHz 2x 8 GB DDR3 RDIMM 1066 MHz 2x 16 GB DDR3 RDIMM 1066 MHz 2x 16 GB DDR3 RDIMM 1066 MHz 2x 32 GB DDR3 RDIMM 1066 MHz Form factor LPa VLP VLP VLP LP
a
LPa LPa
a. If 2.5-inch HDDs are installed, low-profile DIMM features cannot be used (EM04, 8145, EEME and EEMF cannot be used).
Requirement: Because of the design of the on-cover storage connections, if you want to use 2.5-inch HDDs, you must use VLP DIMMs (4 GB or 8 GB). The cover cannot close properly if LP DIMMs and SAS HDDs are configured in the same system. This mixture physically obstructs the cover. However, SSDs and LP DIMMs can be used together. For more information, see 5.6.10, Storage on page 313. There are 16 buffered DIMM slots on the p260 and the p24L, as shown in Figure 5-60.
SMI
DIMM 1 (P1-C1) DIMM 2 (P1-C2) DIMM 3 (P1-C3) DIMM 4 (P1-C4) DIMM 5 (P1-C5) DIMM 6 (P1-C6) DIMM 7 (P1-C7) DIMM 8 (P1-C8) DIMM 9 (P1-C9) DIMM 10 (P1-C10) DIMM 11 (P1-C11) DIMM 12 (P1-C12) DIMM 13 (P1-C13) DIMM 14 (P1-C14) DIMM 15 (P1-C15) DIMM 16 (P1-C16)
SMI
POWER7 Processor 0
SMI
SMI
SMI
SMI
POWER7 Processor 1
SMI
SMI
Figure 5-60 Memory DIMM topology (IBM Flex System p260 Compute Node)
309
The following memory-placement rules must be adhered to: Install DIMM fillers in unused DIMM slots to ensure effective cooling. Install DIMMs in pairs. Both must be the same size. Both DIMMs in a pair must be the same size, speed, type, and technology. Otherwise, you can mix compatible DIMMs from multiple manufacturers. Install only supported DIMMs, as described on the IBM ServerProven website: http://www.ibm.com/servers/eserver/serverproven/compat/us/ Table 5-93 shows the required placement of memory DIMMs for the p260 and the p24L, depending on the number of DIMMs installed.
Table 5-93 DIMM placement - p260 and p24L Number of DIMMs Processor 0 Processor 1
DIMM 10
DIMM 11
DIMM 12
DIMM 13
DIMM 14
DIMM 15 x
2 4 6 8 10 12 14 16
x x x x x x x x x x x x x x x x
x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x
310
IBM PureFlex System and IBM Flex System Products and Technology
DIMM 16
DIMM 1
DIMM 2
DIMM 3
DIMM 4
DIMM 5
DIMM 6
DIMM 7
DIMM 8
DIMM 9
Active Memory Expansion uses processor resources to compress and extract memory contents. The trade-off of memory capacity for processor cycles can be an excellent choice. However, the degree of expansion varies based on how compressible the memory content is. Have adequate spare processor capacity available for the compression and decompression. Tests in IBM laboratories that used sample workloads showed excellent results for many workloads in terms of memory expansion per added processor that was used. Other test workloads had more modest results. You have a great deal of control over Active Memory Expansion usage. Each individual AIX partition can turn on or turn off Active Memory Expansion. Control parameters set the amount of expansion that is wanted in each partition to help control the amount of processor capacity that is used by the Active Memory Expansion function. An IBM Public License (IPL) is required for the specific partition that turns memory expansion on or off. After the memory expansion is turned on, there are monitoring capabilities in standard AIX performance tools, such as lparstat, vmstat, topas, and svmon. Figure 5-61 represents the percentage of processor that is used to compress memory for two partitions with various profiles. The green curve corresponds to a partition that has spare processing power capacity. The blue curve corresponds to a partition constrained in processing power.
2 % CPU utilization for expansion Very cost effective 1
1 = Plenty of spare CPU resource available 2 = Constrained CPU resource already running at significant utilization
Both cases show the following knee of the curve relationships for processor resources that are required for memory expansion: Busy processor cores do not have resources to spare for expansion. The more memory expansion that is done, the more processor resources are required. The knee varies, depending on how compressible the memory contents are. This variability demonstrates the need for a case-by-case study to determine whether memory expansion can provide a positive return on investment. To help you perform this study, a planning tool is included with AIX 6.1 Technology Level 4 or later. You can use the tool to sample actual workloads and estimate both how expandable the partition memory is and how much processor resource is needed. Any Power System model runs the planning tool.
311
Figure 5-62 shows an example of the output that is returned by this planning tool. The tool outputs various real memory and processor resource combinations to achieve the wanted effective memory and proposes one particular combination. In this example, the tool proposes to allocate 58% of a processor core, to benefit from 45% extra memory capacity.
Active Memory Expansion Modeled Statistics: ----------------------Modeled Expanded Memory Size : 8.00 GB Expansion Factor --------1.21 1.31 1.41 1.51 1.61 True Memory Modeled Size -------------6.75 GB 6.25 GB 5.75 GB 5.50 GB 5.00 GB Modeled Memory Gain ----------------1.25 GB [ 19%] 1.75 GB [ 28%] 2.25 GB [ 39%] 2.50 GB [ 45%] 3.00 GB [ 60%] CPU Usage Estimate ----------0.00 0.20 0.35 0.58 1.46
Active Memory Expansion Recommendation: --------------------The recommended AME configuration for this workload is to configure the LPAR with a memory size of 5.50 GB and to configure a memory expansion factor of 1.51. This will result in a memory expansion of 45% from the LPAR's current memory size. With this configuration, the estimated CPU usage due to Active Memory Expansion is approximately 0.58 physical processors, and the estimated overall peak CPU resource required for the LPAR is 3.72 physical processors. Figure 5-62 Output from the AIX Active Memory Expansion planning tool
For more information about this topic, see the white paper, Active Memory Expansion: Overview and Usage Guide, which is available at this website: http://www.ibm.com/systems/power/hardware/whitepapers/am_exp.html
312
IBM PureFlex System and IBM Flex System Products and Technology
5.6.10 Storage
The p260 and p24L has an onboard SAS controller that can manage up to two non-hot-pluggable internal drives. Both 2.5-inch HDDs and 1.8-inch SSDs are supported. The drives attach to the cover of the server, as shown in Figure 5-63.
Figure 5-63 The IBM Flex System p260 Compute Node showing HDD location on top cover
2.5-inch SAS HDDs 7069 8274 None 42D0627 Top cover with HDD connectors for the p260 and p24L 300 GB 10K RPM non-hot-swap 6 Gbps SAS
313
Description 600 GB 10K RPM non-hot-swap 6 Gbps SAS 900 GB 10K RPM non-hot-swap 6 Gbps SAS
1.8-inch SSDs 7068 8207 No drives 7067 None Top cover for no drives on the p260 and p24L None 74Y9114 Top cover with SSD connectors for the p260 and p24L 177 GB SATA non-hot-swap SSD
As shown in Figure 5-63 on page 313, the local drives (HDD or SDD) are mounted to the top cover of the system. When you order your p260 or p24L, select the cover that is appropriate for your system (SSD, HDD, or no drives).
314
IBM PureFlex System and IBM Flex System Products and Technology
The connection for the covers drive interposer on the system board is shown in Figure 5-65.
Figure 5-65 Connection for drive interposer card mounted to the system cover
RAID capabilities
Disk drives and SSDs in the p260 and p24L can be used to implement and manage various types of RAID arrays. They can do so in operating systems that are on the ServerProven list. For the compute node, you must configure the RAID array through the smit sasdam command, which is the SAS RAID Disk Array Manager for AIX. The AIX Disk Array Manager is packaged with the Diagnostics utilities on the Diagnostics CD. Use the smit sasdam command to configure the disk drives for use with the SAS controller. The diagnostics CD can be downloaded in ISO file format from this website: http://www14.software.ibm.com/webapp/set2/sas/f/diags/download/ For more information, see Using the Disk Array Manager in the Systems Hardware Information Center at this website: http://publib.boulder.ibm.com/infocenter/systems/scope/hw/index.jsp?topic=/p7ebj/s asusingthesasdiskarraymanager.htm Tip: Depending on your RAID configuration, you might have to create the array before you install the operating system in the compute node. Before you can create a RAID array, reformat the drives so that the sector size of the drives changes from 512 bytes to 528 bytes. If you decide later to remove the drives, delete the RAID array before you remove the drives. If you decide to delete the RAID array and reuse the drives, you might need to reformat the drives. Change the sector size of the drives from 528 bytes to 512 bytes.
315
Slot 1 requirements: You must have one of the following I/O adapters installed in slot 1 of the Power Systems compute nodes: EN4054 4-port 10Gb Ethernet Adapter (Feature Code #1762) EN2024 4-port 1Gb Ethernet Adapter (Feature Code #1763) IBM Flex System CN4058 8-port 10Gb Converged Adapter (#EC24) In the p260 and p24L, the I/O is controlled by two P7-IOC I/O controller hub chips. This configuration provides additional flexibility when assigning resources within Virtual I/O Server (VIOS) to specific Virtual Machine/LPARs. Table 5-95 shows the available I/O adapter cards.
Table 5-95 Supported I/O adapters for the p260 and p24L Feature code 1762a 1763a EC24a EC26 1764 EC23 EC2E 1761 Description IBM Flex System EN4054 4-port 10Gb Ethernet Adapter IBM Flex System EN2024 4-port 1Gb Ethernet Adapter IBM Flex System CN4058 8-port 10Gb Converged adapter IBM Flex System EN4132 2-port 10Gb RoCE adapter IBM Flex System FC3172 2-port 8Gb FC Adapter IBM Flex System FC5052 2-port 16Gb FC adapter IBM Flex System FC5054 4-port 16Gb FC adapter IBM Flex System IB6132 2-port QDR InfiniBand Adapter Number of ports 4 4 8 2 2 2 4 2
a. At least one 10 Gb (#1762) or 1 Gb (#1763) Ethernet adapter must be configured in each server.
316
IBM PureFlex System and IBM Flex System Products and Technology
Anchor card
As shown in Figure 5-66, the anchor card contains the vital product data chip that stores system-specific information. The pluggable anchor card provides a means for this information to be transferable from a faulty system board to the replacement system board. Before the service processor knows what system it is on, it reads the vital product data chip to obtain system information. The vital product data chip includes information such as system type, model, and serial number.
317
AIX V7.1 with the 7100-00 Technology Level with Service Pack 6, or later AIX V6.1 with the 6100-07 Technology Level, with Service Pack 3 with APAR IV14283 AIX V6.1 with the 6100-07 Technology Level, with Service Pack 4, or later AIX V6.1 with the 6100-06 Technology Level with Service Pack 8, or later AIX V5.3 with the 5300-12 Technology Level with Service Pack 6, or later1 IBM i 6.1 with i 6.1.1 machine code, or later IBM i 7.1, or later Novell SUSE Linux Enterprise Server 11 Service Pack 2 for POWER Red Hat Enterprise Linux 5.7, for POWER, or later Red Hat Enterprise Linux 6.2, for POWER, or later VIOS 2.2.1.4, or later The IBM Flex System p260 Compute Node (model 23X) supports the following operating systems: IBM i 6.1 with i 6.1.1 machine code or later IBM i 7.1 or later VIOS 2.2.2.0 or later AIX V7.1 with the 7100-02 Technology Level or later AIX V6.1 with the 6100-08 Technology Level or later Novell SUSE Linux Enterprise Server 11 Service Pack 2 for POWER Red Hat Enterprise Linux 5.7, for POWER, or later Red Hat Enterprise Linux 6.2, for POWER, or later The IBM Flex System p260 Compute Node (model 23A) supports the following operating systems: AIX V7.1 with the 7100-02 Technology Level with Service Pack 3 or later AIX V6.1 with the 6100-08 Technology Level with Service Pack 3 or later VIOS 2.2.2.3 or later IBM i 6.1 with i 6.1.1 machine code, or later IBM i 7.1 TR3 or later SUSE Linux Enterprise Server 11 Service Pack (SP) 2 for POWER Red Hat Enterprise Linux 6.4 for POWER OS support: Support by some of these operating system versions are post generally availability. For more information about the specific versions and service levels supported and any other prerequisites, see this website: http://www.ibm.com/systems/info/x86servers/serverproven/compat/us/nos/matrix.sh tml
318
IBM PureFlex System and IBM Flex System Products and Technology
5.7.6, System architecture on page 324 5.7.7, IBM POWER7+ processor on page 325 5.7.8, Memory subsystem on page 327 5.7.9, Active Memory Expansion feature on page 329 5.7.10, Storage on page 329 5.7.11, I/O expansion on page 333 5.7.12, System management on page 333 5.7.13, Operating system support on page 334
5.7.1 Specifications
The IBM Flex System p270 Compute Node is a half-wide, Power Systems compute node with the following characteristics: Two POWER7+ dual-chip module (DCM) processor sockets Sixteen memory slots Two I/O adapter slots plus support for the IBM Flex System Dual VIOS Adapter An option for up to two internal drives for local storage The p270 has the specifications that are shown in Table 5-96.
Table 5-96 Specifications for p270 Components Model number Form factor Chassis support Processor Specification 7954-24X Standard-width compute node IBM Flex System Enterprise Chassis Two IBM POWER7+ Dual Chip Modules. Each Dual Chip Module (DCM) contains two processor chips, each with six cores (24 cores total). Cores have a frequency of 3.1 or 3.4 GHz and each core has 10 MB of L3 cache (240 MB L3 cache total). Integrated memory controllers with four memory channels from each DCM. Each memory channel operates at 6.4 Gbps. One GX++ I/O bus connection per processor. Supports SMT4 mode, which enables four instruction threads to run simultaneously per core. Uses 32 nm fabrication technology. IBM P7IOC I/O hub. 16 DIMM sockets. RDIMM DDR3 memory is supported. Integrated memory controller in each processor, each with four memory channels. Supports Active Memory Expansion with AIX V6.1 or later. All DIMMs operate at 1066 MHz. Both LP (low profile) and VLP (very low profile) DIMMs are supported, although only VLP DIMMs are supported if internal HDDs are configured. The usage of 1.8-inch solid-state drives allows the use of LP and VLP DIMMs. 512 GB using 16x 32 GB DIMMs. ECC, Chipkill. Two 2.5-inch non-hot-swap drive bays supporting 2.5-inch SAS HDD or 1.8-inch SATA SSD drives. If LP DIMMs are installed, then only 1.8-inch SSDs are supported. If VLP DIMMs are installed, then both HDDs and SSDs are supported. An HDD and an SSD cannot be installed together. 1.8 TB using two 900 GB SAS HDD drives, or 354 GB using two 177 GB SSD drives.
Chipset Memory
319
Specification IBM ObsidianE SAS controller embedded on system board connects to the two local drive bays. Supports 3 Gbps SAS with a PCIe 2.0 x8 host interface. Supports RAID 0 and RAID 10 with two drives. A second Obsidian SAS controller is available through the optional IBM Flex System Dual VIOS adapter. When the Dual VIOS adapter is installed, each SAS controller controls one drive. Without the Dual VIOS adapter installed: RAID 0 and RAID 10 (two drives) With the Dual VIOS adapter installed: RAID 0 (one drive to each SAS controller) None standard. Optional 1Gb or 10Gb Ethernet adapters. Two I/O connectors for adapters. PCIe 2.0 x16 interface. One external USB port. FSP, Predictive Failure Analysis, light path diagnostics panel, automatic server restart, Serial over LAN support. IPMI compliant. Support for IBM Flex System Manager, and IBM Systems Director. Optional support for a Hardware Management Console (HMC) or an Integrated Virtualization Manager (IVM) console. FSP password, selectable boot sequence. None. Remote management through Serial over LAN and IBM Flex System Manager. 3-year customer-replaceable unit and onsite limited warranty with 9x5/NBD. IBM AIX, IBM i, and Linux. See 5.7.13, Operating system support on page 334 for details. Optional service upgrades are available through IBM ServicePac offerings: 4-hour or 2-hour response time, 8-hour fix time, 1-year or 2-year warranty extension, remote technical support for IBM hardware and selected IBM and OEM software. Width: 215mm (8.5), height 51mm (2.0), depth 493mm (19.4). Maximum configuration: 7.7 kg (17.0 lb).
RAID support Network interfaces PCI Expansion slots Ports Systems management
Security features Video Limited warranty Operating systems supported Service and support
Dimensions Weight
320
IBM PureFlex System and IBM Flex System Products and Technology
IBM light path diagnostics USB 2.0 port Figure 5-67 shows the system board layout of the IBM Flex System p270 Compute Node. POWER7+ Dual Chip Module Two I/O adapter connectors I/O adapter 1
16 DIMM slots
(Disks are mounted on the cover, located over the memory DIMMs.)
Optional SAS controller card (IBM Flex System Dual VIOS Adapter)
Figure 5-67 System board layout of the IBM Flex System p270 Compute Node
321
p260 (Machine type 7895) Model number L3 cache per system Memory min Memory max 32 MB 22X 64 MB 64 MB 23A 40 MB 80 MB 23X 160 MB 160 MB
Power button
Figure 5-68 Front panel of the IBM Flex System p270 Compute Node
The USB port on the front of the Power Systems compute nodes is useful for various tasks, including out-of-band diagnostic tests, hardware RAID setup, operating system access to data on removable media, and local OS installation. It might be helpful to obtain a USB optical (CD or DVD) drive for these purposes, in case the need arises. Tip: There is no optical drive in the IBM Flex System Enterprise Chassis.
322
IBM PureFlex System and IBM Flex System Products and Technology
When the system is removed from the chassis: Use this button to illuminate the light path diagnostic panel on the top of the front bezel, as shown in Figure 5-69.
The LEDs on the light path panel indicate the following LEDs: LP: Light Path panel power indicator S BRD: System board LED (can indicate trouble with a processor or memory) MGMT: Anchor card error (also referred to as the management card) LED. D BRD: Drive or DASD board LED DRV 1: Drive 1 LED (SSD 1 or HDD 1) DRV 2: Drive 2 LED (SSD 2 or HDD 2) ETE: Expansion connector LED If problems occur, you can use the light path diagnostics LEDs to identify the subsystem involved. To illuminate the LEDs with the compute node removed, press the power button on the front panel. This action temporarily illuminates the LEDs of the troubled subsystem to direct troubleshooting efforts towards a resolution. Typically, an administrator already obtained this information from the IBM Flex System Manager or Chassis Management Module before removing the node. However, the LEDs helps with repairs and troubleshooting if onsite assistance is needed. For more information about the front panel and LEDs, see IBM Flex System p270 Compute Node Installation and Service Guide, which is available at this website: http://publib.boulder.ibm.com/infocenter/flexsys/information
323
Up to 14 p270 Compute Nodes can be installed in the chassis in 10U of rack space. The actual number of systems that can be powered on in a chassis depends on the following factors: Number of power supplies that are installed in the chassis Capacity of the power supplies installed in the chassis (2100 W or 2500 W) Power redundancy policy used in the chassis (N+1 or N+N) Table 4-11 on page 93 provides guidelines about what number of p270 systems can be powered on in the IBM Flex System Enterprise Chassis, based on the type and number of power supplies installed.
DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM
Drive SMI
Drive
SMI
SMI
To front panel
SMI
4 bytes each
SMI
PCIe 2.0 x8
I/O connector 1
SMI
SMI
SAS controller on the optional Dual VIOS Adapter installed in the ETE connector Flash NVRAM 256 MB DDR2 TPMD Anchor card/VPD Gb Ethernet ports
FSP
Phy
Figure 5-70 IBM Flex System p270 Compute Node block diagram
The p270 compute node has the POWER7+ processors packaged as dual-chip modules (DCMs). Each DCM consists of two POWER7+ processors. DCMs installed consist of two six-core chips. In Figure 5-70 on page 324, you can see the two DCMs, with eight memory slots for each module. Each module is connected to a P7IOC I/O hub, which connects to the I/O subsystem (I/O adapters and local storage). At the bottom of Figure 5-70 on page 324, you can see a representation of the flexible service processor (FSP) architecture. 324
IBM PureFlex System and IBM Flex System Products and Technology
Introduced in this generation of Power Systems compute nodes is a secondary SAS controller card, which is inserted in the ETE connector. This secondary SAS controller allows independent assignment of the internal drives to separate partitions.
Processor options
Table 5-98 defines the processor options for the p270 Compute Node.
Table 5-98 Processor options Feature code EPRF EPRE Number of sockets 2 2 POWER7+ chips per socket 2 (dual-chip modules) 2 (dual-chip modules) Cores per POWER7+ chip 6 6 Total cores 24 24 Core frequency 3.1 GHz 3.4 GHz L3 cache size per POWER7+ processor 60 MB 60 MB
To optimize software licensing, you can deconfigure or disable one or more cores. The feature is listed in Table 5-99.
Table 5-99 Deconfiguration of cores Feature code 2319 Description Factory Deconfiguration of one core Minimum 0 Maximum 23
This core deconfiguration feature can also be updated after installation by using the field core override option. One core must remain enabled, hence the maximum number of 23 features.
Architecture
IBM uses innovative methods to achieve the required levels of throughput and bandwidth. Areas of innovation for the POWER7+ processor and POWER7+ processor-based systems include (but are not limited to) the following elements: On-chip L3 cache implemented in embedded dynamic random access memory (eDRAM) Cache hierarchy and component innovation Advances in memory subsystem Advances in off-chip signaling Advances in RAS features such as power-on reset and L3 cache dynamic column repair Figure 5-71 shows the POWER7+ processor die layout with the following major areas identified: Eight POWER7+ processor cores (six are enabled in the p270) L2 cache
325
L3 cache Chip power bus interconnect SMP links GX++ interface Memory controllers I/O links
Figure 5-71 POWER7+ processor architecture (6 cores are enabled in the p270)
Table 5-100 shows comparable characteristics between the generations of POWER7+ and POWER7 processors
Table 5-100 Comparing the technology of the POWER7+ and POWER7 processors Characteristic
Technology Die size Maximum cores Maximum SMT threads per core Maximum frequency L2 Cache L3 Cache
POWER7
45 nm 567 mm2 8 4 4.25 GHz 256 KB per core 4 MB or 8MB of FLR-L3 cache per core with each core having access to the full 32 MB of L3 cache, on-chip eDRAM DDR3 Two GX++
POWER7+
32 nm 567 mm2 8 4 4.3 GHz 256 KB per core 10 MB of FLR-L3 cache per core with each core having access to the full 80 MB of L3 cache, on-chip eDRAM DDR3 Two GX++
326
IBM PureFlex System and IBM Flex System Products and Technology
DASD/local storage option dependency on memory form factor: Because of the design of the on-cover storage connections, clients that seek to use SAS HDDs must use VLP DIMMs (4 GB or 8 GB). The cover cannot be closed properly if LP DIMMs and SAS HDDs are configured in the same system. However, SSDs and LP DIMMs can be used together. For more information, see 5.6.10, Storage on page 313.
327
There are 16 buffered DIMM slots on the p270, eight per processor, as shown in Figure 5-60 on page 309.
DIMM 1 (P1-C1) DIMM 2 (P1-C2) DIMM 3 (P1-C3) DIMM 4 (P1-C4) DIMM 5 (P1-C5) DIMM 6 (P1-C6) DIMM 7 (P1-C7) DIMM 8 (P1-C8) DIMM 9 (P1-C9) DIMM 10 (P1-C10) DIMM 11 (P1-C11) DIMM 12 (P1-C12) DIMM 13 (P1-C13) DIMM 14 (P1-C14) DIMM 15 (P1-C15) DIMM 16 (P1-C16)
SMI
SMI
SMI
SMI
SMI
SMI
SMI
SMI
Figure 5-72 Memory DIMM topology (IBM Flex System p270 Compute Node)
The following memory-placement rules must be considered: Install DIMM fillers in unused DIMM slots to ensure proper cooling. Install DIMMs in pairs. Both DIMMs in a pair must be the same size, speed, type, and technology. You can mix compatible DIMMs from multiple manufacturers. Install only supported DIMMs, as described at the IBM ServerProven website: http://www.ibm.com/servers/eserver/serverproven/compat/us/ Table 5-93 on page 310 shows the required placement of memory DIMMs, depending on the number of DIMMs installed.
Table 5-103 DIMM placement: p270 Processor 0 DIMM 10 DIMM 11 DIMM 1 DIMM 2 DIMM 3 DIMM 4 DIMM 5 DIMM 6 DIMM 7 DIMM 8 DIMM 9 Processor 1 DIMM 12 DIMM 13 DIMM 14 DIMM 15 x DIMM 16 x x x x x x
Number of DIMMs 2 4 6 8 10 12 14 16
x x x x x x x x x x x x x x x x
x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x
328
IBM PureFlex System and IBM Flex System Products and Technology
All installed memory DIMMs do not have to be the same size, but it is a preferred practice that the following groups of DIMMs be kept the same size: Slots 1 - 4 Slots 5 - 8 Slots 9 - 12 Slots 13 - 16
5.7.10 Storage
The Power Systems compute nodes have an onboard SAS controller that can manage one or two, non-hot-pluggable internal drives. Both 2.5-inch HDDs and 1.8-inch SSDs are supported; however, the use of 2.5-inch drives imposes restrictions on DIMMs that are used, as described in the next section. The drives attach to the cover of the server, as shown in Figure 5-73. The IBM Flex System Dual VIOS Adapter sits below the I/O adapter that is installed in I/O connector 2. Dual VIOS Adapter (installs under I/O adapter 2) Drives mounted on the underside of the cover
Figure 5-73 The p270 showing the hard disk drive locations on the top cover
329
Optional second SAS adapter, installed in expansion port EC2F IBM Flex System Dual VIOS adapter 1
2.5-inch SAS HDDs 8274 8276 8311 1.8-inch SSDs 8207 177 GB SATA non-hot-swap SSD 2 300 GB 10K RPM non-hot-swap 6 Gbps SAS 600 GB 10K RPM non-hot-swap 6 Gbps SAS 900 GB 10K RPM non-hot-swap 6 Gbps SAS 2 2 2
If you use local drives, you must order the appropriate cover with connections for your drive type. As you can see in Figure 5-63 on page 313, the local drives (HDD or SDD) are mounted to the top cover of the system. Table 5-105 lists the top cover options because you must select the cover feature that matches the drives you want to install: 2.5-inch drives, 1.8-inch drives, or no drives.
Table 5-105 Top cover options for the p270 Feature code 7069 7068 7067 Description Top cover with connectors for 2.5-inch drives for the p270 Top cover with connectors for 1.8-inch drives for the p270 Top cover for no drives on the p270
330
IBM PureFlex System and IBM Flex System Products and Technology
On the system board, the connection for the covers drive interposer is shown in Figure 5-74.
Figure 5-75 Connection for drive interposer card mounted to the system cover
331
The IBM Flex System Dual VIOS Adapter is shown in Figure 5-76. The adapter attaches via the expansion (ETE) connector. Even with the Dual VIOS Adapter installed, an I/O adapter can be installed in slot 2. I/O adapter in slot 1
Figure 5-76 IBM Flex System Dual VIOS Adapter in the p270
RAID capabilities
When two internal drives are installed in the p270 but without the Dual VIOS Adapter installed, RAID-0 or RAID-10 can be configured. Configure the RAID array by running the smit sasdam command, which starts the SAS RAID Disk Array Manager for AIX. The AIX Disk Array Manager is packaged with the Diagnostics utilities on the Diagnostics CD. Run the smit sasdam command to configure the disk drives for use with the SAS controller. The diagnostics CD can be downloaded in ISO file format from the following website: http://www14.software.ibm.com/webapp/set2/sas/f/diags/download/ For more information, see Using the Disk Array Manager in the Systems Hardware Information Center at this website: http://publib.boulder.ibm.com/infocenter/systems/scope/hw/index.jsp?topic=/p7ebj/s asusingthesasdiskarraymanager.htm Tip: Depending on your RAID configuration, you might need to create the array before you install the operating system in the compute node. Before you can create a RAID array, you must reformat the drives so that the sector size of the drives changes from 512 bytes to 528 bytes. If you later decide to remove the drives, delete the RAID array before you remove the drives. If you decide to delete the RAID array and reuse the drives, you might need to reformat the drives so that the sector size of the drives changes from 528 bytes to 512 bytes.
332
IBM PureFlex System and IBM Flex System Products and Technology
Converged Ethernet adapter EC24 IBM Flex System CN4058 8-port 10Gb Converged Adapter
Fibre Channel /O adapters 1764 EC23 EC2E IBM Flex System FC3172 2-port 8Gb FC Adapter IBM Flex System FC5052 2-port 16Gb FC Adapter IBM Flex System FC5054 4-port 16Gb FC Adapter
InfiniBand I/O adapters 1761 IBM Flex System IB6132 2-port QDR InfiniBand Adapter
333
Generally, you do not interact with the Flexible Support Processor directly, but by using tools, such as IBM Flex System Manager, Chassis Management Module, the IBM Hardware Management Console (HMC) and the Integrated Virtualization Manager (IVM). The Flexible Support Processor provides a Serial-over-LAN interface, which is available by using the Chassis Management Module and the console command.
334
IBM PureFlex System and IBM Flex System Products and Technology
5.8.1 Overview
The IBM Flex System p460 Compute Node is a full-wide, Power Systems compute node. It has four POWER7 processor sockets, 32 memory slots, four I/O adapter slots, and an option for up to two internal drives for local storage. The IBM Flex System p460 Compute Node has the specifications that are shown in Table 5-108.
Table 5-108 IBM Flex System p460 Compute Node specifications Components Model numbers Form factor Chassis support Specification 7895-42X and 7895-43X Full-wide compute node. IBM Flex System Enterprise Chassis.
335
Components Processor
Specification p460: Four IBM POWER7 (model 42X) or POWER7+ (model 43X) processors. POWER7 processors: Each processor is a single-chip module (SCM) that contains eight cores (up to 3.55 GHz and 32 MB L3 cache) or four cores (3.3 GHz and 16 MB L3 cache). Each processor has 4 MB L3 cache per core. Integrated memory controller in each processor, each with four memory channels. Each memory channel operates at 6.4 Gbps. One GX++ I/O bus connection per processor. Supports SMT4 mode, which enables four instruction threads to run simultaneously per core. Uses 45 nm fabrication technology. POWER7+ processors: Each processor is a single-chip module (SCM) that contains eight cores (up to 4.1 GHz or 3.6 GHz and 80 MB L3 cache) or four cores (4.0 GHz and 40 MB L3 cache). Each processor has 10 MB L3 cache per core, so 8-core processors have 80 MB of L3 cache total. Integrated memory controller in each processor, each with four memory channels. Each memory channel operates at 6.4 Gbps. One GX++ I/O bus connection per processor. Supports SMT4 mode, which enables four instruction threads to run simultaneously per core. Uses 32 nm fabrication technology.
Chipset Memory
IBM P7IOC I/O hub. 32 DIMM sockets. RDIMM DDR3 memory supported. Integrated memory controller in each processor, each with four memory channels. Supports Active Memory Expansion with AIX 6.1 or later. All DIMMs operate at 1066 MHz. Both LP and VLP DIMMs are supported, although only VLP DIMMs are supported if internal HDDs are configured. The use of 1.8-inch SSDs allows the use of LP and VLP DIMMs. 1 TB using 32x 32 GB DIMMs. ECC, Chipkill. Two 2.5-inch non-hot-swap drive bays that support 2.5-inch SAS HDD or 1.8-inch SATA SSD drives. If LP DIMMs are installed, only 1.8-inch SSDs are supported. If VLP DIMMs are installed, both HDDs and SSDs are supported. An HDD and an SSD cannot be installed together. 1.8 TB that uses two 900 GB SAS HDD drives, or 354 GB using two 177 GB SSD drives. RAID support by using the operating system. None standard. Optional 1 Gb or 10 Gb Ethernet adapters. Two I/O connectors for adapters. PCI Express 2.0 x16 interface. One external USB port. FSP, Predictive Failure Analysis, light path diagnostics panel, automatic server restart, Serial over LAN support. IPMI compliant. Support for IBM Flex System Manager, and IBM Systems Director. Power-on password, selectable boot sequence. None. Remote management by using Serial over LAN and IBM Flex System Manager.
Maximum internal storage RAID support Network interfaces PCI Expansion slots Ports Systems management Security features Video
336
IBM PureFlex System and IBM Flex System Products and Technology
Specification 3-year customer-replaceable unit and onsite limited warranty with 9x5/NBD. IBM AIX, IBM i, and Linux.
Optional service upgrades are available through IBM ServicePacs: 4-hour or 2-hour response time, 8-hour fix time, 1-year or 2-year warranty extension, remote technical support for IBM hardware and selected IBM and OEM software. Width: 437 mm (17.2"), height: 51 mm (2.0), depth: 493 mm (19.4). Maximum configuration: 14.0 kg (30.6 lb).
Dimensions Weight
337
32 DIMM slots
Figure 5-77 Layout of the IBM Flex System p460 Compute Node
338
IBM PureFlex System and IBM Flex System Products and Technology
Power button
The USB port on the front of the Power Systems compute nodes is useful for various tasks. These tasks include out-of-band diagnostic procedures, hardware RAID setup, operating system access to data on removable media, and local OS installation. It might be helpful to obtain a USB optical (CD or DVD) drive for these purposes, in case the need arises. Tip: There is no optical drive in the IBM Flex System Enterprise Chassis. The power-control button on the front of the server (see Figure 5-55 on page 302) has the following functions: When the system is fully installed in the chassis: Use this button to power the system on and off. When the system is removed from the chassis: Use this button to illuminate the light path diagnostic panel on the top of the front bezel, as shown in Figure 5-79.
339
The LEDs on the light path panel indicate the status of the following devices: LP: Light Path panel power indicator S BRD: System board LED (might indicate trouble with processor or MEM) MGMT: Flexible Support Processor (or management card) LED D BRD: Drive or DASD board LED DRV 1: Drive 1 LED (SSD 1 or HDD 1) DRV 2: Drive 2 LED (SSD 2 or HDD 2) ETE: Sidecar connector LED (not present on the IBM Flex System p460 Compute Node) If problems occur, the light path diagnostics LEDs help with identifying the subsystem involved. To illuminate the LEDs with the compute node removed, press the power button on the front panel. Pressing the button temporarily illuminates the LEDs of the troubled subsystem to direct troubleshooting efforts. You usually obtain this information from the IBM Flex System Manager or Chassis Management Module before you remove the node. However, having the LEDs helps with repairs and troubleshooting if onsite assistance is needed. For more information about the front panel and LEDs, see IBM Flex System p260 and p460 Compute Node Installation and Service Guide, which is available at this website: http://www.ibm.com/support
340
IBM PureFlex System and IBM Flex System Products and Technology
DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM
SMI SMI SMI SMI 4 bytes each SMI SMI SMI SMI FSP BCM5387 Ethernet switch Systems Management connector Gb Ethernet ports
Phy
POWER7 Processor 0
GX++ 4 bytes
PCIe to PCI
USB controller
To front panel
I/O connector 2
POWER7 Processor 1
DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM
Flash NVRAM 256 MB DDR2 TPMD Anchor card/VPD SMI SAS SMI SMI SMI 4 bytes each SMI SMI SMI SMI FSPIO HDDs/SSDs
POWER7 Processor 2
I/O connector 4
POWER7 Processor 3
Figure 5-80 IBM Flex System p460 Compute Node block diagram
341
The four processors in the IBM Flex System p460 Compute Node are connected in a cross-bar formation, as shown in Figure 5-81.
POWER7 Processor 0
POWER7 Processor 1
4 bytes each
POWER7 Processor 2
POWER7 Processor 3
Figure 5-81 IBM Flex System p460 Compute Node processor connectivity
5.8.6 Processor
The IBM POWER7 processor represents a leap forward in technology and associated computing capability. The multi-core architecture of the POWER7 processor is matched with a wide range of related technologies to deliver leading throughput, efficiency, scalability, and RAS. Although the processor is an important component in servers, many elements and facilities must be balanced across a server to deliver maximum throughput. The design philosophy for POWER7 processor-based systems is system-wide balance, in which the POWER7 processor plays an important role. Table 5-109 defines the processor options for the p460.
Table 5-109 Processor options for the p460 Feature code POWER7 EPR2 EPR4 EPR6 POWER7+ 4 8 8 4 4 4 16 32 32 3.3 GHz 3.2 GHz 3.55 GHz 16 MB 32 MB 32 MB Cores per POWER7 processor Number of POWER7 processors Total cores Core frequency L3 cache size per POWE7 processor
EPRK
EPRH EPRJ
4 8 8
4 4 4
16 32 32
40 MB 80 MB 80 MB
342
IBM PureFlex System and IBM Flex System Products and Technology
To optimize software licensing, you can deconfigure or disable one or more cores. The feature is listed in Table 5-110.
Table 5-110 Deconfiguration of cores Feature code 2319 Description Factory Deconfiguration of 1-core Minimum 0 Maximum 1 less than the total number of cores (For EPR5, the maximum is 7)
POWER7 architecture
IBM uses innovative methods to achieve the required levels of throughput and bandwidth. Areas of innovation for the POWER7 processor and POWER7 processor-based systems include (but are not limited to) the following elements: On-chip L3 cache that is implemented in embedded dynamic random-access memory (eDRAM) Cache hierarchy and component innovation Advances in memory subsystem Advances in off-chip signaling The superscalar POWER7 processor design also provides the following capabilities: Binary compatibility with the prior generation of POWER processors Support for PowerVM virtualization capabilities, including PowerVM Live Partition Mobility to and from IBM POWER6 and IBM POWER6+ processor-based systems Table 5-81 on page 291 shows the POWER7 processor die layout with major areas identified: Eight POWER7 processor cores, L2 cache, L3 cache and chip power bus interconnect, SMP links, GX++ interface, and integrated memory controller.
GX++ Bridge
Memory Controller
C1 Core L2
C1 Core L2
C1 Core L2
C1 Core L2
SMP
Figure 5-82 POWER7 processor architecture
Memory Buffers
343
POWER7+ architecture
The POWER7+ architecture builds on the POWER7 architecture. IBM uses innovative methods to achieve the required levels of throughput and bandwidth. Areas of innovation for the POWER7+ processor and POWER7+ processor-based systems include (but are not limited to) the following elements: On-chip L3 cache implemented in embedded dynamic random access memory (eDRAM) Cache hierarchy and component innovation Advances in memory subsystem Advances in off-chip signaling Advances in RAS features such as power-on reset and L3 cache dynamic column repair The superscalar POWER7+ processor design also provides the following capabilities: Binary compatibility with the prior generation of POWER processors Support for PowerVM virtualization capabilities, including PowerVM Live Partition Mobility to and from POWER6, POWER6+, and POWER7 processor-based systems Figure 5-83 shows the POWER7+ processor die layout with major areas identified: Eight POWER7+ processor cores L2 cache L3 cache Chip power bus interconnect SMP links GX++ interface Memory controllers I/O links
344
IBM PureFlex System and IBM Flex System Products and Technology
Components Processor cores Max execution threads core/chip L2 cache per core/per chip On-chip L3 cache per core / per chip DDR3 memory controllers Compatibility
5.8.7 Memory
Each POWER7 processor has two integrated memory controllers in the chip. Industry standard DDR3 RDIMM technology is used to increase reliability, speed, and density of memory subsystems.
Use a minimum of 2 GB of RAM per core. The functional minimum memory configuration for the system is 4 GB (2x2 GB) but that is not sufficient for reasonable production use of the system.
Chapter 5. Compute nodes
345
a. If 2.5-inch HDDs are installed, low-profile DIMM features cannot be used (EM04, 8145, EEME, and EEMF cannot be used).
Requirement: Because of the design of the on-cover storage connections, if you use SAS HDDs, you must use VLP DIMMs (4 GB or 8 GB). The cover cannot close properly if LP DIMMs and SAS HDDs are configured in the same system. Combining the two physically obstructs the cover from closing. For more information, see 5.6.10, Storage on page 313.
346
IBM PureFlex System and IBM Flex System Products and Technology
There are 16 buffered DIMM slots on the p260 and the p24L, as shown in Figure 5-84. The IBM Flex System p460 Compute Node adds two more processors and 16 more DIMM slots, which are divided evenly (eight memory slots) per processor.
SMI
DIMM 1 (P1-C1) DIMM 2 (P1-C2) DIMM 3 (P1-C3) DIMM 4 (P1-C4) DIMM 5 (P1-C5) DIMM 6 (P1-C6) DIMM 7 (P1-C7) DIMM 8 (P1-C8) DIMM 9 (P1-C9) DIMM 10 (P1-C10) DIMM 11 (P1-C11) DIMM 12 (P1-C12) DIMM 13 (P1-C13) DIMM 14 (P1-C14) DIMM 15 (P1-C15) DIMM 16 (P1-C16)
SMI
POWER7 Processor 0
SMI
SMI
SMI
SMI
POWER7 Processor 1
SMI
SMI
The memory-placement rules must be considered: Install DIMM fillers in unused DIMM slots to ensure efficient cooling. Install DIMMs in pairs. Both DIMMs in a pair must be the same size, speed, type, and technology. You can mix compatible DIMMs from multiple manufacturers. Install only supported DIMMs, as described on the IBM ServerProven website: http://www.ibm.com/servers/eserver/serverproven/compat/us/
347
For the IBM Flex System p460 Compute Node, Table 5-114 shows the required placement of memory DIMMs, depending on the number of DIMMs installed.
Table 5-114 DIMM placement on IBM Flex System p460 Compute Node CPU 0 CPU 1 CPU 2 Number of DIMMs
CPU 3
DIMM 10
DIMM 11
DIMM 12
DIMM 13
DIMM 14
DIMM 15
DIMM 16
DIMM 17
DIMM 18
DIMM 19
DIMM 20
DIMM 21
DIMM 22
DIMM 23
DIMM 24
DIMM 25
DIMM 26
DIMM 27
DIMM 28
DIMM 29
DIMM 30 x
DIMM 31 x
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32
x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x
x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x
348
IBM PureFlex System and IBM Flex System Products and Technology
DIMM 32
DIMM 1
DIMM 2
DIMM 3
DIMM 4
DIMM 5
DIMM 6
DIMM 7
DIMM 8
DIMM 9
Both cases show the following knee of the curve relationships for processor resources that are required for memory expansion: Busy processor cores do not have resources to spare for expansion. The more memory expansion that is done, the more processor resources are required.
349
The knee varies, depending on how compressible the memory contents are. This variation demonstrates the need for a case-by-case study to determine whether memory expansion can provide a positive return on investment. To help you perform this study, a planning tool is included with AIX 6.1 Technology Level 4 or later. You can use this tool to sample actual workloads and estimate both how expandable the partition memory is and how much processor resource is needed. Any Power System model runs the planning tool. Figure 5-86 shows an example of the output that is returned by this planning tool. The tool outputs various real memory and processor resource combinations to achieve the required effective memory, and proposes one particular combination. In this example, the tool proposes to allocate 58% of a processor core, to benefit from 45% extra memory capacity.
Active Memory Expansion Modeled Statistics: ----------------------Modeled Expanded Memory Size : 8.00 GB Expansion Factor --------1.21 1.31 1.41 1.51 1.61 True Memory Modeled Size -------------6.75 GB 6.25 GB 5.75 GB 5.50 GB 5.00 GB Modeled Memory Gain ----------------1.25 GB [ 19%] 1.75 GB [ 28%] 2.25 GB [ 39%] 2.50 GB [ 45%] 3.00 GB [ 60%] CPU Usage Estimate ----------0.00 0.20 0.35 0.58 1.46
Active Memory Expansion Recommendation: --------------------The recommended AME configuration for this workload is to configure the LPAR with a memory size of 5.50 GB and to configure a memory expansion factor of 1.51. This will result in a memory expansion of 45% from the LPAR's current memory size. With this configuration, the estimated CPU usage due to Active Memory Expansion is approximately 0.58 physical processors, and the estimated overall peak CPU resource required for the LPAR is 3.72 physical processors. Figure 5-86 Output from the AIX Active Memory Expansion planning tool
For more information about this topic, see the white paper Active Memory Expansion: Overview and Usage Guide, which is available at this website: http://www.ibm.com/systems/power/hardware/whitepapers/am_exp.html
5.8.9 Storage
The p460 has an onboard SAS controller that can manage up to two, non-hot-pluggable internal drives. The drives attach to the cover of the server, as shown in Figure 5-87 on page 351. Even though the p460 is a full-wide server, it has the same storage options as the p260 and the p24L.
350
IBM PureFlex System and IBM Flex System Products and Technology
The type of local drives that are used affects the form factor of your memory DIMMs. If HDDs are chosen, only VLP DIMMs can be used because of internal spacing. There is not enough room for the 2.5-inch drives to be used with LP DIMMs (currently the 2 GB and 16 GB sizes). Verify your memory choice to make sure that it is compatible with the local storage configuration. The use of SSDs does not have the same limitation, and so LP DIMMs can be used with SSDs.
Figure 5-87 The IBM Flex System p260 Compute Node showing hard disk drive location
2.5 inch SAS HDDs 7066 8274 8276 8311 None 42D0627 49Y2022 81Y9654 Top cover with HDD connectors for the IBM Flex System p460 Compute Node (full-wide) 300 GB 10K RPM non-hot-swap 6 Gbps SAS 600 GB 10K RPM non-hot-swap 6 Gbps SAS 900 GB 10K RPM non-hot-swap 6 Gbps SAS
351
Description Top Cover with SSD connectors for IBM Flex System p460 Compute Node (full-wide) 177 GB SATA non-hot-swap SSD
None
Top cover for no drives on the IBM Flex System p460 Compute Node (full-wide)
On covers that accommodate drives, the drives attach to an interposer that connects to the system board when the cover is properly installed, as shown in Figure 5-88.
The connection for the covers drive interposer on the system board is shown in Figure 5-89.
Figure 5-89 Connection for drive interposer card mounted to the system cover
352
IBM PureFlex System and IBM Flex System Products and Technology
353
Description IBM Flex System CN4058 8-port 10Gb Converged adapter IBM Flex System EN4132 2-port 10Gb RoCE adapter IBM Flex System FC3172 2-port 8Gb FC Adapter IBM Flex System FC5052 2-port 16Gb FC adapter IBM Flex System FC5054 4-port 16Gb FC adapter IBM Flex System IB6132 2-port QDR InfiniBand Adapter
Number of ports 8 2 2 2 4 2
a. At least one 10 Gb (#1762 or #EC24) or 1 Gb (#1763) Ethernet adapter must be configured in each server.
Anchor card
The anchor card, which is shown in Figure 5-90, contains the vital product data chip that stores system-specific information. The pluggable anchor card provides a means for this information to be transferred from a faulty system board to the replacement system board. Before the service processor knows what system it is on, it reads the vital product data chip to obtain system information. The vital product data chip includes information such as system type, model, and serial number.
355
The p460 model 43X supports the following operating systems: AIX V7.1 with the 7100-02 Technology Level with Service Pack 3 or later AIX V6.1 with the 6100-08 Technology Level with Service Pack 3 or later VIOS 2.2.2.3 or later IBM i 6.1 with i 6.1.1 machine code, or later IBM i 7.1 TR3 or later SUSE Linux Enterprise Server 11 Service Pack (SP) 2 for POWER Red Hat Enterprise Linux 6.4 for POWER Important: Support by some of these operating system versions is post generally availability. See the following IBM ServerProven website for the latest information about the specific versions and service levels supported and any other prerequisites: http://www.ibm.com/systems/info/x86servers/serverproven/compat/us/nos/matrix.sh tml
Figure 5-91 IBM Flex System PCIe Expansion Node attached to a compute node
The ordering information for the PCIe Expansion Node is listed in Table 5-117.
Table 5-117 PCIe Expansion Node ordering number and feature code Part number 81Y8983 Feature codea A1BV Description IBM Flex System PCIe Expansion Node
a. The feature code listed is for both the System x sales channel (HVEC) using x-config and the Power Systems sales channel (AAS) using e-config.
356
IBM PureFlex System and IBM Flex System Products and Technology
The part number includes the following items: IBM Flex System PCIe Expansion Node Two riser assemblies Interposer cable assembly Double-wide shelf Two auxiliary power cables (for adapters that require additional +12 V power) Four removable PCIe slot air flow baffles Documentation CD that contains the Installation and Service Guide Warranty information and Safety flyer and Important Notices document The PCIe Expansion Node is supported when it is attached to the compute nodes that are listed in Table 5-118.
Table 5-118 Supported compute nodes p24L p260 p270 N p460 N x220 x222 x240 Ya x440 N Part number Expansion Node
81Y8983
Ya
5.9.1 Features
The PCIe Expansion Node has the following features: Support for up to four standard PCIe 2.0 adapters: Two PCIe 2.0 x16 slots that support full-length, full-height adapters (1x, 2x, 4x, 8x, and 16x adapters supported) Two PCIe 2.0 x8 slots that support low-profile adapters (1x, 2x, 4x, and 8x adapters supported) Support for PCIe 3.0 adapters by operating them in PCIe 2.0 mode Support for one full-length, full-height double-wide adapter (using the space of the two full-length, full-height adapter slots) Support for PCIe cards with higher power requirements The Expansion Node provides two auxiliary power connections, up to 75 W each for a total of 150 W of more power by using standard 2x3, +12 V six-pin power connectors. These connectors are placed on the base system board so that they both can provide power to a single adapter (up to 225 W), or to two adapters (up to 150 W each). Power cables are used to connect from these connectors to the PCIe adapters and are included with the PCIe Expansion Node. Two Flex System I/O expansion connectors The I/O expansion connectors are labeled I/O expansion 3 connector and I/O expansion four connector in Figure 5-95 on page 360. These I/O connectors expand the I/O capability of the attached compute node.
357
x240 node
Figure 5-92 PCIe Expansion Node attached to a node showing the four PCIe slots
A double wide shelf is included with the PCIe Expansion Node. The compute node and the expansion node must be attached to the shelf, and then the interposer cable is attached, which links the two electronically. Figure 5-93 shows installation of the compute node and the PCIe Expansion Node on the shelf.
Compute Node
Figure 5-93 Installation of a compute node and PCIe Expansion Node on to the tray
358
IBM PureFlex System and IBM Flex System Products and Technology
After the compute node and PCIe Expansion Node are installed onto the shelf, an interposer cable is connected between them. This cable provides the link for the PCIe bus between the two components (this cable is shown in Figure 5-94). The cable consists of a ribbon cable with a circuit board at each end.
5.9.2 Architecture
The architecture diagram is shown on Figure 5-95 on page 360. PCIe version: All PCIe bays on the expansion node operate at PCIe 2.0. The interposer link is a PCIe 2.0 x16 link, which is connected to the switch on the main board of the PCIe Expansion Node. This PCIe switch provides two PCIe connections for bays 1 and 2 (the full-length, full-height adapters slots) and two PCIe connections for bays 3 and 4 (the low profile adapter slots).
359
There are two other I/O adapter bays (x16) that are available that connect into the midplane of the enterprise chassis. You can use these bays to set up a single wide node to make use of a double-wide nodes I/O bandwidth to the midplane.
Compute Node PCIe Expansion Node Interposer cable - PCIe 2.0 x16
I/O 1
I/O 2
Expansion connector
I/O 3
I/O 4
x16 PCIe switch x16 x16 PCIe 2.0 x16 FHFL PCIe 2.0 x16 FHFL
x16
Processor 2
x8
x8
PCIe 2.0 x8 LP
Processor 1
Number of installed processors: Two processors must be installed in the compute node because the expansion connector is routed from processor 2. Table 5-119 shows the adapter to I/O bay mapping.
Table 5-119 Adapter to I/O bay mapping I/O expansion slot Slot 1 (Compute Node) Port on the adapter Port 1 Port 2 Port 3a Port 4a Slot 2 (Compute Node) Port 1 Port 2 Port 3a Port 4a Corresponding I/O module bay in the chassis Module bay 1 Module bay 2 Module bay 1b Module bay 2b Module bay 3 Module bay 4 Module bay 3b Module bay 4b
360
IBM PureFlex System and IBM Flex System Products and Technology
PCIe 2.0 x8 LP
Corresponding I/O module bay in the chassis Module bay 1 Module bay 2 Module bay 1b Module bay 2b Module bay 3 Module bay 4 Module bay 3** Module bay 4**
a. Ports 3 and 4 require that a four-port card be installed in the expansion slot. b. Might require one or more port upgrades to be installed in the I/O module.
361
Description NVIDIA Tesla M2090 (full-height adapter) NVIDIA GRID K1 for IBM Flex System PCIe Expansion Node NVIDIA GRID K2 for IBM Flex System PCIe Expansion Node NVIDIA Tesla K20 for IBM Flex System PCIe Expansion Node Intel Xeon Phi 5110P for IBM Flex System PCIe Expansion Node IBM 4765 Crypto Card
Maximum supported 1b 1c 1c 1c 1c 1c
a. Withdrawn from marketing b. If the NVIDIA Tesla M2090 is installed in the Expansion Node, an adapter cannot be installed in the other full-height slot. The low-profile slots and Flex System I/O expansion slots can still be used. c. If installed, only this adapter is supported in the system. No other PCIe adapters can be installed. d. Orderable as separate MTM 4765-001 feature 4809. Available via AAS (e-config) only.
For the current list of adapters that are supported in the Expansion Node, see the IBM ServerProven site at: http://ibm.com/systems/info/x86servers/serverproven/compat/us/flexsystems.html For information about the IBM High IOPS adapters, the following IBM Redbooks Product Guides are available: IBM High IOPS MLC Adapters: http://www.redbooks.ibm.com/abstracts/tips0907.html IBM High IOPS Modular Adapters: http://www.redbooks.ibm.com/abstracts/tips0937.html IBM High IOPS SSD PCIe Adapters: http://www.redbooks.ibm.com/abstracts/tips0729.html Although the design of Expansion Node facilitates a much greater set of standard PCIe adapters, Table 5-120 on page 361 lists the adapters that are supported. If the PCI Express adapter that you require is not on the ServerProven website, use the IBM ServerProven Opportunity Request for Evaluation (SPORE) process to confirm compatibility with the configuration.
362
IBM PureFlex System and IBM Flex System Products and Technology
Description IBM Flex System IB6132 2-port FDR InfiniBand Adapter IBM Flex System FC5022 2-port 16Gb FC Adapter IBM Flex System FC3172 2-port 8Gb FC Adapter IBM Flex System FC3052 2-port 8Gb FC Adapter IBM Flex System FC5052 2-port 16Gb FC Adapter IBM Flex System FC5054 4-port 16Gb FC Adapter IBM Flex System FC5172 2-port 16Gb FC Adapter
Not supported: At the time of writing, the IBM Flex System EN6132 2-port 40Gb Ethernet Adapter was not supported in the IBM Flex System PCIe Expansion Node. For the current list of adapters that are supported in the Expansion Node, see the IBM ServerProven site at: http://ibm.com/systems/info/x86servers/serverproven/compat/us/flexsystems.html For more information about these adapters, see the IBM Redbooks Product Guides for Flex System in the Adapters category: http://www.redbooks.ibm.com/portals/puresystems?Open&page=pg&cat=adapters
Figure 5-96 IBM Flex System Storage Expansion Node Chapter 5. Compute nodes
363
a. The feature code listed is for both the System x sales channel (HVEC) using x-config and the Power Systems sales channel (AAS) using e-config.
The part number includes the following items: The IBM Flex System Storage Expansion Node Expansion shelf, onto which you install the Compute Node and Storage Expansion Node IBM Warranty information booklet Product documentation CD that includes an installation and service guide The following features are included: Sliding tray to allow access to up to 12 SAS/SATA or SSD storage Hot-swappable drives Supports RAID 0, 1, 5, 6, 10, 50, and 60 512 MB or 1 GB with cache-to-flash super capacitor offload Includes an expansion shelf to physically support the Storage Expansion Node and its compute node Light path diagnostic lights to aid in problem determination Feature on Demand upgrades to add advanced features
68Y8588
Ya
Ya
Two processors: Two processors must be installed in the x220 or x240 compute node because the expansion connector used to connect to the Storage Expansion Node is routed from processor 2.
364
IBM PureFlex System and IBM Flex System Products and Technology
Figure 5-97 shows the Storage Expansion Node front view when it is attached to an x240 compute node.
Figure 5-97 Storage Expansion Node front view: Attached to an x240 compute node
The Storage Expansion Node is a PCIe Generation 3 and a SAS 2.1 complaint enclosure that supports up to twelve 2.5-inch drives. The drives can be HDD or SSD, and both SAS or SATA. Drive modes that are supported are JBOD or RAID-0, 1, 5, 6, 10, 50, and 60. The drives are accessed by opening the handle on the front of the Storage Expansion Node and sliding out the drive tray, which can be done while it is operational (hence the terracotta touch point on the front of the unit). The drive tray extended part way out, while connected to an x240 compute node, is shown in Figure 5-98. With the drive tray extended, all 12 hot-swap drives can be accessed on the left side of the tray. Do not keep the drawer open: Depending on your operating environment, the expansion node might power off if the drawer is open for too long. Chassis fans might increase in speed. The drawer should be closed fully for proper cooling and to protect system data integrity. There is an LED to indicate that the drawer is not closed and that the drawer has been open too long, and that thermal thresholds are reached.
Pull-handle
LED panel
Figure 5-98 Storage Expansion Node with drive tray part way extended
365
The Storage Expansion Node is connected to the compute node through its expansion connector. Management and PCIe connections are provided by this expansion connector, as shown in Figure 5-99. Power is obtained from the enterprise chassis midplane directly, not through the compute node.
Compute Node Storage Expansion Node PCIe 3.0 x8
Cache
Drive tray 12 11 Processor 2 10 9 Processor 1 8 7 5 6 External drive LEDs 3 4 SAS expander 6x SAS 1 2
The LSI SAS controller in the expansion node is connected directly to the PCIe bus of Processor 2 of the compute node. The result is that the compute node sees the disks in the expansion node as locally attached. Management of the Storage Expansion Node is through the IMM2 on the compute node.
a. The feature code listed is for both the System x sales channel (HVEC) using x-config and the Power Systems sales channel (AAS) using e-config.
366
IBM PureFlex System and IBM Flex System Products and Technology
FoD upgrades are system-wide: The FoD upgrades are the same ones that are used with the ServeRAID M5115 available for use internally in the x220 and x240 compute nodes. If you have an M5115 installed in the attached compute node and installed any of these upgrades, then those upgrades are automatically activated on the LSI controller in the expansion node. You do not need to purchase the FoD upgrades separately for the expansion node. RAID 6 Upgrade (90Y4410) Adds support for RAID 6 and RAID 60. This is an FoD license. Performance Upgrade (90Y4412) The Performance Upgrade for IBM Flex System (implemented using the LSI MegaRAID FastPath software) provides high-performance I/O acceleration for SSD-based virtual drives by using a low-latency I/O path to increase the maximum I/O per second (IOPS) capability of the controller. This feature boosts the performance of applications with a highly random data storage access pattern, such as transactional databases. Part number 90Y4412 is an FoD license. SSD Caching Enabler for traditional hard disk drives (90Y4447) The SSD Caching Enabler for IBM Flex System (implemented by using the LSI MegaRAID CacheCade Pro 2.0) accelerates the performance of HDD arrays with only an incremental investment in SSD technology. The feature enables the SSDs to be configured as a dedicated cache to help maximize the I/O performance for transaction-intensive applications, such as databases and web serving. The feature tracks data storage access patterns and identifies the most frequently accessed data. The hot data is then automatically stored on the SSDs that are assigned as a dedicated cache pool on the ServeRAID controller. Part number 90Y4447 is a Feature on Demand license. This feature requires at least one SSD drive be installed.
a. The feature code listed is for both the System x sales channel (HVEC) using x-config and the Power Systems sales channel (AAS) using e-config.
No support for expansion cards: Unlike the PCIe Expansion Node, the Storage Expansion Node cannot connect more I/O expansion cards.
367
10davK SAS hard disk drives 90Y8877 90Y8872 81Y9650 00AD075 NL SATA 81Y9722 81Y9726 81Y9730 A1NX A1NZ A1AV IBM 250 GB 7.2K 6 Gbps NL SATA 2.5" SFF HS HDD IBM 500 GB 7.2K 6 Gbps NL SATA 2.5" SFF HS HDD IBM 1TB 7.2K 6 Gbps NL SATA 2.5" SFF HS HDD A2XC A2XD A282 A48S IBM 300GB 10K 6Gbps SAS 2.5" SFF G2HS HDD IBM 600GB 10K 6Gbps SAS 2.5" SFF G2HS HDD IBM 900 GB 10K 6 Gbps SAS 2.5" SFF HS HDD IBM 1.2TB 10K 6Gbps SAS 2.5'' G2HS HDD
10K and 15K Self-encrypting drives (SED) 00AD085 A48T IBM 1.2TB 10K 6Gbps SAS 2.5'' G2HS SED
SAS-SSD Hybrid drive 00AD102 A4G7 IBM 600GB 10K 6Gbps SAS 2.5'' G2HS Hybrid
Solid-state drives - Enterprise 41Y8331 41Y8336 41Y8341 49Y6129 49Y6134 49Y6139 49Y6195 A4FL A4FN A4FQ A3EW A3EY A3F0 A4GH S3700 200GB SATA 2.5" MLC HS Enterprise SSD S3700 400GB SATA 2.5" MLC HS Enterprise SSD S3700 800GB SATA 2.5" MLC HS Enterprise SSD IBM 200GB SAS 2.5" MLC HS Enterprise SSD IBM 400GB SAS 2.5" MLC HS Enterprise SSD IBM 800GB SAS 2.5" MLC HS Enterprise SSD IBM 1.6TB SAS 2.5" MLC HS Enterprise SSD
Solid-state drives - Enterprise Value 90Y8643 00AJ000 00AJ005 00AJ010 00AJ015 A2U3 A4KM A4KN A4KP A4KQ IBM 256GB SATA 2.5" MLC HS Enterprise Value SSD S3500 120GB SATA 2.5" MLC HS Enterprise Value SSD S3500 240GB SATA 2.5" MLC HS Enterprise Value SSD S3500 480GB SATA 2.5" MLC HS Enterprise Value SSD S3500 800GB SATA 2.5" MLC HS Enterprise Value SSD
a. The feature code listed is for both the System x sales channel (HVEC) using x-config and the Power Systems sales channel (AAS) using e-config.
368
IBM PureFlex System and IBM Flex System Products and Technology
The front of the Storage Expansion Node has a number of LEDs on the lower right front, for identification and status purposes, which are shown in Figure 5-100. The Node is used for indicating a light path fault. Internally, there are a number of light path diagnostic LEDs that are used for fault identification.
2 4 6 8 10 12 1 3 5 7 9 1 1
Drive activity LEDs
Figure 5-100 LEDs on the front of the Storage Expansion Node
In addition to the lights that are described in Table 5-127, there are LEDs locally on each of the drive trays. A green LED indicates disk activity and an amber LED indicates a drive fault. These LEDs can be observed when the drive tray is extended and the unit operational. With the Storage Expansion Node removed from a chassis and its cover removed, there are internal LEDs located below the segmented cable track. There is a light path button that can be pressed and any light path indications can be observed. This button operates when the unit is not powered up because a capacitor provides a power source to illuminate the light path.
369
When the light path diagnostics button is pressed, the light path LED is illuminated, which shows the button is functional. If a fault is detected, the relevant LED also lights. Figure 5-101 and Table 5-128 shows the various LEDs and their statuses.