Unit3 Architecture 221209

Unit 3: IBM DS8000 architecture and hardware overview
Copy right IBM Corporation 2009 Course materials may not be reproduced in whole or in part without the prior written permission of IBM.
5.1
Unit objectives
After completing this unit, you should be able to: Discuss the hardware and architecture of the DS8000 Use virtualization terminology describing configuration of the DS8000 subsystem Describe the physical hardware components and resources Describe the models and features provided by each model Describe the types of disk arrays that can be configured for a DS8000 subsystem Explain the cabling between adapters and drive sets
Copyright IBM Corporation 2009
DS8000 highlights
5.1
DS8000 highlights
New processor family - POWER5+ RISC (DS8000 Turbo)
DS8100 model 931 DS8300 model 932/ 9B2 for LPAR
Significant extensions to enable scalability

64K logical volumes (CKD, FB, or mixed) Expanded volume sizes, dynamic volume add/delete
I/O adapters
Fibre Channel/FICON host adapter (4 ports, 4 Gb/s) ESCON host adapter (2 ports, 18 MB/s) FC-AL device adapter (4 ports, 2 Gb/s)
FC-AL disks
73 GB, 146 GB, 300 GB or 450 GB at 15K rpm FATA disk drives of 500 GB or 1 TB / 7200 rpm 73 GB or 146 GB solid state disks
DS8000 series models (2107)

DS8000 models feature:
High performance High-capacity series of disk storage Design supporting continuous operations
Redundancy Hot replacement/updates
IBM POWER5 server technology
DS8000 models consist of:
Integrated with the IBM Virtualization Engine technology
Graphical user interface (GUI) or command line interface (CLI) allows: For high availability, hardware components are redundant
Storage unit One or two (recommended) Management Consoles (MC)
Performing logical configurations and Copy Services management functions
DS8000 R2 highlights
R2: Announcing new features for ALL models: IBM POWER5+ processor: New DS8000 Turbo (93x/9Bx) Processor memory for POWER 5+ processor 4 Gb FCP/FICON adapter (available on all models 92x/9Ax and 93x/9Bx) 500 GB 7200 rpm FATA drives (available on all models 92x/9Ax and 93x/9Bx) Three-site Metro/Global Mirror Earthquake resistance kit Ethernet adapter pair (for TPC RM support) Performance Accelerator (models 932 and 92E only) 300 GB 15,000 rpm fibre channel drives HyperPAV (System z)
R3: Announcing new features for ALL models: Dynamic increase in size of existing volumes
No need to move data or delete/redefine May be used to prevent out of space conditions or to migrate from one volume size to another
Storage pool striping Adaptive multi-stream pre-fetching FlashCopy space efficient (SE) New Secure Socket Layer (SSL) option for call home New console, the System Storage Productivity Center (SSPC) to manage the full data center from a single point.
R4: Announcing new features for ALL models 450 GB/15,000 RPM DDMs 1 TB / 7.2 RPM FATA DDMs 73 GB and 146 GB SSDs RAID 6 DSCLI and Storage Manager changes GUI response time improvement on panel loads Disk encryption Variable LPAR Extended address volumes IPv6 Secure data overwrite service offering Laptop HMC
R5: Announcing a complete new model! new IO towers new DA cards new CECs new CPUs (Power6) increased performance increased stability easier upgrade path
more in a separate presentation!
Web resources: DS8000 microcode information

Storage support search
http://www-912.ibm.com/TotalStorageSearch/index.jsp
Search = DS8000 Microcode Release Notes (Look under technical documents) Search = DS6000 Microcode Release Notes (Must initiate download process to view)
DS8000
General support
http://www.ibm.com/servers/storage/support/disk/ds8300/
DS8000 code bundle cross-reference (contents and supported code levels)

http://www.ibm.com/support/docview.wss?uid=ssg1S1002949
Microcode bundle release note information

DS8000 CLI and Storage Manager client software levels and download
DS8000 supported operating systems for servers

IBM:
System i: OS/400, i5/OS, Linux, and AIX System p: AIX and Linux System z: z/OS, z/VM, and Linux Windows, Linux, VMware, and NetWare HP-UX AlphaServer: Tru64 UNIX Solaris OSX IRIX
Intel servers:
Hewlett-Packard: Sun:
Apple Macintosh SGI Origin servers: Fujitsu Primepower

Check the System Storage Interoperability Center (SSIC) for complete and updated Information:
http://www-03.ibm.com/systems/support/storage/config/ssic/displayesssearchwithoutjs.wss?start_over=yes
Host connectivity: IBM SDD and MPIO

SDD provides the following functions:
Enhanced data availability Automatic path failover Dynamic I/O load balancing across multiple paths Path selection policies for the host system Concurrent download of licensed machine code
With DS6000 and DS8000, SDD is supported on the following operating systems:
Windows NetWare AIX HP-UX Sun Solaris Linux
SDD can coexist with RDAC (DS4000 multipath driver) on most operating systems as long they manage separate HBAs. SDD cannot be used with most other multipath drivers (in other words, Veritas, PV-Links, Powerpath)
Interfaces to manage DS8000 (1 of 2)

IBM System Storage DS Storage Manager GUI (Web-based GUI)
Program interface used to perform logical configurations and Copy Services management functions Installed via GUI (graphical mode) or unattended (silent mode) Accessed through Web browser Offers:
Simulated configuration (offline) Real-time configuration (online)
Logical configuration and Copy Services for a network-attached storage unit Create, modify, save logical configuration when disconnected Apply to a network-attached storage unit
DS command line interface (DSCLI: script-based)

Open hosts invoke and manage FlashCopy, Metro and Global Mirror functions
Handle batch processes and scripts Check storage unit configuration and perform specific application functions For example:
Check and verify storage unit configuration Check current Copy Services configuration used by storage unit Create new logical storage and Copy Services configuration settings Modify or delete logical storage and Copy Services configuration settings
Both
Available for several operating systems

Interfaces to DS8000 (2 of 2)
DS open application programming interface (API)
Non-proprietary storage management client application supporting:
Routine LUN management activities (creation, mapping, masking) Creation or deletion of RAID volume spaces Copy Services functions: FlashCopy, PPRC
Helps to integrate configuration management support into existing storage resource management (SRM) applications Enables automation of configuration management through customerwritten applications Complements the use of Web-based DS-SM and script-based DSCLI Implemented through IBM System Storage Common Information Model (CIM) agent Uses CIM technology to manage proprietary devices as open system devices through storage management applications Allows these applications to communicate with a storage unit Used by TPC for disk
Middleware application providing CIM-compliant interface
DS8000 HMC for management and remote support
5.1
DS8000 Storage-Hardware Management Console

Focal point for: Dedicated workstation installed inside DS8000
Similar to eServer POWER5 HMC Also known as Storage System Management Console (MC) Dedicated workstation physically installed inside your system Automatically monitors the state of system Notifies user and IBM when service required (call home) Connected to customer network
Enables centralized management through GUI, CLI, or open API
Configuration, Copy Services, maintenance
External management console (optional, feature code 1100) Internal management console (feat code 1110)
For redundancy with high availability
DS8000 Storage-Hardware Management Console

Provides the following:
Local service
Interface for local service personnel
Remote service
Call home and call back
Storage facility configuration

LPAR management (HMC) Supports logical storage configuration via preinstalled System Storage DS Storage Manager in online mode only
Service appliance (closed system)
Network Interface Server for logical configuration and invocation of advanced Copy Services functions Connection to storage facility (DS8000) through redundant private Ethernet networks only
DS8000 HMC and POWER5 HMC

AIX
Partition 1
AIX
Partition 2
POWER5 Hypervisor
Unassigned Resources
Ethernet
HMC
Status Command/Response Virtual Consoles
Non-Volatile RAM
Processors Mem Regions I/O Slots
LPAR Allocation Tables
Service Processor
Perm Temp
P5 HMC features:
Logical partition configuration Dynamic logical partitioning Capacity and resource management System status HMC management Service functions (microcode update, ) Remote HMC interface
DS8000 HMC and a pair of Ethernet switches

Every DS8000 base frame comes with a pair of Ethernet switches installed and cabled to the processor complex. The HMC has: 3 Ethernet ports
2 connect to the private Ethernet switches 1 connects to the customer network
One PCI modem for asynchronous Call home support. Corresponding private Ethernet ports of the external HMC (FC1110) would be plugged into port 2 of the switches as shown in next foil. To interconnect two DS8000 base frames, FC1190 would provide a pair of 31m Ethernet cables to connect from port 16 of each switch in the second base frame into port 15 of the first frame.
If the second HMC is installed in the second DS8000, it would remain plugged into port 1 of its Ethernet switches.
DS8000 HMC and Ethernet switch plugging
PCI Modem
DS8000 HMC: Network configuration

HMC network consists of:
Redundant private Ethernet networks for connection to the Storage Facility(ies) Customer network configured to allow access from the HMC to IBM through a secure Virtual Private Network (VPN)
Call home to IBM Services is possible through dial-up (PCI Modem in the HMC) or Internet connection VPNs Dial-up or Internet connection VPNs are also available for IBM service to provide remote service and support Recommended configuration is to connect HMC to customers public network for support Network connectivity and remote support is managed by the HMC
Support will use WebSM GUI for all service actions Downloading of problem determination data favors the use of a high-speed network
New Laptop version of the HMC with R4.2

Laptop HMC replace the xSeries HMC
ThinkPad W500 Model 4061-AP1. Will ship on new boxes, since 2009 No field replacement for previous xSeries HMC. The laptop HMC is available for internal and external (rack mounted) HMC.
Laptop HMC Mounting Tray
This view shows the laptop HMC display opened and ready to use. Note that the USB cables plugged into the side of the laptop HMC and the power cord at the rear. You can also see the power cord transformer strapped to the rear of the tray with Velcro. To open the DVD tray, the laptop HMC can rotated in clockwise direction.
Laptop HMC DVD Access
Here the turntable and laptop HMC are fully rotated 90 degrees and the DVD drives DVD tray is extended.
Laptop HMC Network and Modem Connections

eth 2 is an RJ-45 (Ethernet) connector and attaches to the customers network using a standard Ethernet cable eth 0 and 3 are USB (Universal Serial Bus) connectors that attach to the private black and grey network using USB cables with a USB to Ethernet cable adaptor. These two USB to Ethernet cable adaptors are held secure in a bracket at the rear of the laptop HMCs mounting tray. The USB Modem connection is also a USB (Universal Serial Bus) connector
DS8000 remote access features

Call home (outbound connectivity)
Automatic problem reporting DS8000 is designed with a call home function
In the event of a failure, the call home function generates a trouble ticket with the IBM support organization IBM support determines the failing component and dispatches a customer engineer with the replacement part
Remote service and support (inbound connectivity)

With remote support enabled, IBM technical support can log into the HMC to troubleshoot a problem and view logs, dumps, and traces interactively This can reduce lag time to send such information to IBM and can shorten problem determination time In the case of complex problems, IBM technical support teams can engage a specialist quickly to resolve the problems as quickly as possible
DS8000 HMC network topology
DS8000 Subsystems
Customer Customer Network Network
DMZ VPN
Redundant HMC Ethernet Fabric eth eth modem eth integrated Firewall Proxy
Opt. Firewall provided by customer
Internet Internet
VPN DMZ
HMC: DMZ: VPN: eth:
Hardware Management Console Demilitarized Zone Virtual Private Network Ethernet Port
IBM IBM Network Network

IBM Remote Support infrastructure
How virtual private network (VPN) operates

The VPN server is located behind IBM firewall, which is designed to be secure The VPN client is located behind the customer firewall
The customer has control over opening a connection to access the client Neither IBM technical support nor non-authorized personnel can access the client without customers permission
The VPN Server security complies with IBM corporate security standards ITCS104
This is an IBM internal security measure for all IBM secure data.
Remote support (Inbound)

VPN Gateway Firewall
VPN Tunnel
DS-6000 SMC Firewall
RS3
IB M S u p p o rt
Call home (Outbound)
Customer Site
IBM Site
DS8000 HMC: Remote service security

Server authentication via private/public key:
Each HMC generates a certificate based on the private key that the HMC will use for Secure Sockets Layer (SSL) based encryption and decryption. The IBM SSR transmits the certificate for the installed HMC to a database maintained within the IBM secure network. IBM personnel then will retrieve the HMC specific certificate from the database and use this key (public key) to establish the communication session with the HMC needed service.
SSH over VPN for command line access:

Secure Shell (SSH) is used for command line access from a remote IBM location (for example: putty ssh session with public key). The SSH daemon on the HMC accepts client connections only if an IBM VPN is up and a Product Engineer is currently logged on to the HMC. SSH client authentication is done through private/public key Copyright IBM Corporation 2009 algorithm.
Hardware components
5.1
DS8000: Primary frame topology

Standard 19in rack mounting space
Dense HDD Packaging

16 drives per pack
Dual FC-AL Loop Switches

Point to Point Isolation Tw o Simultaneous Operations per loop
Redundant Power
Storage-Hardware Maintenance Console Processor Complex IBM eServer p5 570

Dual 2-w ay or Dual 4-w ay
4 I/O Enclosure Bays

Each bay supports 4 Host Adapters and 2 Device Adapters
BBU:
Battery Backup Units
Host Adapter
4 FCP/FICON Ports or 2 ESCON Ports
Front
Device Adapter
4 FC-AL Ports
Rear
DS8000 terminology
Storage complex Storage unit
A group of DS8000s managed by a single Storage-Hardware Management Console A single DS8000 including expansion frames. One P5-570 p Series server Two processor complexes form a redundant pair Divided into one LPAR (models 931 or 932) or two LPARs (model 9B2) The software that uses an LPAR: Has access to a percentage of resources available on the processor complex for the LPAR At GA, this percentage is 50% (model 9B2) or 100 % (models 931 or 932) Union of 2 LPARs, one from each processor complex Each LPAR hosts one storage server.
Processor complex
Storage server
Storage facility image (SFI)
DS8000 data flow

The normal flow of data for a write is the following:
1. Data is written to cache memory in the owning server. 2. Data is written to NVS memory of the alternate server. 3. The write is reported to the attached host as having been completed. 4. The write is destaged from the cache memory to disk. 5. The write is then discarded from the NVS memory of the alternate server.
Under normal operation, both DS8000 servers are actively processing I/O requests.
DS8000 hardware components detail
Processor complex
Processor complex
DS8000 processor complex (1 of 3)
Processor complex
Processor complex
DS8000 processor complex: POWER5 server
CEC enclosures in the Model 921/931 each have one processor card (2-way) CEC enclosures in the Model 922/932 and 9A2/9B2 each have two processor cards (4-way)
CEC: Computer Electronic Complex CEC enclosures contain components such as the processor cards, cache memory, and CEC hard drives.
IBM eServer p570

Scales from a 1-way to a 16-way SMP using 4U building blocks Dynamic LPAR and micro-partitioning Simultaneous Multi-threading (SMT) 2-way Self-healing features 4-way
Bit-steering (bit sparing) Chipkill ECC (8-bit packet correct) ECC on processor cache memories L3 cache line deletes Memory scrubbing Dynamic processor deallocation N+1 power and cooling Hot-plug PCI In-place service First fault data capture
8-way 12-way 16-way
Other RAS attributes
Near Linear Scaling
Optimized for storage
High I/O bandwidth RIO-2 Large robust memories 4K memory allocation

Thread0 active Thread1 active No Thread active
FX0 FX1 LS0 LS1 FP0 FP1 BRX CRL
Execution units utilization


Complex is comprised of IBM eServer System p POWER5 servers (921, 922, and 9A2) New DS8000 Turbo (931, 932, and 9B2) are using POWER5+ processors
15 % performance improvement. 2.2 GHz for POWER5+ 2- and 4-way 2-way 1.5 GHz (3X on ESS 800) 4-way 1.9GHz (6X on ESS 800)
The POWER5 processor supports logical partitioning
The p5 hardware and Hypervisor manage the real to virtual memory mapping to provide robust isolation between LPARs. IBM has been doing LPARs for 20 years in mainframes and 8 years in System p. A 4-way has two processors to one LPAR and two processors to the other LPAR. LPARs only possible in the 4-way P5s (RIO-G cannot be shared in 2-way).
Post GA, 25-75 possible.
At GA, LPARs are split 50-50, so:
Cache memory ranges from 16 GB to 256 GB Persistent memory ranges from 1 GB to 8 GB: dependent on cache size Copyright IBM Corporation 2009
DS8300 model 8A2/9B2 4-way with LPARs
Server LPAR concept overview

An LPAR:
Uses hardware and firmware to logically partition resources Is a subset of logical resources that are capable of supporting an operating system Consists of CPUs, memory, and I/O slots that are a subset of the pool of available resources within a system
Very flexible granularity according to AIX level (5.2, 5.3, and so on) No need to conform to physical boundaries of building blocks
In an LPAR:
An operating system instance runs with dedicated (AIX 5.2) or shared (AIX 5.3) resources: processors, memory, and I/O slots These resources are assigned to the logical partition The total amount of assignable resources is limited by the physically installed resources in the system Isolation between LPARs to prevent unauthorized access between partition boundaries Fault isolation such that one LPARs operation does not interfere with the operation of other LPARs Support for multiple independent workloads, different operating systems, operating system levels, applications, and so on
LPARs provide:
LPAR applied to storage facility images (SFI)
Processor compl ex 0 Storage Facili ty Image 1 Storage Facili ty Image 2
Processor compl ex 1
LPAR11
DS8300
Comprised of two eServers P5 570 (Processor complex) Each processor complex supports one or more LPARs Currently, each processor complex divided into two LPARs
LPAR01
LPAR02
LPAR12
An LPAR in a processor complex

Set of resources to support exec of an operating system
LPARxy x=Pr ocessorcom plex num ber y=Stor age facility num ber
Delivered AS IS, no need using the HMC to configure
(Persistent memory)
DS8000 persistent memory

The 2107 does not use NVS cards, NVS batteries, or NVS battery chargers Data that would have been stored in the 2105 NVS cards resides in the 2107 CEC cache memory
A part of the system cache is configured to function as NVS storage
In case of power failure, if the 2107 has pinned data in cache, it is written to an extra set of two disk drives located in each of the CEC enclosures Two disk drives total in each CEC:
For LIC (LVM Mirrored AIX 5.3 + DS8000 code) For pinned data and other CEC functions
During the recovery process, the pinned data can be restored from the extra set of CEC disk drives just as it would have been from the NVS Copyright IBM Corporation cards on2009 the ESS 800
DS8000 I/O enclosure
Processor complex
Processor complex
RIO-G and I/O enclosures

Also called I/O drawers Contain six PCI-X slots: 3.3V, 133 MHz blind swap hot-plug:
Four port host adapter cards with four ports each: FCP or FICON adapter ports Two device adapter cards with four ports each: Four FC-AL ports per card Two FC-AL loops per card
Accesses cache via RIO-G internal bus Each adapter has its own PowerPC processor Owned by processors in LPAR Uses system power control network (SPCN)
Controls and monitors the status of the power and cooling within the I/O enclosure Cabled as a loop between the different I/O enclosures
DS8000 I/O enclosures (aka I/O drawers)
SPCN : System Pow er Control Netw ork
DS8000 RIO-G port: Layout example

Up to four I/O enclosures in same RIO-G loop Up to 20 I/O enclosures to P5-570 system
Max effective bandwidth: 2000 MB/SEC per RIO-G loop,
Each RIO-G port can operate at 1 GHz in bidirectional mode and is capable of passing data in each direction on each cycle of the port. Maximum data rate per I/O Enclosure: 4 GB/s . It is designed as a high performance self-healing interconnect. The p5-570 provides two external RIO-G ports, and an adapter card adds two more. Two ports on each processor complex form a loop. Figure shows an illustration of how the RIO-G cabling is laid out in a DS8000 that has eight I/O drawers. This would only occur if an expansion frame were installed. The DS8000 RIO-G cabling will vary based on the model.
DS8000 host adapters
Processor complex
Processor complex
Host adapter with four fibre channel ports

Configured each as FCP or FICON
More FICON logical paths: ESS (1024) versus DS8000 (2048) One FICON channel addresses 16,384 devices One HA card covers all the 65,280 devices that a DS8000 supports (64k -256) Up to 16 HA into a DS8100 or 32 HA into a DS8300 16 FICON channel ports to each single device Current System z channel subsystems limited to eight channel paths per device Front end of: 128 ports for DS8300 (8 times ESS) 64 ports for DS8100 (4 times ESS)
DS8000 FCP/FICON host adapters

DS8000 has four LC 2 Gb or 4 Gb FC ports (two host adapter models) Ports auto-negotiates to 1 Gbps, 2 Gbps, or 4 Gbps
Each port independently auto-negotiates to either 1/2 Gbps link speed on 2 Gb host adapter models or 2/4 Gbps link speed on 4 Gb host adapter models.
Ports can be independently configured to FCP or FICON protocols
The personality of the port is changeable via the DS storage management tools (GUI or CLI).
Ports cannot operate as FCP and FICON simultaneously FCP port can be longwave or shortwave
Shortwave ports support a distance of 300m (non-repeated) Longwave ports support a distance of 10Km (non-repeated)
Note: For FCP, configure the ports as follows:

Switched point-to-point for fabric topology FC-AL for point-to-point topology
DS8000 FICON/FCP host adapters
QDR Fibre Channel Protocol Engine
PPC 750GX
Processor 1 GHz
Fibre Channel Protocol Engine
Data Protection Data Mov er ASIC Flash
Buf f er
Protocol Chipset
QDR
Data Mover
PCI-X 64 Bit 133 MHz Four 2 or 4 Gbps Fibre Channel ports
New high function/high performance ASIC Metadata creation/checking Configured at port level Fibre or FICON
SW or LW
DS8000 4 Gb host adapter performances

New 4 Gb host adapters are designed to improve by 50% single port throughput performance .
4 Gb / 2 GB HA performance comparison
DS8000 device adapters
Processor complex
Processor complex
Fibre channel device adapters with 2 Gbps ports

DA performs RAID logic
Offloads servers of that workload Each port has up to five times the throughput of previous SSA-based DA ports DS8000 configured for array across loops (AAL) Eight RAID 5, 6 or RAID 10 DDMs spread over two loops
DS8000 device adapters

Device adapters support RAID 5, RAID 6 or RAID 10 FC-AL switched fabric topology FC-AL dual ported drives are connected to FC switch in the disk enclosure backplane Two FC-AL loops connect disk enclosures to device adapters Array across loops is standard configuration option in DS8000
Two simultaneous I/O ops per FC-AL connection possible Switched FC-AL or switched bunch of disks (SBOF) used for back-end access
Device adapters are attached to a FC switch with the enclosure Four paths to each drive: 2 FC-AL loops X dual port access
(Detailed later with storage enclosures cabling)
DS8000 RAID device adapter
PPC 750FX
NVRAM
Processor 500 MHz

SDRAM
Bridge
Fibre Channel Protocol Engine RAID: Data Protection Data Mov er ASIC Buf f er
Fibre Channel Protocol Engine
Protocol Chipset
Data Mover
PCI-X 64 Bit 133 MHz
Four 2 Gbps fibre channel ports New high function/high performance ASIC Metadata checking
Performance evolution: From the model 800 to the DS8000
Device adapter
Device Adapter
I/O enclosure slot numbering rear view

Host Adapter 0 Host Adapter 1 Device Adapter 2 RIO Adapter 6 Host Adapter 3 4 Host Adapter 0 1 2 3 5 Device Adapter
cards counted left to right, starting with 0 ports counted top to bottom, starting with 0 naming scheme in DSCLI (lsioport): I0XYZ X=enclosure number (0-7) Y=card number (0-5) Z=port number (0-3) e.g. I0312: IO enclosure 3 (1st rack, bottom,
right)
card 1 (2nd slot) port 2 (3rd from top)

58
I/O ports numbering DS CLI lsioport display base frame

I0000 I0001 I0002 I0003 Slot 1 I0010 I0011 I0012 I0013 I0030 I0031 I0032 I0033 I0040 I0041 I0042 I0043 Slot 5 I0100 I0101 I0102 I0103 Slot 1 I0110 I0111 I0112 I0113 I0130 I0131 I0132 I0133 I0140 I0141 I0142 I0143 Slot 5
Slot 2 Slot 4 Enclosure 1 I0210 I0211 I0212 I0213 I0230 I0231 I0232 I0233
I0200 I0201 I0202 I0203 Slot 1
I0240 I0241 I0242 I0243 Slot 5
I0300 I0301 I0302 I0303 Slot 1
I0340 I0341 I0342 I0343 Slot 5
Slot 2 Slot 4 Enclosure 3
I/O ports numbering DS CLI lsioport display expansion frame

I0400 I0401 I0402 I0403 Slot 1 I0410 I0411 I0412 I0413 I0430 I0431 I0432 I0433 I0440 I0441 I0442 I0443 Slot 5 I0500 I0501 I0502 I0503 Slot 1 I0510 I0511 I0512 I0513 I0530 I0531 I0532 I0533 I0540 I0541 I0542 I0543 Slot 5
I0600 I0601 I0602 I0603 Slot 1
I0640 I0641 I0642 I0643 Slot 5
I0700 I0701 I0702 I0703 Slot 1
I0740 I0741 I0742 I0743 Slot 5
16 drive disk enclosure

Backplane Top Half (8 DDMs)
Bottom Half (8 DDMs) Interface card (FCIC) Disk Drive Module (DDM)
DS8000: Disk enclosures installed in pairs: one in front and one in back
ISS loop architectural overview
Switched connections to each DDM Each DDM connected to 2 switches Each switch connected to 2 device adapters Each device adapter belongs to a different CEC 4 connections to each DDM high redundancy
62
DS8000 FC-AL/ switched FC-AL

FC-AL
Loop supports only one operation at a time Arbitration of competition Intermittent failure issues Increasing time as number of devices grows
Switched FC-AL
Drives attached in point-to-point connection Faster arbitration message processing 200 MB/sec external transfer rate Improved RAS Switch detects individual failures Intermittent/permanent
Switched FC-AL advantages

DS8000 uses switched FC-AL technology to link the device adapter (DA) pairs and the DDMs. Switched FC-AL uses the standard FC-AL protocol, but the physical implementation is different. The key features of switched FC-AL technology are:
Standard FC-AL communication protocol from DA to DDMs Direct point-to-point links are established between DA and DDM No arbitration and no performance degradation Isolation capabilities in case of DDM failures provide easy problem determination Predictive failure statistics Simplified expansion: No cable rerouting required when adding another disk enclosure
The DS8000 architecture employs dual redundant switched FC-AL access to each of the disk enclosures. The key benefits of doing this are:
Two independent switched networks to access the disk enclosures Four access paths to each DDM in DS8000 architecture (dual switches) Each device adapter port operates independently Double the bandwidth over traditional FC-AL loop implementations
Each DDM is attached to two separate fibre channel switches.

This means that with two device adapters, we have four 2Gb/sec effective data paths to each disk
When a connection is made between the device adapter and a disk, the connection is a switched connection that uses arbitrated loop protocol.
This means that a mini-loop is created between the device adapter and the disk Results in four simultaneous and independent connections, one from each device adapter port
DS8000: Storage enclosure and DA cabling
Architecture: Major points for performance
5.1
DS8000 frames
Base frame:
The base frame contains two processor complexes: eServer p5 570 servers
Each of them contains the processor and memory that drive all functions within the DS8000.
The base frame can contain up to eight disk enclosures; each can contain up to 16 disk drives. The base frame contains four I/O enclosures.
In a maximum configuration, the base frame can hold 128 disk drives. I/O enclosures provide connectivity between the adapters and the processors. The adapters contained in the I/O enclosures can be either device or host adapters (DAs or HAs).
The communication path used for adapter to processor complex communication is the RIO-G loop.
Expansion frames:
Each expansion frame can hold up to 16 disk enclosures which contain the disk drives. Expansion frames can contain four I/O enclosures and adapters if they are the first expansion frame that is attached to either a model 932 or a model 9B2.
In a maximum configuration, an expansion frame can hold 256 disk drives.
IBM System Storage DS8100 (2-way)
Up to 128 disks
Power supplies HMC
IBM eServer System p POWER5 servers
Batteries
I/O drawers
DS8300 (4-way with two expansion frames)

Power supplies HMC
Up to 640 Disks
p5 (POWER5) servers Batteries I/O drawers
DS8300 (4-way with 4 expansion frames)
(maximum configuration)
DS8100 (model 921/931) 2-way

Up to 16 host adapters (HA)
FCP/FICON HA: Four independent ports ESCON HA: Two ports
Up to 4 device adapter (DA) pairs

DA pairs 0 / 1 / 2 / 3 Automatically configured from DDMs
2 2 0 0 C0 C1
3 3 1 1 2 2 0 0
b
Maximum configuration (384 DDMs)

DA pair 0 = 128 DDMs DA pair 1 = 64 DDMs DA pair 2 = 128 DDMs DA pair 3 = 64 DDMs
Balanced configuration at 256 DDMs: In other words, 64 DDMs per DA pair DA (card) plugging order: 2 / 0 / 3 / 1
0/1 1/0 2/3 3/2
DS8300 (models 922/932 and 9A2/9B2) 4-way

Up to 32 host adapters
FCP/FICON HA: Four independent ports ESCON HA: Two ports
Bad decision same adapter
Good idea for a pool: 2 adapters
Up to eight DA pairs
DA pairs 0 to 7 Automatically configured from DDMs
2 2 0 0 C0 C1
6 6 4 4 7 7 5 5
b
3 3 1 1 2 2 0 0
b
Maximum configuration (640 DDMs)

DA pairs 1, 3-7 = 64 DDMs DA pairs 2, 0 = 128 DDMs Balanced configuration at 512 DDMs: In other words, 64 DDMs per DA pair DA (card) pair plugging order: 2/0/6/4/7/5/3/1
0/1 1/0 2/3 3/2
4/5 5/4 6/7 7/6 Bad decision same adapter
DS8300 with five frames

Cabling diagram for third and fourth expansion frames
2 2 0 0 C0 C1
b
6 6 4 4 7 7 5 5 4/5 5/4 6/7 7/6
3 3 1 1 2 2 0 0
b
6 6 4 4 7 7 5 5
b
3 3 1 1 0 0 1 1
b
0/1 1/0 2/3 3/2
4/6 6/4
DS8000 cache management
SARC & AMP
5.1
Sequential prefetching in adaptive replacement cache (SARC)

SARC basically attempts to determine four things:
When data is copied into the cache Which data is copied into the cache Which data is evicted when the cache becomes full How the algorithm dynamically adapts to different workloads
SARC uses:
Demand paging for all standard disk I/O Sequential pre-fetch for sequential I/O patterns
DS8000 caching using SARC

Best caching algorithms in industry Over 20 years experience Features
Self-learning algorithms Adaptively and dynamically learn what data should be stored in cache based upon the recent access and frequency needs of the hosts Adaptive replacement cache Most advanced and sophisticated algorithm to determine what data in cache is removed to accommodate newer data Prefetching Predictive algorithm to anticipate data prior to a host request and loads it into cache Leading performance Been proven to improve cache hit by up to 100% over previous IBM caching algorithms and improve I/O response time by 25% More efficient use of cache Intelligent caching algorithm profiles host access patterns to determine what data is stored Need less cache than competitors
Cache Hit Ratio
Benefits of adaptive replacement caching
1 0.8 0.6 0.4 0.2 0 0 64 128 192 256 Cache Size (GB) z/OS Open
Benefits
Nimrod Megiddo and Dharmendra S. Modha, "Outperf orming LRU with an Adaptiv e Replacement Cache Algorithm," IEEE Computer, pp. 4-11, April
Copyright 2004. IBM Corporation 2009
DS8000 caching using AMP

What is AMP?
A breakthrough caching technology from IBM Research called Adaptive Multistream Prefetching (AMP)
Can dramatically improve performance for common sequential and batch processing workloads
AMP optimizes cache efficiency by incorporating an autonomic, workloadresponsive, self-optimizing prefetching technology .
The algorithm dynamically decides what to prefetch and when to prefetch Delivers up to a two-fold increase in the sequential read capacity of RAID 5 arrays The bandwidth for a fully configured DS8000 remains unchanged May improve sequential read performance for smaller configurations and single arrays Reduces the potential for array hot spots due to extreme sequential workload demands May significantly reduce elapsed time for sequential read applications constrained by array bandwidth such as BI and critical batch processing workloads
AMP doubles sequential read bandwidth for a single RAID 5 array
DS8000 RAS features (Reliability, availability, and serviceability)
5.1
Processor complex RAS

Processor complex has the same RAS features as the P5-570, which is an integral part of the DS8000 architecture. IBM Server p5 system main RAS features:
First Fault Data Capture Boot process and operating system monitoring Environmental monitoring Self-healing Memory reliability, fault tolerance, and integrity
Error Checking Correction (ECC) Memory scrubbing and thresholding
N+1 redundancy Resource deallocation Concurrent maintenance
Server RAS (1 of 2)
The DS8000 provides data integrity when performing write operations and server failover.
Metadata check: The metadata is checked by various internal components to validate the integrity of the data as it moves throughout the disk system or sent back to the host. Server failover and failback:
LSS and server affinity:
LSS with even number have an affinity with server 0 LSS with odd number have an affinity with server 1
When a host operating system issues a write to a logical volume, the DS8000 host adapter directs that write to the server that owns the LSS to which that logical volume is a member.
Server RAS (2 of 2)
Under normal operation, both DS8000 servers are actively processing I/O requests
Each write is placed into the cache memory of the server owning the volume and also into the NVS memory of the alternate server.
Failover: In case of one server failure, the remaining server is able to take over all of its functions
RAID arrays which are connected to both servers can be accessed from the device adapters of the remaining server. Since the DS8000 has only one copy of data in cache of remaining server, it will now take the following steps:
It de-stages the contents of its NVS to the disk subsystem. The NVS and cache of remaining server are divided in two, half for the odd LSSs and half for the even LSSs. Remaining server now begins processing the writes (and reads) for all the LSSs.
Failback: When the failed server has been repaired, failback process is activated
It starts in less than 8 seconds, will finish in less than 15 minutes, and is invisible to the attached hosts.
Hypervisor: Storage image independence

Logical view: virtual Storage Facility images Physical view: physical storage unit
Storage Facility image 1
L IC M em ory P ro cessor R IO -G I/O I/O L IC M em ory P ro cesso r
Storage Facility image 2

L IC M em ory P rocessor R I O -G I/O I/O L IC M em ory P rocessor
LPAR Hypervisor
takes par tof takes par tof takes par tof
RIO-G I/O Memory Processor I/O Memory Processor
Server failover
Normal flow of data for a write:
1. Data is written to cache memory in the owning server. 2. Data is written to NVS memory of the alternate server. 3. The write is reported to the attached host as having been completed. 4. The write is destaged from the cache memory to disk. 5. The write is then discarded from the NVS memory of the alternate server.
NV S f orodd LS Ss NV S f oreven LS S s
Cache m em or y f oreven LS Ss Ser ver 0
Cache m em or y f orodd LS S s Ser ver 1
After a failover, remaining server processes all I/Os with cache and NVS divided by two, one for odd LSSs and one for even LSSs.
NV S f orodd LS S s
V S N V SN o r f o r f o dd even LS S s LS S s
Cache m em or y f oreven LS S s Server 0
C ach eC ach e f o r f or even o dd LS S s LS S s
Ser ver 1
Failover
NVS recovery after complete power loss

DS8000 preserves fast writes Battery backup units (BBUs) ensures fast writes are not lost Both power supplies stopped
Batteries not used to keeping disks spinning Scenario at power-off
All HA I/O blocked Each server copies NVS data to internal disk Two copies made per server When copy process complete, each server shuts down AIX When AIX shutdown complete for both servers (or time out expires), the DS8000 is powered down
Scenario at power-on
Processor complexes power-on and perform power-on self-test Each server boots up During boot-up, each server detects NVS data on its disks and destages it to FC-AL disks When battery units reach a certain level of charge, the servers come online
NVS contents preserved indefinitely Note: The servers will not come online until the batteries are fully
Host connection availability

On DS8000 host, adapters are shared between the servers. It is preferable for hosts to have at least two connections to separate host adapters in separate I/O enclosures.
This configuration allows the host to survive a hardware failure on any component on either path. This is also important because during a microcode update, an I/O enclosure may need to be taken offline.
Multi-pathing software help ensure availability.

Subsystem Device Driver (SDD) is able to manage both path failover and preferred path determination.
SDD is usable with ESS800, DS6000, DS8000, or SVC.
Disk subsystem (1 of 2)
RAID 5, RAID 6, and RAID 10
RAID 5 (7+P or 6+P+S) or RAID 10 (2x4 or 2x3 + 2S)
RAID 6 adds extra parity drive, the Q drive, now 6+P+Q versus 7+P or 5+P+Q+S versus 6+P+S
DS8000 does not support non-RAID configurations (JBODs).
Spare disk creation
A minimum of one spare is created for each array site defined until the following conditions are met:
A minimum of four spares per DA pair A minimum of four spares of the largest capacity array site on the DA pair A minimum of two spares of capacity and RPM greater than or equal to the fastest array site of any given capacity on the DA pair
Floating spare
The DS8000 microcode may choose to migrate new spare disks to a more optimal position to better balance the spares across the DA pairs, the loops, and the enclosures.
Useful after a drive replacement that became a spare drive
Disk subsystem (2 of 2)
Each DDM attached to two FC switches
Each disk has two separate connections on the backplane.
Each DA connected to the two switches DDMs hot-pluggable Incorporates predictive Failure Analysis (PFA)
Anticipates failures
Performs disk scrubbing

All disk sectors periodically read and bad bits corrected (incl. spares)
Power and cooling

Completely redundant power and cooling in N+1 mode Battery backup units (BBU)
Used for NVS (part of the servers memory) Can be replaced concurrently
Rack power control cards (RPC)

Two RPC cards for redundancy Each card can control power of an entire DS8000
Power fluctuation protections

DS8000s tolerate a momentary power interruption for approximately 30 ms. After that time, servers start copying content of NVS to internal SCSI disks.
Microcode update
Concurrent code update Installation process:
HMC can hold six different versions of code Each server can hold three different versions of code Internal HMC code update New DS8000 LIC downloaded on the internal HMC LIC uploaded from HMC to each DS8000 server internal storage New firmware can be loaded from HMC directly into each device
May require server reboot with failover of its logical subsystems to the other server
Update of servers operating system and LIC

Each server updated one at a time with failover of its logical subsystems to the other server
Host adapters firmware update

Each adapter impacted for less than 2.5 s, which should not affect connectivity (Fast Load) Longer interruption managed by hosts multipathing software
Storage-Hardware Management Console

Redundant Ethernet switches
Each switch used in a separate Ethernet network with non-routable private IP addresses assigned in networks 172.16/16 and 172.17/16 192.168.16.x and 192.168.17.x 10.0.16.x and 10.0.17.x
Redundant HMCs
Each DS8000 can be connected via the redundant Ethernet switches to both HMCs.
Unit summary
Having completed this unit, you should be able to: Discuss the hardware and architecture of the DS8000 Use virtualization terminology describing configuration of the DS8000 subsystem Describe the physical hardware components and resources Describe the models and features provided by each model Describe the types of disk arrays that can be configured for a DS8000 subsystem
ESCC Enterprise Storage Competence Center
DS8000 Models Overview R2

2107-931
Base rack, 2 way
2107-932
Base rack, 4 way
2107-9B2
Base rack, 4 way, LPAR model
242x-9yy
New models which include the warranty in the model number x: 1, 2, 3, 4 (years of warranty) For all rack models
242x-92E
Expansion rack to a 931 or 932
242x-9BE
Expansion rack to a 9B2
93 DS8000 | Concepts and Architecture | K. Jehnen 2008 IBM Corporation

Unit3 Architecture 221209

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Unit3 Architecture 221209

Hochgeladen von

Copyright:

Verfügbare Formate

Unit 3: IBM DS8000 architecture and hardware overview

Significant extensions to enable scalability

DS8000 series models (2107)

IBM POWER5 server technology

DS8000 models consist of:

Integrated with the IBM Virtualization Engine technology

Storage unit One or two (recommended) Management Consoles (MC)

Performing logical configurations and Copy Services management functions

Copyright IBM Corporation 2009

Copyright IBM Corporation 2009

Copyright IBM Corporation 2009

more in a separate presentation!

Copyright IBM Corporation 2009

Web resources: DS8000 microcode information

DS8000 code bundle cross-reference (contents and supported code levels)

Microcode bundle release note information

Copyright IBM Corporation 2009

DS8000 supported operating systems for servers

Apple Macintosh SGI Origin servers: Fujitsu Primepower

Host connectivity: IBM SDD and MPIO

Interfaces to manage DS8000 (1 of 2)

DS command line interface (DSCLI: script-based)

Available for several operating systems

Middleware application providing CIM-compliant interface

DS8000 HMC for management and remote support

DS8000 Storage-Hardware Management Console

Configuration, Copy Services, maintenance

Copyright IBM Corporation 2009

DS8000 Storage-Hardware Management Console

Storage facility configuration

Service appliance (closed system)

Copyright IBM Corporation 2009

DS8000 HMC and POWER5 HMC

Status Command/Response Virtual Consoles

Processors Mem Regions I/O Slots

LPAR Allocation Tables

Copyright IBM Corporation 2009

DS8000 HMC and a pair of Ethernet switches

DS8000 HMC and Ethernet switch plugging

Copyright IBM Corporation 2009

DS8000 HMC: Network configuration

Copyright IBM Corporation 2009

New Laptop version of the HMC with R4.2

Copyright IBM Corporation 2009

Laptop HMC Mounting Tray

Copyright IBM Corporation 2009

Laptop HMC DVD Access

Copyright IBM Corporation 2009

Laptop HMC Network and Modem Connections

Copyright IBM Corporation 2009

DS8000 remote access features

Remote service and support (inbound connectivity)

DS8000 HMC network topology

Customer Customer Network Network

Opt. Firewall provided by customer

HMC: DMZ: VPN: eth:

IBM IBM Network Network

Copyright IBM Corporation 2009

How virtual private network (VPN) operates

Remote support (Inbound)

Call home (Outbound)

DS8000 HMC: Remote service security

SSH over VPN for command line access: