<Insert Picture Here>

Oracle Exadata Storage Server Architecture Overview

Agenda

Exadata Software Architecture Exadata Storage Layout Exadata Storage Scale-Out Architecture

2008 Oracle Corporation Proprietary and Confidential

Exadata Software Architecture

2009 Oracle Corporation Proprietary and Confidential

Exadata Architecture
RAC Database DB Server DB Server DB Instance DBRM ASM DB Instance DBRM ASM
Enterprise Manager

Single-Instance Database DB Server

DB Instance DBRM ASM

InfiniBand Switch/Network

iDB Protocol over InfiniBand with Path Failover OEL

OEL OEL
CELLSRV

CELLSRV

MS IORM RS Exadata Cell MS IORM RS Exadata Cell

CELLSRV

MS IORM RS Exadata Cell

Cell Control CLI

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Server Overview

Exadata Storage Server

Data Data

Data

Network Fabric

Evaluate Disk, CPU, and Network Throughput Standalone

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Server Overview

Database Server
RDBMS/ASM instance

Exadata Storage Server

Data Data
SGA

Data
IO Client IO Layer ASM layer

?
Network Fabric

How do we connect the IO/ASM layer of ASM/DB to Exadata?

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Server Overview

Database Server
RDBMS/ASM instance
SGA

Exadata Storage Server

Meta data

Meta data

Meta data

Data Data
IO Client IO Layer cellsrv libcell ASM layer

Data

Network Fabric

libcell linked with DB/ASM talks to cellsrv iDB protocol is born Multiple threads in cellsrv Threads perform asynchronous IO to disks and network 7

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Server Overview

Database Server
RDBMS/ASM instance
SGA

Exadata Storage Server

Meta data

Meta data

Meta data

Data Data
IO Client IO Layer cellsrv libcell ASM layer

Data

Network Fabric

Smart Scan enabled in RDBMS and CELLSRV

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Server Overview

Database Server
RDBMS/ASM instance
SGA

Exadata Storage Server

Meta data

Meta data

Meta data

Data Data
IO Client IO Layer cellsrv libcell ASM layer

Data

/etc/oracle/cell/network /etc/oracle/cell/network-config

Local IP
cellinit.ora

Network Fabric

cellinit.ora decides which network takes Storage traffic

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Server Overview

Database Server
RDBMS/ASM instance
SGA

Exadata Storage Server

Meta data

Meta data

Meta data

Data Data
IO Client IO Layer cellsrv libcell ASM layer

Data

Cells
cellip.ora

/etc/oracle/cell/network /etc/oracle/cell/network-config

Local IP
cellinit.ora

Network Fabric

cellip.ora on database/ASM host maintains list of cells New cells can be added to cellip.ora dynamically
10

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Server Overview

Database Server
RDBMS/ASM instance
SGA

Exadata Storage Server

Meta data

Meta data

Meta data

Data Data
dskm IO Client IO Layer cellsrv libcell ASM layer

Data

Cells

diskmon

cellip.ora

/etc/oracle/cell/network /etc/oracle/cell/network-config

Local IP
cellinit.ora

Network Fabric

Master diskmon (diskmon) starts with CSS and communicates with cellsrv Slave diskmon (dskm) is part of every instance and communicates with master diskmon Handles cell failures, IO fencing, IO resource management plan propagation

2008 Oracle Corporation Proprietary and Confidential

11

Exadata Storage Server Overview

Database Server
RDBMS/ASM instance
SGA

Exadata Storage Server

Meta data

Meta data

Meta data

Data Data
dskm IO Client
cellcli

Data

IO Layer ASM layer libcell diskmon

Cells
cellip.ora

cellsrv ms

/etc/oracle/cell/network /etc/oracle/cell/network-config

Local IP
cellinit.ora

Network Fabric

Cellcli allows user interaction and configuration Management Server (MS) displays and manages creation and deletion of griddisks, changes in hardware, SNMP traps, alerts, email, metrics etc
12

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Server Overview

Database Server
RDBMS/ASM instance
SGA

Exadata Storage Server

Meta data

Meta data

Meta data

Data Data
dskm IO Client
cellcli

Data

IO Layer ASM layer libcell diskmon

Cells
cellip.ora

cellsrv ms

rs

/etc/oracle/cell/network /etc/oracle/cell/network-config

Local IP
cellinit.ora

Network Fabric

Restart Server (RS) monitors CELLSRV and MS, Backup RS monitors core RS RS monitors for process aliveness, memory usage etc

2008 Oracle Corporation Proprietary and Confidential

13

Exadata Storage Server Overview

Database Server
RDBMS/ASM instance
SGA

Exadata Storage Server

Meta data

Meta data

Meta data

Data Data
dskm IO Client
cellcli

Data

IO Layer ASM layer diskmon

Cells
cellip.ora

cellsrv ms
adrci

libcell rs
ADR

/etc/oracle/cell/network /etc/oracle/cell/network-config

Local IP
cellinit.ora

Network Fabric

Trace files and alert logs in Automatic Diagnostic Repository on cell alert.log (from RS and CELLSRV), ms-odl.log, ms-odl.trc, rs*trc, svtrc*.trc

2008 Oracle Corporation Proprietary and Confidential

14

Exadata Storage Server Overview

Database Server
RDBMS/ASM instance
SGA

Exadata Storage Server

Meta data

Meta data

Meta data

Data Data
dskm IO Client
cellcli

Data

IO Layer ASM layer diskmon

Exadata cells
cellip.ora

cellsrv ms
adrci

libcell rs
ADR

/etc/oracle/cell/network /etc/oracle/cell/network-config

Local IP
cellinit.ora

Infiniband Fabric

Infiniband fabric requires RPMs from OFED/OEL, RedHat 5.1, OEL 5.1 Exadata Storage Server works only with 11.1.0.7 Database/ASM

2008 Oracle Corporation Proprietary and Confidential

15

Exadata Storage Server Overview

From the Enterprise Enterprise Manager OMS
cellcli

Exadata Storage Server

Meta data

Meta data

Meta data

Data Data Agent

Data

cellsrv ms
adrci

dcli

rs
ADR

ssh/cellcli

Network Fabric

EM Plugin Provides central location for metrics and alerts across cells No agent runs on cell dcli allows user to run commands across cells
16

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Layout

2009 Oracle Corporation Proprietary and Confidential

17

Exadata Storage Layout

Physical disks map to a Cell Disks Cell Disks partitioned into one or multiple Grid Disks ASM diskgroups created from Grid Disks Transparent above the ASM layer

Grid Disk 1 Cell Disk

Sys Area

ASM disk Grid Disk n

Sys Area

Physical Disk

ASM disk

2008 Oracle Corporation Proprietary and Confidential

18

Exadata Storage Layout Example

Cell Disks

Cell Disk
Exadata Cell Exadata Cell

Cell Disk is the entity that represents a physical disk residing within a Exadata Storage Cell

Automatically discovered and activated

2008 Oracle Corporation Proprietary and Confidential

19

Exadata Storage Layout Example

Grid Disks

Grid Disk
Exadata Cell

Exadata Cell

Cell Disks are logically partitioned into Grid Disks

Grid Disk is the entity allocated to ASM as an ASM disk Minimum of one Grid Disk per Cell Disk Can be used to allocate hot, warm and cold regions of a Cell Disk or to separate databases sharing Exadata Cells

2008 Oracle Corporation Proprietary and Confidential

20

Exadata Storage Layout Example

ASM Disk Groups and Mirroring

Hot ASM Disk Group

Exadata Cell Exadata Cell Hot Cold Cold Cold Cold Hot Hot Hot Hot Cold

Cold ASM Disk Group

Hot Cold

Two ASM disk groups defined

One for the active, or hot portion, of the database and a second for the cold or inactive portion

ASM striping evenly distributes I/O across the disk group ASM mirroring is used protect against disk failures

Optional for one or both disk groups

21

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Layout Example

ASM Mirroring and Failure Groups

ASM Failure Group

Exadata Cell Exadata Cell Hot Cold Cold Cold Cold Hot Hot Hot Hot Cold

ASM Failure Group

Hot Cold

ASM Disk Group

ASM mirroring is used protect against disk failures ASM failure groups are used to protect against cell failures

2008 Oracle Corporation Proprietary and Confidential

22

Exadata Scale-Out Architecture

2009 Oracle Corporation Proprietary and Confidential

23

Scale-Out Architecture

Business Data Protection

ASM

The database, ASM, and Exadata Cells each play a role in Oracles scale-out storage architecture Responsibilities are placed in the optimal location
DB Business Data Protection ASM Reliable Storage Pool Exadata Cell Database Intelligent Storage

Reliable Storage Pool

Exadata

Database Intelligent Storage

Seamless integration simplifies management

2008 Oracle Corporation

24

Exadata Scale-Out Overview

Cells implement Oracles iDB data protocol

Grid Disks

iDB data protocol

Expose storage as collections of intelligent network disks called Grid Disks to ASM and the DB iDB is conceptually similar to iSCSI but has extensive DB intelligence iDB is layered on top of ZDP network protocol

ZDP net protocol

To ensure full scale-out benefits, cells never communicate with each other
Cross cell operations are implemented in ASM or DB Cell independence ensures no performance bottlenecks, and no cascading failures Key to scalability architecture
25

2008 Oracle Corporation

Exadata Cell Integration for Scale-Out

Cell appliance design eliminates storage configuration missteps and administrative overhead Simple provisioning

Cell grid disks are automatically made visible to ASM No OS level LUNs or mount points to setup and manage

Cross cell ASM mirroring is automatically configured for grid disks ASM failure group topology Multiple Grid disks per physical disk allow multiple ASMs to use the same cells

2008 Oracle Corporation

26

Automatic Storage Management

Integrated cluster volume

ASM Disk ASM Disk ASM Disk ASM Disk ASM Disk

manager Flexible data distribution (striping) Mirroring Automatic data re-balancing Free

ASM creates a reliable storage pool from Exadata Grid disks

27

2008 Oracle Corporation

ASM Scale-out Data Distribution

Files automatically spread across disks to balance load

ASM
DB File #2

DB File #1

ASM manages storage in megabyte allocation units Each DB file consists of a set of allocation units The location of a files allocation units are individually tracked by ASM ASM evenly spreads allocation units across all cells and disks in the grid All disks evenly utilized Optimal performance

2008 Oracle Corporation

28

ASM Data Re-Distribution

Cells can be easily added or removed from the storage grid

ASM
DB File #2

DB File #1

ASM migrates a fraction of the allocation units to the new cell Online and transparent to application Minimal data movement to new or removed cells

New Cell

Online Provisioning and Capacity Management

29

2008 Oracle Corporation

Protection from Hardware Failure

DB File #1

ASM
DB File #2

ASM implements mirroring at the allocation unit level Primary and mirror copy of allocation units are placed on separate storage cells Automatically remirrors across all remaining cells when a disk or array fails Failure of disk or array is transparent to database

Designed to Tolerate Failures of Cells and Disks

30

2008 Oracle Corporation

Protection from Brownout

ASM temporarily freezes I/O to unresponsive cells
Redirect read IOs to mirror copy Track write IOs that are missed

DB File #1
Fast mirror resync

ASM
DB File #2

Writes replayed when cell begins responding No need to remirror all the unchanged data Benefits

Freeze I/O to Cell

Fast recovery from transient failures e.g Cell crash or temporary hang Can be used for planned maintenance Cell software or component upgrade

Designed to Gracefully Tolerate Cell Brownouts

31

2008 Oracle Corporation

Data Protection Solutions

All single points of failure eliminated by the Exadata Storage architecture Hardware Assisted Resilient Data (HARD) built in to Exadata Storage

Prevent data corruption before it happens

Data Guard provides disaster protection and data corruption protection

Automatically maintained second copy of database

Flashback provides human error protection

Snapshot-like capabilities to rewind database to before error

Recovery Manager (RMAN) provide backup to disk

Archiving and corruption protection Can be used with Oracle Secure Backup (OSB) or third party tape backup software

These work just as they do for traditional non-Exadata storage

Users and database administrator use familiar tools

32

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Management & Administration

Enterprise Manager

Manage & administer Database and ASM

Exadata Storage Plug-in

Enterprise Manager Grid Control Plug-in to monitor & manage Exadata Storage Cells

Comprehensive CLI

Local Exadata Storage cell management Distributed shell utility to execute CLI across multiple cells

Integrated Lights Out Management - ILOM

Remote management and administration of hardware

2008 Oracle Corporation Proprietary and Confidential

33

 2008 Oracle Corporation Proprietary and Confidential

34