Sie sind auf Seite 1von 34

<Insert Picture Here>

Oracle Exadata Storage Server Architecture Overview

Agenda
Exadata Software Architecture Exadata Storage Layout Exadata Storage Scale-Out Architecture

2008 Oracle Corporation Proprietary and Confidential

Exadata Software Architecture

2009 Oracle Corporation Proprietary and Confidential

Exadata Architecture
Single-Instance Database DB Server DB Instance DBRM ASM RAC Database DB Server DB Server DB Instance DBRM ASM DB Instance DBRM ASM
Enterprise Manager

InfiniBand Switch/Network

iDB Protocol over InfiniBand with Path Failover OEL

OEL
CELLSRV

OEL
CELLSRV

MS IORM RS Exadata Cell

MS IORM RS Exadata Cell

CELLSRV

MS IORM RS Exadata Cell

Cell Control CLI

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Server Overview


Exadata Storage Server

Data

Data

Data

Network Fabric

Evaluate Disk, CPU, and Network Throughput Standalone


5

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Server Overview


Exadata Storage Server

Database Server
RDBMS/ASM instance

Data

Data

Data

SGA

IO Client IO Layer ASM layer

?
Network Fabric

How do we connect the IO/ASM layer of ASM/DB to Exadata?


6

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Server Overview


Exadata Storage Server
Meta data Meta data Meta data

Database Server
RDBMS/ASM instance
SGA

Data

Data

Data

IO Client IO Layer cellsrv ASM layer libcell

Network Fabric

libcell linked with DB/ASM talks to cellsrv iDB protocol is born Multiple threads in cellsrv Threads perform asynchronous IO to disks and network 7

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Server Overview


Exadata Storage Server
Meta data Meta data Meta data

Database Server
RDBMS/ASM instance
SGA

Data

Data

Data

IO Client IO Layer cellsrv ASM layer libcell

Network Fabric

Smart Scan enabled in RDBMS and CELLSRV


2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Server Overview


Exadata Storage Server
Meta data Meta data Meta data

Database Server
RDBMS/ASM instance
SGA

Data

Data

Data

IO Layer cellsrv ASM layer libcell

/etc/oracle/cell/network-config

IO Client

Local IP
cellinit.ora

Network Fabric

cellinit.ora decides which network takes Storage traffic

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Server Overview


Exadata Storage Server
Meta data Meta data Meta data

Database Server
RDBMS/ASM instance
SGA

Data

Data

Data

IO Layer cellsrv ASM layer libcell

/etc/oracle/cell/network-config

IO Client

Cells
cellip.ora

Local IP
cellinit.ora

Network Fabric

cellip.ora on database/ASM host maintains list of cells New cells can be added to cellip.ora dynamically
2008 Oracle Corporation Proprietary and Confidential

10

Exadata Storage Server Overview


Exadata Storage Server
Meta data Meta data Meta data

Database Server
RDBMS/ASM instance
SGA

Data

Data

Data

dskm

IO Layer cellsrv ASM layer libcell

/etc/oracle/cell/network-config

IO Client

Cells
cellip.ora

diskmon

Local IP
cellinit.ora

Network Fabric

Master diskmon (diskmon) starts with CSS and communicates with cellsrv Slave diskmon (dskm) is part of every instance and communicates with master diskmon Handles cell failures, IO fencing, IO resource management plan propagation
11

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Server Overview


Exadata Storage Server
Meta data Meta data Meta data

Database Server
RDBMS/ASM instance
SGA

Data

Data

Data

dskm

cellcli

IO Layer ASM layer libcell

/etc/oracle/cell/network-config

IO Client cellsrv ms

Cells
cellip.ora

diskmon

Local IP
cellinit.ora

Network Fabric

Cellcli allows user interaction and configuration Management Server (MS) displays and manages creation and deletion of griddisks, changes in hardware, SNMP traps, alerts, email, metrics etc
2008 Oracle Corporation Proprietary and Confidential

12

Exadata Storage Server Overview


Exadata Storage Server
Meta data Meta data Meta data

Database Server
RDBMS/ASM instance
SGA

Data

Data

Data

dskm

cellcli

IO Layer ASM layer libcell

/etc/oracle/cell/network-config

IO Client cellsrv ms

Cells
cellip.ora

diskmon

Local IP
cellinit.ora

rs

Network Fabric

Restart Server (RS) monitors CELLSRV and MS, Backup RS monitors core RS RS monitors for process aliveness, memory usage etc

2008 Oracle Corporation Proprietary and Confidential

13

Exadata Storage Server Overview


Exadata Storage Server
Meta data Meta data Meta data

Database Server
RDBMS/ASM instance
SGA

Data

Data

Data

dskm

cellcli

IO Layer ASM layer

/etc/oracle/cell/network-config

IO Client cellsrv ms
adrci

Cells
cellip.ora

diskmon

libcell rs
ADR

Local IP
cellinit.ora

Network Fabric

Trace files and alert logs in Automatic Diagnostic Repository on cell alert.log (from RS and CELLSRV), ms-odl.log, ms-odl.trc, rs*trc, svtrc*.trc

2008 Oracle Corporation Proprietary and Confidential

14

Exadata Storage Server Overview


Exadata Storage Server
Meta data Meta data Meta data

Database Server
RDBMS/ASM instance
SGA

Data

Data

Data

dskm

cellcli

IO Layer ASM layer

cellsrv

ms
adrci

diskmon

/etc/oracle/cell/network-config

IO Client

Exadata cells
cellip.ora

libcell rs
ADR

Local IP
cellinit.ora

Infiniband Fabric

Infiniband fabric requires RPMs from OFED/OEL, RedHat 5.1, OEL 5.1 Exadata Storage Server works only with 11.1.0.7 Database/ASM

2008 Oracle Corporation Proprietary and Confidential

15

Exadata Storage Server Overview


Exadata Storage Server
Meta data Meta data Meta data

From the Enterprise

Enterprise Manager
Agent dcli
adrci

Data

Data

Data
OMS
cellcli

cellsrv

ms

rs

ADR

ssh/cellcli

Network Fabric

EM Plugin Provides central location for metrics and alerts across cells No agent runs on cell dcli allows user to run commands across cells
2008 Oracle Corporation Proprietary and Confidential

16

Exadata Storage Layout

2009 Oracle Corporation Proprietary and Confidential

17

Exadata Storage Layout


Physical disks map to a Cell Disks Cell Disks partitioned into one or multiple Grid Disks ASM diskgroups created from Grid Disks Transparent above the ASM layer

Grid Disk 1 Physical Disk Cell Disk


Sys Area

ASM disk

Grid Disk n
Sys Area

ASM disk

2008 Oracle Corporation Proprietary and Confidential

18

Exadata Storage Layout Example


Cell Disks

Cell Disk

Exadata Cell

Exadata Cell

Cell Disk is the entity that represents a physical disk residing within a Exadata Storage Cell
Automatically discovered and activated

2008 Oracle Corporation Proprietary and Confidential

19

Exadata Storage Layout Example


Grid Disks

Grid Disk

Exadata Cell

Exadata Cell

Cell Disks are logically partitioned into Grid Disks


Grid Disk is the entity allocated to ASM as an ASM disk Minimum of one Grid Disk per Cell Disk Can be used to allocate hot, warm and cold regions of a Cell Disk or to separate databases sharing Exadata Cells

2008 Oracle Corporation Proprietary and Confidential

20

Exadata Storage Layout Example


ASM Disk Groups and Mirroring

Hot ASM Disk Group


Hot Cold

Exadata Cell

Exadata Cell

Cold ASM Disk Group


Hot Cold

Hot Cold

Hot Cold

Hot Cold

Hot Cold

Two ASM disk groups defined


One for the active, or hot portion, of the database and a second for the cold or inactive portion

ASM striping evenly distributes I/O across the disk group ASM mirroring is used protect against disk failures
Optional for one or both disk groups
2008 Oracle Corporation Proprietary and Confidential

21

Exadata Storage Layout Example


ASM Mirroring and Failure Groups

ASM Failure Group


Hot Cold

Exadata Cell

Exadata Cell

ASM Failure Group


Hot Cold

Hot Cold

Hot Cold

Hot Cold

Hot Cold

ASM Disk Group

ASM mirroring is used protect against disk failures ASM failure groups are used to protect against cell failures

2008 Oracle Corporation Proprietary and Confidential

22

Exadata Scale-Out Architecture

2009 Oracle Corporation Proprietary and Confidential

23

Scale-Out Architecture
The database, ASM, and Exadata Cells each play a role in Oracles scale-out storage architecture Responsibilities are placed in the optimal location
DB Business Data Protection ASM Reliable Storage Pool Exadata Cell Database Intelligent Storage

Business Data Protection

ASM
Reliable Storage Pool

Exadata
Database Intelligent Storage

Seamless integration simplifies management

2008 Oracle Corporation

24

Exadata Scale-Out Overview


Cells implement Oracles iDB data protocol
Expose storage as collections of intelligent network disks called Grid Disks to ASM and the DB iDB is conceptually similar to iSCSI but has extensive DB intelligence iDB is layered on top of ZDP network protocol

Grid Disks
iDB data protocol ZDP net protocol

To ensure full scale-out benefits, cells never communicate with each other
Cross cell operations are implemented in ASM or DB Cell independence ensures no performance bottlenecks, and no cascading failures Key to scalability architecture

2008 Oracle Corporation

25

Exadata Cell Integration for Scale-Out


Cell appliance design eliminates storage configuration missteps and administrative overhead Simple provisioning
Cell grid disks are automatically made visible to ASM No OS level LUNs or mount points to setup and manage

Cross cell ASM mirroring is automatically configured for grid disks ASM failure group topology Multiple Grid disks per physical disk allow multiple ASMs to use the same cells

2008 Oracle Corporation

26

Automatic Storage Management


Integrated cluster volume
ASM Disk ASM Disk ASM Disk ASM Disk ASM Disk

manager Flexible data distribution (striping) Mirroring Automatic data re-balancing Free

ASM creates a reliable storage pool from Exadata Grid disks


2008 Oracle Corporation

27

ASM Scale-out Data Distribution


Files automatically spread across disks to balance load

ASM
DB File #1 DB File #2

ASM manages storage in megabyte allocation units Each DB file consists of a set of allocation units The location of a files allocation units are individually tracked by ASM ASM evenly spreads allocation units across all cells and disks in the grid All disks evenly utilized Optimal performance

2008 Oracle Corporation

28

ASM Data Re-Distribution


Cells can be easily added or removed from the storage grid
ASM migrates a fraction of the allocation units to the new cell Online and transparent to application Minimal data movement to new or removed cells

ASM
DB File #1

DB File #2

New Cell
Online Provisioning and Capacity Management
29

2008 Oracle Corporation

Protection from Hardware Failure


ASM implements mirroring at the allocation unit level Primary and mirror copy of allocation units are placed on separate storage cells Automatically remirrors across all remaining cells when a disk or array fails Failure of disk or array is transparent to database

DB File #1

ASM

DB File #2

Designed to Tolerate Failures of Cells and Disks


2008 Oracle Corporation

30

Protection from Brownout


ASM temporarily freezes I/O to unresponsive cells

DB File #1

ASM

Redirect read IOs to mirror copy Track write IOs that are missed

DB File #2

Writes replayed when cell begins responding


Fast mirror resync

No need to remirror all the unchanged data Benefits Freeze I/O to Cell
Fast recovery from transient failures e.g Cell crash or temporary hang Can be used for planned maintenance Cell software or component upgrade

Designed to Gracefully Tolerate Cell Brownouts


2008 Oracle Corporation

31

Data Protection Solutions


All single points of failure eliminated by the Exadata Storage architecture Hardware Assisted Resilient Data (HARD) built in to Exadata Storage
Prevent data corruption before it happens

Data Guard provides disaster protection and data corruption protection


Automatically maintained second copy of database Snapshot-like capabilities to rewind database to before error Archiving and corruption protection Can be used with Oracle Secure Backup (OSB) or third party tape backup software Users and database administrator use familiar tools
32

Flashback provides human error protection

Recovery Manager (RMAN) provide backup to disk

These work just as they do for traditional non-Exadata storage

2008 Oracle Corporation Proprietary and Confidential

Exadata Storage Management & Administration


Enterprise Manager
Manage & administer Database and ASM

Exadata Storage Plug-in


Enterprise Manager Grid Control Plug-in to monitor & manage Exadata Storage Cells

Comprehensive CLI
Local Exadata Storage cell management Distributed shell utility to execute CLI across multiple cells

Integrated Lights Out Management - ILOM


Remote management and administration of hardware
33

2008 Oracle Corporation Proprietary and Confidential

2008 Oracle Corporation Proprietary and Confidential

34