<Insert Picture Here>

Exadata Storage Configuration

Agenda
Exadata Server First Boot Exadata Storage Allocation ASM & Database Configuration Database & Application Changes

2008 Oracle Corporation Proprietary and Confidential

-2-

Exadata First Server Boot

2009 Oracle Corporation Proprietary and Confidential

-3-

Exadata Server Initialization

Initial Network Preparation Lights-Out 100 (LO100c) Network Configuration First Boot Sequence Performance Sanity Test Create Cell

2008 Oracle Corporation Proprietary and Confidential

-4-

Initial Network Preparation

3 IP Addresses Per Exadata Server
1 for Bonded Dual-port InfiniBand HCA 1 for Embedded Ethernet Port 1 for LO100c Remote Management Port

2008 Oracle Corporation Proprietary and Confidential

-5-

Exadata Server First Boot

1. Power on Exadata Server 2. Configure Remote Management (LO100c). 3. Answer First Boot Questions:
Fully Qualified Hostname Domain Name Service (DNS) Servers Network Time Protocol (NTP) Servers Timezone IP Addresses (Net masks, Gateway) for eth0 and bond0

4. Change default passwords

Root, celladmin, cellmonitor

2008 Oracle Corporation Proprietary and Confidential

-6-

Lights Out (LO100c) Setup

2008 Oracle Corporation Proprietary and Confidential

-7-

First Boot Sequence

Fully qualified hostname (Example: host.company.com): myhost.company.com Nameserver: mynameserv.company.com Add more nameservers (y/n) [n]:n Select country by number, [n]ext, [l]ast:228 Select zone by number, [n]ext: 20 Selected timezone: America/Los_Angeles Is this correct (y/n) [y]: y The current ntp server(s): 0.us.pool.ntp.org Do you want to change it (y/n) [n]: y Primary (mandatory) ntp server: ntp1.company.com Secondary (optional, but recommended) ntp server: ntp2.company.com Tertiary ntp server: You selected: <list NTP servers with modified information> Is it correct (y/n) [y]: y <list of all available network interfaces> Select interface name to configure or press Enter to continue: eth0 Selected interface. Eth0 IP address: 10 Netmask: 255.255.252.0 Gateway (IP address or space for none): 10 Continue configuring or re-configuring interfaces? (y/n) [y]: y <list of all available network interfaces> Select interface name to configure or press enter to continue: bond0 Selected interface. bond0 IP address: 192.168.50.76 Netmask: 255.255.255.0 Continue configuring or re-configuring interfaces? (y/n) [y]: n <list of all available network interfaces with modified information> Is this correct? (y/n): y

2008 Oracle Corporation Proprietary and Confidential

-8-

Performance Sanity Test

Simple Test to verify performance of
Individual Disks Disk Controller

calibrate CellCLI command

Per Disk Read Throughput Per Disk IOPs Disk Controller Read Throughput Pass / Fail

2008 Oracle Corporation Proprietary and Confidential

-9-

Calibrate CellCLI command

[root@cell01 ~]# cellcli CellCLI: Release 11.1.3.0.0 - Production on Tue Oct 04 22:13:21 PDT 2008 Copyright (c) 2007, 2008, Oracle. Cell Efficiency ratio: 1.0 All rights reserved.

CellCLI> calibrate Aggregate random read throughput across all luns: 991.86 MBPS Aggregate random read IOs per second (IOPS) across all luns: 3926 Controller read throughput: 1057.41 MBPS Lun 6_1 on drive [1I:6_1:1 ] random read throughput: 98.81 MBPS, Lun 6_2 on drive [1I:6_1:2 ] random read throughput: 99.83 MBPS, Lun 6_3 on drive [1I:6_1:3 ] random read throughput: 97.46 MBPS, Lun 6_4 on drive [1I:6_1:4 ] random read throughput: 97.17 MBPS, Lun 6_5 on drive [1I:6_1:5 ] random read throughput: 101.84 MBPS, Lun 6_6 on drive [1I:6_1:6 ] random read throughput: 93.90 MBPS, Lun 6_7 on drive [1I:6_1:7 ] random read throughput: 99.90 MBPS, Lun 6_8 on drive [1I:6_1:8 ] random read throughput: 96.78 MBPS, Lun 6_9 on drive [1I:6_1:9 ] random read throughput: 101.09 MBPS, Lun 6_10 on drive [1I:6_1:10] random read throughput: 97.38 MBPS, Lun 6_11 on drive [1I:6_1:11] random read throughput: 99.32 MBPS, Lun 6_12 on drive [1I:6_1:12] random read throughput: 94.69 MBPS, CALIBRATE results are within an acceptable range.

and and and and and and and and and and and and

334 335 335 335 338 332 337 337 333 339 336 339

IOPS IOPS IOPS IOPS IOPS IOPS IOPS IOPS IOPS IOPS IOPS IOPS

2008 Oracle Corporation Proprietary and Confidential

- 10 -

Creating a New Exadata Cell

[celladmin@cell01 ~]$ cellcli CellCLI: Release 11.1.3.0.0 - Production on Tue Oct 04 22:13:21 PDT 2008 Copyright (c) 2007, 2008, Oracle. Cell Efficiency ratio: 1.0 All rights reserved.

CellCLI> CREATE CELL cell01 realmName=my_realm, interconnect1=bond0, smtpServer='my_mail.example.com', smtpFromAddr='exadata.cell01@example.com', smtpPwd=<exadata cell01 email password>, smtpToAddr='jf.v@example.com', notificationPolicy='critical,warning,clear', notificationMethod='mail' Cell cell01 successfully created Starting CELLSRV services The STARTUP of CELLSRV services was successful CellCLI>

2008 Oracle Corporation Proprietary and Confidential

- 11 -

Exadata Storage Allocation

2009 Oracle Corporation Proprietary and Confidential

- 12 -

Exadata Storage Allocation

LUN

Celldisk

Griddisk

ASM disk

Grid Disk 1 Physical Disk or LUN Cell Disk

System Area

Grid Disk n
System Area

2008 Oracle Corporation Proprietary and Confidential

- 13 -

Exadata System Area

First Two Disks 30 GB System Space each
Software RAID 1 Exadata Software, OS, space for trace files 700MB iSCSI Metadata Disk for OCR, Voting Disks Celldisk / Griddisk Metadata

Remaining Ten Disks < 100MB Celldisk / Griddisk Metadata

2008 Oracle Corporation Proprietary and Confidential

- 14 -

Create Celldisks
CREATE CELLDISK ALL to create all celldisks
- Default name is CD<id>_<cellname> - Idempotent operation skips any existing celldisks

Can also create single celldisk at a time

2008 Oracle Corporation Proprietary and Confidential

- 15 -

Creating Cell Disks

CellCLI> CellDisk CellDisk CellDisk CellDisk CellDisk CellDisk ... CellDisk CREATE CELLDISK ALL CD_1_cell01 successfully created CD_10_cell01 successfully created CD_11_cell01 successfully created CD_12_cell01 successfully created CD_2_cell01 successfully created CD_3_cell01 successfully created CD_9_cell01 successfully created

CellCLI> LIST CELLDISK CD_10_cell01 CD_11_cell01 CD_12_cell01 CD_1_cell01 CD_2_cell01 ... CellCLI>

normal normal normal normal normal

2008 Oracle Corporation Proprietary and Confidential

- 16 -

Create Griddisks
Divide each celldisk into multiple griddisks Griddisk = ASM disk CREATE GRIDDISK ALL to create multiple griddisks
Creates one griddisk per celldisk Default Name is <prefix>_<celldiskname> Size of each griddisk Skips existing griddisks with same name

Size is optional will use all remaining space Order of griddisk creation (hottest coldest)

2008 Oracle Corporation Proprietary and Confidential

- 17 -

Creating Grid Disks

CellCLI> CREATE GRIDDISK ALL PREFIX=data, size=300G GridDisk data_CD_1_cell01 successfully created ... GridDisk data_CD_12_cell01 successfully created CellCLI> CREATE GRIDDISK ALL PREFIX=recov GridDisk recov_CD_1_cell01 successfully created ... GridDisk recov_CD_12_cell01 successfully created

CellCLI> LIST GRIDDISK data_CD_1_cell01 ... data_CD_12_cell01 recov_CD_1_cell01 ... recov_CD_12_cell01 CellCLI> exit [celladmin@cell01 ~]$

active active active active

2008 Oracle Corporation Proprietary and Confidential

- 18 -

Exadata CellCLI Basic Commands

Basic subset of CellCLI commands
ALTER CELL [STARTUP | SHUTDOWN | RESTART ] SERVICES [ALL | MS | RS | CELLSRV] CREATE CELLDISK ALL CREATE GRIDDISK ALL PREFIX=<prefix>, SIZE=<size> LIST [ LUN | CELLDISK | GRIDDISK ] [<name>] [DETAIL] DROP [CELLDISK | GRIDDISK] ALL

Detailed Description in the User Guide

2008 Oracle Corporation Proprietary and Confidential

- 19 -

ASM and Database Configuration

2009 Oracle Corporation Proprietary and Confidential

- 20 -

ASM and Database Configuration

Both Database and ASM installation software must be 11.1.0.7 RAC - Install OCR and voting disks on Exadata storage ASM instance initialization parameter:
ASM_DISKSTRING = 'o/*/data*'

Database instances initialization parameter:

COMPATIBLE='11.1.0.7'

2008 Oracle Corporation Proprietary and Confidential

- 21 -

ASM and Database Configuration

Create two new files at each ASM & Database node in
/etc/oracle/cell/network-config

cellinit.ora
IP addresses of the Local InfiniBand interfaces

cellip.ora
Universe of remote Exadata Servers that can be accessed IP addresses of remote Exadata Servers InfiniBand interfaces Needs to be identical on all nodes of the ASM cluster

2008 Oracle Corporation Proprietary and Confidential

- 22 -

ASM & Database Node Config Files

# mkdir -p /etc/oracle/cell/network-config # chown oracle:dba /etc/oracle/cell/network-config # chmod ug+wx /etc/oracle/cell/network-config
$ cd /etc/oracle/cell/network-config

$ cat /etc/oracle/cell/network-config/cellinit.ora ipaddress1=192.168.50.23/24

$ cat /etc/oracle/cell/network-config/cellip.ora cell="192.168.50.27:5042" cell="192.168.50.28:5042" cell="192.168.50.29:5042"

2008 Oracle Corporation Proprietary and Confidential

- 23 -

RAC OCR and Voting Disks

Each Exadata Server exports an iSCSI Metadata Disk 700MB RAID1 Shared storage for OCR and Voting Disks Pick the same 3 Exadata Servers on all hosts HP Oracle Database Machine Run /opt/oracle/cellos/db.iscsi.sh on each DB node
Specify local host name IP address of InfiniBand Interface of 3 Exadata Servers Create OCR and Voting Disk partitions on iSCSI disk

Other Database Servers Configure iSCSI initiator manually

2008 Oracle Corporation Proprietary and Confidential

- 24 -

iSCSI Initiator Script

[root@dscbac01 ~]# ./db.iscsi.sh
Are you setting this node as member of Oracle RAC (y/n) [n]: y Configuring iSCSI drives for OCR and voting disks Client name of iSCSI initiator (hostname of this Oracle Database server): dscbac01 Owner of ocrvot devices (Oracle Database owner usually user oracle) [oracle]: Group of ocrvot devices (Oracle Database owner group, usually dba) [dba]: Permissions of ocrvot devices (default 640) [0640]: IP address of the Infiniband interface bond0 of the cell providing ocrvota: 192.168.50.209 IP address of the Infiniband interface bond0 of the cell providing ocrvotb: 192.168.50.210 IP address of the Infiniband interface bond0 of the cell providing ocrvotc: 192.168.50.211 Client name of iSCSI initiator (hostname of this Oracle Database server): dscbac01 IP address of the Infiniband interface bond0 of the cell providing ocrvota: 192.168.50.209 IP address of the Infiniband interface bond0 of the cell providing ocrvotb: 192.168.50.210 IP address of the Infiniband interface bond0 of the cell providing ocrvotc: 192.168.50.211 Is this correct (y/n) [y]:

Waiting 5 seconds for iSCSI to come up ... Add ip-192.168.50.209:3260-iscsi-iqn.2001-04.com.dscbac01s.us.oracle.com-disk0 to the persistent storage Add ip-192.168.50.210:3260-iscsi-iqn.2001-04.com.dscbac02s.us.oracle.com-disk0 to the persistent storage Add ip-192.168.50.211:3260-iscsi-iqn.2001-04.com.dscbac03s.us.oracle.com-disk0 to the persistent storage Starting udev: [ OK ]

2008 Oracle Corporation Proprietary and Confidential

- 25 -

ASM Discovery and Diskgroups

Universe of Exadata Servers specified in cellip.ora ASM discovery string to subset Exadata Servers
Can specify NULL discovery string (default) format is o/<exadata_connect_info>/<griddisk_name> <exadata_connect_info> format in cellip.ora

Failure Groups Created by Default

One failure group per Exadata Server

New parameters for create diskgroup

cell.smart_scan_capable=TRUE compatible.rdbms=11.1.0.7.0 compatible.asm=11.1.0.7.0 au_size = 4M (recommended)
2008 Oracle Corporation Proprietary and Confidential

- 26 -

Exadata Only ASM Diskgroup

CREATE DISKGROUP data NORMAL REDUNDANCY 'o/*/data*' ATTRIBUTE 'compatible.rdbms' = '11.1.0.7.0', 'compatible.asm' = '11.1.0.7.0', 'cell.smart_scan_capable' = 'TRUE', 'au_size' = '4M';
Failure group CELL01
o/<Cell 1 IP address>/data_cd_1_cell01 o/<Cell 1 IP address>/data_cd_2_cell01 o/<Cell 1 IP address>/data_cd_12_cell01

Failure group CELL02

o/<Cell 2 IP address>/data_cd_1_cell02 o/<Cell 2 IP address>/data_cd_2_cell02

o/<Cell 2 IP address>/data_cd_12_cell02

2008 Oracle Corporation Proprietary and Confidential

- 27 -

Database and Application Changes

2009 Oracle Corporation Proprietary and Confidential

- 28 -

Database / Application Changes

Requires ASM to manage Exadata storage Transparent to SQL, users, applications SQL Offload processing only
Direct path reads (default for PQ) Tablespace in Exadata only diskgroup

2008 Oracle Corporation Proprietary and Confidential

- 29 -

Exadata Server Configuration Summary

Exadata Server First Boot
Setup LO100c First Boot Sequence

Exadata Storage Allocation

Create celldisks Subdivide celldisks into griddisks

ASM & Database Configuration

Optional OCR & Voting Disks on Exadata Storage New configuration files (cellinit.ora, cellip.ora) New diskgroup parameters

Database & Application Changes

No changes needed to use Exadata storage

2008 Oracle Corporation Proprietary and Confidential

- 30 -

 2008 Oracle Corporation Proprietary and Confidential

- 31 -