Exadata Migration

How to modernize your database platform to realize consolidation savings

Chris Bradham

About Applied Engineering Management

Applied Engineering Management (AEM) Corporation Founded in 1986 as a
100% woman-owned business
More than 25 years of profitable growth
Headquartered in Chantilly, VA with offices located in major metropolitan
areas including; Los Angeles, San Antonio, and Jacksonville
Diversified client base including Fortune 500 and major government
agencies
Industry recognized awards and certifications for performance, capability
and delivery

Chris Bradham
Oracle DBA experience 1997 to present (Oracle 7 to 11.2)
Replication, Materialized Views, Data Guard, GoldenGate, Exadata, RAC
Part-time Instructor George Mason University (OCA/OCP)
Oracle Certified Exadata Implementation Specialist, Oracle Certified
Professional (11g), Oracle Performance Tuning Certified(11g), ITIL
Foundation, Security+
Oracle Application Server, SAN Administration, Linux Administration,
VMWare, IBM / Sun Blade Administration
chris.bradham@aemcorp.com

Whats being covered?

Technology Refresh
Legacy Environment / Options
Exadata Components
Operational Considerations
Migration Considerations
Results of Migration
Lessons Learned
References
Q & A

Disclaimer

The US Navy, Commander Navy Installations Command (CNIC), nor

any related government organization in no way endorse Oracle
Corporation or the Oracle Exadata Database Machine. The opinions
expressed during the presentation are solely those of the presenter
and in no way reflects those of the government or any government
interest.

Background Information
Global multi-service DoD Web-based Housing application
Over 300 schemas
750 Gb of data

4,300 Active Users

4.2 million annual log ins

4,500 Reports Generated Per Day

AEM Corporation responsible for Hosting / Operations & Maintenance / Technology Refresh

Legacy Configuration

Physical Application Servers

4 Gbps Fibre Channel
Clarion Disk Array (10k / 15k RPM)
5 Node RAC Servers
Oracle 10gR2 RAC install

Pre-Tech Refresh Issues

Legacy hardware over six years old
Patches (5 nodes, slower machines)
Deployments, data updates time consuming
Large or complex reports often hang
Node evictions due to network / disk speed issues
Oracle 10.2.0.4 Support ended 6/31/11

Data Warehouse delay due to performance requirements

(Oracle Streams attempt)

Alternative 1 : Based on Legacy Solution

Virtualized application servers

Network bonding
8 Gbps Fibre Channel
EMC Disk Array
5 Node RAC Servers
Oracle 11gR2 RAC install

Alternative 2 : Based on Exadata Solution

Virtualized application servers
Network bonding
40 Gbps backbone
Oracle Storage Servers
2 node Quarter Rack
Oracle 11gR2 RAC preconfigured

Surprise, we chose Exadata!

Exadata Selection Points

Licensing fees made Exadata the low cost solution
Total database hardware solution
o2 nodes vs. 5 nodes
oNumber of Vendors
Patch Strategy
Sizable and expandable
Oracle vested to help DoD succeed
Storage Indexes / Smart Scan / Smart Flash Cache

Throughput Gb/Second
80.0
75

70.0
60.0

2 Gbps Fibre Channel x2

50.0

4 Gbps Fibre Channel x2

40.0
8 Gbps Fibre Channel x2
37

30.0

Exadata 1/4 - Disk

25

20.0
12.5

10.0
0.4
0.0

0.8

1.6

5.4

Exadata 1/2 - Disk

16

Exadata Full - Disk

Exadata 1/4 - Disk & Flash
Exadata 1/2 - Disk & Flash
Exadata Full - Disk & Flash

Exadata Planning/Considerations
Pre-configuration customer worksheet (DBM Configuration spreadsheet)
Initial Setup Activities
Network Configuration / Software Installation
Procurement considerations (2 Quarter Racks)
Space Choices
High Capacity (3 Tb / 7.2 K RPM)
High Performance ( 600 Gb / 15k RPM)
Space Allocation (ASM Disk Group layout)
78% DATA, 20% RECO, 2% DBFS_DG
o25 Tb
<- Production (Triple Mirror)
o8 Tb
<- Test (Mirror)

Tech Refresh Challenges

100% hardware replacement and Data Center move
Narrow window for outage to avoid negative impact on end users
Performance of system, database growth, and network bandwidth
Exadata unproven in DoD space at the time (Security)
Upgrading Database versions (data/code/reports)

Lots of change, what if issues surface???

Summary of Choices (X2)

X2-8 Full

X2-2 Full

X2-2 Half

X2-2 Quarter

$$$

$$$

$$

Database Grid Cores

160

96

48

24

Database Grid Memory (GB)

4096

768 (max 1152)

384 (max 576)

192 (max 288)

InfiniBand switches

Ethernet switch

KVM

No

Yes

Yes

Yes

Exadata Storage Servers

14

14

Storage Grid CPU Cores

168

168

84

36

5.3 TB

5.3 TB

2.6 TB

1.1 TB

High Perf

100 TB

100 TB

50 TB

21.6 TB

High Cap

504 TB

504 TB

252 TB

108 TB

High Perf

45 TB

45 TB

22.5 TB

9.5 TB

High Cap

224 TB

224 TB

112 TB

48 TB

High Perf

30 TB

30 TB

15 TB

6.5 TB

High Cap

150 TB

150 TB

75 TB

32 TB

Initial / Licensing Cost

Database Servers

Raw Flash Capacity

Raw Storage
Capacity
Usable mirrored
capacity
Usable Triple
mirrored capacity

Summary of Choices (X3)

X3-8 Full

X3-2 Full

X3-2 Half

X3-2 Quarter

X3-2 Eighth

$$$

$$$

$$

Database Grid Cores

160

128

64

32

16

Database Grid Memory (GB)

4096

1024 (max
2048)

InfiniBand switches

Ethernet switch

KVM

Yes

Yes

Yes

Yes

Yes

Exadata Storage Servers

14

14

Storage Grid CPU Cores

168

168

84

36

36

22.4 TB

22.4 TB

11.2 TB

4.8 TB

2.4TB

Initial / Licensing Cost

Database Servers

Raw Flash Capacity

512 (max 1024) 256 (max 512) 256 (max 512)

Raw Storage
Capacity

High Perf

100 TB

100 TB

50 TB

21.6 TB

10.8TB

High Cap

504 TB

504 TB

252 TB

108 TB

54 TB

Usable mirrored
capacity

High Perf

45 TB

45 TB

22.5 TB

9.5 TB

4.5 TB

High Cap

224 TB

224 TB

112 TB

48 TB

23 TB

Smart Flash Cache Considerations

Helps with
Write-Through cache voids caching data that will not be reused
Holds hot data, much faster than disk (small, random I/O)
Data not duplicated from cache in other Storage Servers
Reduce latency of log write by simultaneous write to flash / disk
(faster writes) with minimal space (512 Mb)
Write-Back cache 11.2.0.3.9
Dont touch except for
Alter table <table_name> flash_cache keep;
Create Flash Disks out of the Flash Cache
Reassign portion for TEMP tablespace on index builds

Database Node Considerations

Database Consolidation
SGA Settings
AMM Bad! ASMM Good! (set minimum values)
Huge Pages (reduce overhead)
Wheres the shared storage space?
DBFS is the answer (fix_control=8,ac_timeout=60 and SGA=2Gb)
Is everything setup correctly?
Exachk is the answer
Indexes / Hints / Compression
Large Segments <- 8 Mb Initial / Next Size with Autoallocate
TEMP <- BIGFILE, Autoextend 1 Gb, Uniform 1 Mb

Exadata Patch Management

Multiple Patches
Infiniband (once per year)
DB Nodes / Storage Server (quarterly)
Bundle Patch (BP) DB Software (quarterly)
Additional components (Ethernet switch, KVM, PDU)
Bug Fixes included so important to apply
One-off patches / opatch before and after patch
Rolling option time a consideration

Security
DoD 8570 Requirements
Security Technical Implementation Guide (STIG)
oOracle installation not customizable
oDBFS and idle_time dont play well together
oAutomatic Service Request (ASR) / Configuration Manager Limitation
oGrid Control / Third Party Certificates (September release)
oBanners / SQLNET.ORA settings impact on tools
Dont assume security settings will not have impact. Must TEST!!!

Migration Strategies
10.2.0.4 to 11.2.0.x Options Considered
DBFS with external tables (5 to 7 GB/sec file system I/O throughput)
GoldenGate with datapump (near-zero downtime)
Datapump
Factors
Maintenance window
Risk of data loss
Familiarity with technology
Whatever the choice, perform multiple trial runs for optimal settings.

2011 Technical Refresh (Data Center move)

On 9/9/11 at 7pm application
servers at legacy site were Turned
Off:
Transferred all data to Data Center
Reloaded data for all installations
Network outage occurred during
data transfer (2 hours)
On 9/10/11at 7am New System
Testing was Initiated
Users were on the system by 3pm
In Future-> X3 or Storage Expansion

Migration Timeline

1/4 Rack
Grid
Delivered
Migrate Control
(Test) Options Setup

1/11

2/11

Chris Initial DB
First Setup /
Load
Day

3/11

STIG
Test

Text

Setup
DBFS

Migration
Test

4/11

Apply BP
Test

5/11

Initial
DB
Oracle Setup / STIG Prod/
CAB
Apply BP
Load

6/11

Migration
Selection

7/11

1/4 Rack
Delivered
(Prod)

8/11

Cutover
to
Exadata

9/11

Migration
Grid
Test
Control
Setup

Post Tech Refresh Performance (in hours)

Event

Legacy

Exadata

BOR1 Process

4 hours

1 hour

BOR2 Process

4 minutes

23 seconds

Datamart IU

3 hours

35 minutes

MV refresh

37 minutes

13 minutes

Exadata Lessons Learned

Ensure hosting center can accommodate Exadatas dimensions (door height)
Staff requirements (more communication necessary)
Smart Scan <- direct path reads, table access full, fast full index scans,
parallel with parallel_degree_policy not auto
Chained rows / small tables void smart scans
Ideally 2 Exadata Database machines
EHCC 10x space and performance (DML)
Expect CPU utilization to decrease

In-Memory

X2-8

Massive Memory

2010

All I/Os to Memory

X3
2012

Exadata Lessons Learned (cont.)

Grid Control for monitoring / managing components
Platinum Support, major assistance
Exachk and opatch before / after patching
Standard tuning principles apply (OLTP)
Bring over OLTP database AS-IS
Time, Experience keys to stability

References
http://www.oracle.com/technetwork/database/options/olap/olap-exadata-x2-2performance-1429042.pdf
http://www.scribd.com/doc/57026846/Exadata-Smart-Flash-Cache-366203
http://www.oracle.com/technetwork/database/exadata/exadata-technical-whitepaper134575.pdf
http://www.oracle.com/technetwork/server-storage/engineeredsystems/exadata/exadata-storage-technical-overview-128045.pdf
http://www.oracle.com/technetwork/database/features/availability/exadata-consolidation522500.pdf
http://www.ioug.org/Portals/0/Documents/Anuj%20Mohan.pdf
http://nyoug.org/Presentations/2011/March/Niemiec_Keynote_Exadata.pdf
http://www.oracle.com/technetwork/articles/oem/exadata-commands-part1-402441.html
http://www.centroid.com/webinars/ExadataChangestheGame.pdf
http://www.centroid.com/knowledgebase/blog/using-dbfs-and-external-tables

References (cont.)
Database Machine and Exadata Storage Server (888828.1)
Oracle Exadata Database Machine exachk (1070954.1)
Oracle Exadata Best Practices (757552.1)
Best Practices for OLTP on the Sun Oracle Database Machine (1269706.1)
Best Practices for Data Warehousing on Database Machine (1297112.1)
Oracle Sun Database Machine Application Best Practices for Data Warehousing
(1094934.1)
Oracle Sun Database Machine Diagnosability and Troubleshooting Best Practices
(1274324.1)
Expert Oracle Exadata (Osborne, Johnson, Poder)

Questions???

Backup Slides

Exadata Rack Options

2 six Core Processors / 96 Gb RAM per DB node
2 six Core Processors / 24 Gb RAM per Storage Server
Dual ported 40 Gb/sec InfiniBand
Quarter Rack
2 DB nodes
2 Infiniband switches
Half Rack
4 DB nodes
3 Infiniband switches
Full Rack
8 DB nodes
3 Infiniband switches
Infiniband 10x faster than Fiber Channel

Cache Hierarchy (Full Rack X2-2)

Database DRAM

768 GB Raw Capacity

100 GB / Second

Flash Cache

5 TB Raw Capacity

50 GB / Second

Disk

100 300 TB Raw Capacity

21 GB / Second

Quarter Rack Selected

2 Xeon-based Dual-processor Database Servers (Sun Fire
X4170 M2)
24 cores (12 per server)
192 GB memory expandable to 288 GB (96 GB per
server expandable to 144 GB)
10 GigE connectivity to Data Center
4 x 10GbE ports (2 per server)
1.1 TB High Speed Flash
3 Exadata Storage Servers X2-2
All with High Performance 600GB disks
OR
All with High Capacity 3 TB disks
2 Sun Datacenter InfiniBand Switch
36-port Managed QDR (40Gb/s) switch
1 Admin Cisco Ethernet switch
Can Upgrade to a Half
Keyboard, Video, Mouse (KVM) hardware
or just add storage
Redundant Power Distributions Units (PDUs)

Rack

Storage Components and Expansion (X2)

-Local Storage (750 GB)
-Storage Servers (12 Disks per server)
Full

Half

Quarter

InfiniBand switches

Ethernet switch

Yes

Yes

Yes

Exadata Storage Servers

18

Storage Grid CPU Cores

216

108

48

6.75 TB

3.4 TB

1.5 TB

High Perf

128 TB

64 TB

28 TB

High Cap

648 TB

324 TB

144 TB

Usable mirrored
capacity

High Perf

58 TB

29 TB

13 TB

High Cap

288 TB

144 TB

64 TB

Usable Triple
mirrored capacity

High Perf

38 TB

19 TB

8.5TB

High Cap

192 TB

96 TB

43 TB

KVM

Raw Flash Capacity

Raw Storage
Capacity