Exadata

Presented by: Kerry Osborne

February 23, 2012

whoami

Worked with Oracle Since 1982 (V2) Working with Exadata since early 2010 Work for Enkitec (www.enkitec.com) (Enkitec owns a Half Rack V2/X2) Many Exadata customers and POCs Many Exadata Presentations (many to Oracle) Exadata Book

Blog: kerryosborne.oracle-guy.com

Whats the Point?

Can we get near Exadata performance without buying an Exadata?
! Commodity Hardware ! Published Specs ! Specs are Easily Reproduced (or Exceeded) ! So the Question Comes Up Frequently

Note: This presentation was originally proposed as a session for OpenWorld 2010 by Kevin Closson.
4

Poll Can You Get Exadata Like Performance w/o Buying an Exadata?
Yes No I think I can build a better mousetrap (for less money) It absolutely cannot be done

Maybe I think I might be able to get pretty close

What is Exadata?
Exadata Database Servers

Exadata Storage Servers

11gR2 / ASM

iDB / RDS cellsrv

*Half Rack
6

Whats the Plan?

Important Architectural Features
! Flash Cache ! Pipe (Infiniband) ! Compute Resources (CPUs) ! Total Storage ! Redundancy (RAC?) ! Manageability (Dial Home, ILOM, etc) ! remember there are tradeoffs

Filling the Buckets*

Compute Capacity (cores) 132 (48+84) Storage (TB Usable) ~ 80 (252 raw) Flash Cache (TB) ~ 2.6 Pipe (Gb/s) 40 Redundancy ? Manageability ? Cost ?

*Half Rack
8

First Iteration
! 4 - Sun Fire X4170 M2 Servers ! 16 8G HBAs and a Switch ------------ (need to step up to VMAX) ! EMC VNX7500 ! 10 200G Fast Cache ! 84 - 7.2K, 3T Drives

Copy Exadata Specs

But Were Already Off in the Weeds!

And were not accounting for additional CPUs on storage tier.
9

Storyville
Imagine a system that spends 4.5 hours every night doing a batch update of a Billion+ row table one row at a time. Which buckets are most important?

! Storage? ! Capacity ! Throughput ! Latency ! Pipe? ! CPU? ! Memory?

10

High Performance: Large Scale Retail Comparison

11

High Performance: Large Scale Retail Overview

12

Customer Decided to Pursue DIY Route -

13

Second Iteration
! 1 Dell R910 (2x8 CPU, 256G RAM, PCIe) ! 70G buffer cache ! 4 8G HBAs and switch(es) ! Hitachi SAN (AMS 2500) ! 2 trays of 200G SSD (30) ! 1 tray of rusty spiny things (38x450G) No redundancy Not enough storage Did get write back cache Long Running Queries still take a while But Not Bad!

SSD & Big SGA

14

DIY Results:
Batch Job: ~ 50 minutes 56 Min Query: ~ 15 minutes Costs: hardware roughly the same as half rack Oracle software quite a bit less

15

Third Iteration

Lots O CPU & A Big Pipe

! 2 - Sun Fire X4800 Servers (plenty of CPU, memory, PCIe) ! 4 QDR InfiniBand HCAs and switch(es) ! Sun ZFS 7320 ! 2TB Read Cache ! 4 trays of 20 X 7.2K, 3T Drives ! per James request we can use RDMA ! also supports HCC

Getting close, but whats it going to cost?

16

Digression: Got Balance?

! DB Grid must generate I/O requests ! Generating I/O requests require CPU ! Storage must be able to deliver the I/O ! Need enough devices, etc ! Transport mechanism must be adequate ! DB Grid must ingest the I/O ! Consuming I/O requires CPU

Basic idea is that we must be able to consume what is produced.

17

Hardware Conclusion
! Exadata Architecture Provides a Roadmap ! Flash Based Storage ! Big Pipes (Infiniband) ! Low Latency (RDMA) ! RAC Provides Ability to Scale Out ! Unlikely that you can build it for anywhere near the cost ! But you can probably build something adequate for specific WLs

18

Hardware is only half the story:

Remember: ! CPUs on Storage Cells Can Be Used For DB Processing ! So We Need More CPU on DB Servers To Compensate ! And the associated DB/RAC licensing costs ! We May Also Need More DB Server Memory ! All Because of the Storage Software
19

The Big Ah Ha!

The Bottleneck on Many (Most) Large Databases is between the Disk and the DB Server(s)! How to Speed Up? Make the Pipe Bigger/Faster Reduce the Volume

* The fast way to do anything is not to do it!

20

Offloading The Secret Sauce

Offloading vs. Smart Scan (whats the difference) Offloading generic term meaning doing work at the storage layer instead of at the database layer Smart Scan query optimizations covered by cell smart table/index scan wait events

21

Smart Scan Optimizations

Column Projection Predicate Filtering Storage Indexes Simple Joins Function Offloading Virtual Column Evaluation HCC Decompression Decryption

22

Demo Time

23

Exadata Software Performance

24

High Transaction Volume: Telco Provider

! Customer Runs Dell, 16 Core Machines in Multiple RAC Instances ! Very High Volume of OLTP and Data Warehouse Type Queries on Same Database ! Performance Differences Were Too Excessive to Graph
SQL
Process 1: 6-Month Data Volume Process 2: 3-Month Data Volume Process 3: 1-Year Data Volume Process 4: 2-Month Data Volume Update SCN_CALL_PARTY_LOG Update SCN_CALL_PARTY_IDENT_LOG Select SCN_CALL_PARTY_EXTDATA_LOG

Current 52 min 51 min 50 min 48 min 13 min 7 min 6.75 min

Exadata 19.5 sec 11.5 sec 37.5 sec 9.4 sec 1.05 sec .23 sec .47 sec

Times Faster 160 x 269 x 81 x 308 x 744 x 1871 x 868 x

25

The Kbler-Ross grief cycle

Exposure to Exadata
26

Questions?
Contact Information : Kerry Osborne
kerry.osborne@enkitec.com kerryosborne.oracle-guy.com www.enkitec.com

Fastest Growing Companies in Dallas

27