Oracle Exadata and OVM

Best Practice Overview

Sept 2016

Copyright 2016, Oracle and/or its affiliates. All rights reserved. |

Topics Covered

Use Cases
Exadata OVM Software Requirements
Exadata Isolation Considerations
Exadata OVM Sizing and Prerequisites
Exadata OVM Deployment Overview
Exadata OVM Administration and Operational Life Cycle
Migration, HA, Backup/Restore, Upgrading/Patching
Monitoring, Resource Management

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 2

 Exadata Virtual Machines
High-Performance Virtualized Database Platform

VMs provide CPU, memory, OS, and sysadmin isolation for

FINANCE consolidated workloads
No Additional Cost Hosting, cloud, cross department consolidation, test/dev,
non-database or third party applications
X2-2, X3-2, X4-2, X5-2, X6-2 SALES Exadata VMs deliver near raw hardware performance
DB 11.2 and 12c
I/Os go directly to high-speed InfiniBand bypassing hypervisor
DATABASE IN-MEMORY

Combine with Exadata network and I/O prioritization to

achieve unique full stack isolation
SUPPLY
CHAIN Trusted Partitions allow licensing by virtual machine

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 3

Exadata Consolidation Options
VMs have good Isolation but poor efficiency and
high management
Dedicated VMs have separate OS, memory, CPUs, and patching
DB Servers
Isolation without need to trust DBA, System Admin

VM VM VM
Database consolidation in a single OS is highly

More Efficient
Virtual efficient but less isolated

More Isolation
Machines
DB Resource manager isolation adds no overhead
Resources can be shared much more dynamically
Many DBs in But, must trust admins to configure systems correctly
one Server
Best strategy is to combine VMs with database
native consolidation
Database 12c Multiple trusted DBs or Pluggable DBs in a VM
Multitenant
Few VMs per server to limit overhead of fragmenting
CPUs/memory/patching etc.
Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 4
Software Architecture Comparison
Database Server: Bare Metal / Physical versus OVM

Bare Metal / OVM Database Server

Physical domU-3
Database Server dom0
Oracle GI/DB
domU-2homes
OracleExadata
GI/DB (Linux)
domU-1homes
Oracle GI/DB homes OracleExadata
GI/DB (Linux)
Exadata (Linux, homes
Exadata (Linux, fw) Xen, fw) Exadata (Linux)

No change to Storage Grid, Networking, or Other

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 5

Exadata VM Usage
VMs on Exadata focused on Database Consolidation with possibility of
consolidating some apps for convenience but Exadata features mostly benefit
database
Exadata VMs can only run certified Oracle Linux versions
Windows, RedHat, and other guest operating systems are not supported
Exadata VMs can also be used to virtualize other products, but they are only
recommended for light virtualization of other products
E.g. management tools, ETL tools, security tools etc.
Exadata VMs are not recommended for virtualization of heavyweight
applications e.g. E-business Suite or SAP application tier
Exalogic, Supercluster, or Private Cloud Appliance are recommended for virtualizing these
Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 6
Exadata OVM Requirements
Hardware Requirements
Exadata X6-2, X5-2, X4-2, X3-2, X2-2 are supported
Software Requirements
Exadata software 12.1.2.2.0 or higher
Supplied software (update with standard Exadata database server patching - see MOS 888828.1)
dom0: Oracle VM Server 3.2, Oracle Linux 5, UEK 2.6.39-400
domU: Oracle Linux 6, UEK 2.6.39-400
Grid Infrastructure / Database
Recommend 12.1.0.2 or 11.2.0.4 plus latest quarterly patch
Minimum 11.2.0.3 BP20

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 7

Exadata Isolation Considerations
Each VM RAC cluster has its own Exadata grid disks and ASM Disk Groups
Setting Up Oracle ASM-Scoped Security on Oracle Exadata Storage Servers
802.1Q VLAN Tagging for Client and Management Ethernet Networks
Dbnodes configured w/ OEDA during deployment (requires pre-deployment switch config)
Or configure manually post-deployment
Client network - MOS 2018550.1 Management network - MOS 2090345.1

InfiniBand Partitioning with PKEYs for Exadata Private Network

OS and InfiniBand switches configured w/ OEDA during deployment
Storage Server administration isolation through ExaCLI

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 8

Exadata OVM Sizing Recommendations
Use Reference Architecture Sizing Tool to determine CPUs, memory, disk
space needed by each database
Sizing evaluation should be done prior to deployment since OEDA will deploy your
desired VM configuration in an automated and simple manner.
Changes can be made post deployment, but requires many more steps
Sizing approach does not really change except for accommodating DOM0, and
additional system resources per VM
Sizing tool currently does not size virtual systems
Dom0 is automatically sized (4 vCPU, 8 GB RAM) and should not be
changed
Dom0 vCPU allocation is shared with VMs. Dom0 typically uses less than 4 vCPUs,
but can use up to 4 vCPUs depending on system workload
Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 9
Exadata OVM Sizing Recommendations
Memory
Do not over-provision physical memory - sum of VM memory used cannot exceed physical memory
Minimum 16 GB RAM per VM to support Oracle databases. This amount of memory required for
Linux, java, ASM, GI, etc.
Add additional memory for each database (DB binaries, PGA, SGA, process private memory)
VM memory can not be changed online, but can be changed by rebooting VM

CPU
Can over-provision CPU but workload performance conflicts can arise if all VMs become fully active
Minimum of 2 vCPUs* to allow for all database processes, backgrounds, and GI
vCPUs per VM must be multiple of 2
vCPUs assigned to a VM are dynamically changeable

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 10

Exadata OVM Local Disk Sizing Recommendations
Total local disk space available for VMs is 1.6TB on X4/X5/X6-2, 3.7TB with disk expansion kit
Disk space required per VM is 190 GB (sparse) for Linux+Oracle
120 GB is required to run one Oracle database version (approx 8 GB required per additional DB Version).
70 GB mount point for swap and root file systems
Additional application-specific disk space needs to be added, as necessary
VMs are copied from existing templates using OCFS2 sparse clones. This means much less physical disk
space is initially needed. As data is modified, it will consume more physical disk space.
Minimum physical disk space per VM is 120 GBs which will grow with database and application activity. Short
lived test/dev VMs can assume 100 GBs physical. Long lived VMs should assume no sparse allocation benefits.
DomU local space can be extended after initial deployment by adding local disk images
DomU space can be extended with shared storage (e.g. ACFS, DBFS, external NFS) for user & application
files. Shared storage should not be used for Oracle or Linux binaries and configuration files since shared
storage issues could cause entire system to crash or hang.

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 11

Option To Grow DB Server Local Disk Storage
2 socket database servers have 8 disk bays, only 4 are populated out of the
factory
Virtual Machines need more storage on the database servers
X6-2 and X5-2 database servers now support 8 x 600 GB HDDs
Only two supported configurations 4 drives or 8 drives
Servers will ship with only 4 drives out of the factory, customers can add 4 more hard
drives in the field
Minimum software version Exadata Storage Software 12.1.2.3.0

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 12

 Deployment Specifications and Limits
Hardware X2-2 X3-2 X4-2 X5-2 X6-2
Physical Memory 72 GB 256 GB 256 GB 256 GB 256 GB
/Node (default/max) 144 GB (max) 512 GB (max) 512 GB (max) 768 GB (max) 768 GB (max)
Min Mem per VM 16 GB min + additional DBs or app memory
Max Mem per VM 720 GB
Mem Defaults (GB) Small - 16; Medium - 32; Large 64 (Templates adjustable at install time)
Physical CPU 24 32 48 72 88
threads*/node
Min vCPUs* per VM 2
Max vCPUs per VM Physical Threads minus 4
Default Template Small - 2; Medium - 4; Large 8 (Templates adjustable at install time)
vCPUs
Total Available Disk 700 GB 700 GB 1.6 TB 1.6 TB (or 3.7 TB 1.6 TB (or 3.7 TB
Space for all DomU with DB Storage with DB Storage
Expansion Kit) Expansion Kit)
Minimum Used Disk 190 GB Sparse allocated per VM for initial deployment. 120 GB is required to run one Oracle database
Space per VM version (approx 8 GB required per additional DB Version).
70 GB mount point for swap and root file systems
*1 core = 1 OCPU = 2 threads = 2 vCPUs
Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 13
Deployment Overview
OEDA is the only tool that should be used to create VMs on Exadata
1. Create configuration with OEDA Configuration Tool
2. Prepare customer environment for OEDA deployment
Configure DNS, configure switches for VLANs (if necessary)
3. Prepare Exadata system for OEDA deployment
switch_to_ovm.sh; reclaimdisks.sh; applyElasticConfig.sh
4. Deploy system with OEDA Deployment Tool
Note: OS VLAN config can be done by OEDA or post deployment (MOS 2018550.1)

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 14

OEDA Configuration Tool
Advanced Network Configuration

Ethernet Network 802.1Q VLAN Tagging

For OVM define VM-specific VLAN IDs in
Cluster configuration pages later on
Ethernet switches (customer and Cisco)
must have VLAN tag configuration done
before OEDA deployment
InfiniBand Network Partitioning with
PKEYS

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 15

OEDA Configuration Tool
Identify Nodes

Screen to decide OVM or Physical

All OVM
All Physical
Some OVM, some physical

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 16

OEDA Configuration Tool
Define Clusters

Decide
Number of VM clusters to create
Dbnodes and Cells that will make
up those VM clusters
What is a VM cluster?
1 or more user domains on
different database servers running
Oracle GI/RAC, each accessing the
same shared Exadata storage
managed by ASM.

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 17

OEDA Configuration Tool
Cluster Configuration

Each VM cluster has its own

configuration page
Grid infrastructure installed in
each VM (grid disks owned by
a VM cluster) - Example
VM cluster 1
DATAC1 and RECOC1 across all cells
VM cluster 2
DATAC2 and RECOC2 across all cells

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 18

OEDA Configuration Tool
Cluster Advanced Network Configuration

Ethernet VLAN ID and IP details

To separate Ethernet traffic across VMs,
use distinct VLAN ID and IP info for each
cluster
InfiniBand PKEY and IP details
Compute Cluster network for dbnode-
to-dbnode RAC traffic. Separate IB
traffic by using distinct Cluster PKEYs
and IP info for each cluster.
Storage network for dbnode-to-cell or
cell-to-cell traffic - same for all clusters
Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 19
OEDA Configuration Tool
Review and Edit

This page lists all network details for

each VM guest in all VM clusters

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 20

OEDA Configuration Tool
Installation Template

Verify proper settings

for all VM clusters in
Installation Template so
the environment can
properly configured
before deployment
(DNS, switches, VLANs,
etc.).

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 21

OEDA Configuration Tool
Network Requirements
Component Domain Network Example hostname
dom0 Mgmt eth0 dm01dbadm01
(one per database server) Mgmt ILOM dm01dbadm01-ilom
Mgmt eth0 dm01dbadm01vm01
Database servers domU Client bondeth0 dm01client01vm01
(one or more per Client VIP dm01client01vm01-vip
database server) Client SCAN dm01vm01-scan
Private ib dm01dbadm01vm01-priv1
Mgmt eth0 dm01celadm01
Storage servers (same as physical) Mgmt ILOM dm01celadm01-ilom
Private ib dm01celadm01-priv1
Switches (same as physical)

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 22

Exadata OVM Basic Maintenance
Refer to Exadata Database Maintenance Guide: Managing Oracle VM
Domains on Oracle Exadata Database Machine
Show Running Domains, Monitoring, Startup, Shutdown
Modify Memory, CPU, local disk space in a user domain
Remove/Create RAC VM Cluster
Expand Oracle RAC VM cluster
Create User Domain without Grid Infrastructure (e.g. App VM)
Moving a User Domain to a Different Database Server
Deleting a User Domain from an Oracle RAC VM Cluster
Running exachk

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 23

Exadata OVM Basic Maintenance
Backing Up and Restoring Oracle Databases on Oracle VM User Domains
Expanding /EXAVMIMAGES on User Domains after Database Server Disk Expansion
Implementing Tagged VLAN Interfaces
Implementing InfiniBand Partitioning across OVM RAC Clusters on Oracle Exadata

Backing up the Management Domain (dom0) and User Domains (domU)

in an Oracle Virtual Server Deployment

Migrating a Bare Metal Oracle RAC Cluster to an OVM RAC Cluster

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 24

Exadata OVM Migration
Dynamic or online method to change physical to virtual
Data Guard or backups can be used to move databases minimum downtime
Convert one node or subset of nodes to virtual at a time
Migrating an existing physical Exadata rack to use virtual requires
Backing up existing databases, redeploying existing HW with OEDA and then
Restoring Databases
Duplicating the databases to existing Exadata OVM configuration
If moving from source to a new target, standard Exadata migration practices still
apply. Refer to Best Practices for Migrating to Exadata Database Machine

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 25

Exadata OVM Migration
Dynamic or online method to change physical to virtual using any of the
procedures below
Migrate to OVM RAC cluster using the existing bare metal Oracle RAC cluster with zero downtime
Migrate to OVM RAC cluster by creating a new OVM RAC cluster with minimal downtime
Migrate to OVM RAC cluster using Oracle Data Guard with minimal downtime
Migrate to OVM RAC cluster using RMAN backup and restore with complete downtime

For requirements and detailed steps, refer to My Oracle Support note

2099488.1.

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 26

Backup/Restore of Virtualized Environment
Dom0 Standard backup/restore practices to external
DomU Two Methods
Backup from Dom0: Snapshot the VM image and backup snapshot externally
Backup from DomU: Standard OS backup/restore practices apply
Database backups/restore: Standard Exadata MAA backup/restore
practices still apply with Exadata or with ZFS Storage
Refer to Exadata Maintenance Guide:
Backing up the Management Domain (dom0) and User Domains (domU) in an Oracle
Virtual Server Deployment
Recovering in an Oracle Virtual Server Deployment

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 27

Upgrade/Patching on OVM
Refer to Exadata Maintenance Guide
Execute exachk prerequisite checks
From one management domain (dom0)
From one user domain (domU) in each RAC VM cluster
Standard GI Upgrade per cluster still applies
Standard DB upgrade per database still applies
Standard Cell and InfiniBand patching still applies
For Database server updates,
Use patchmgr or dbnodeupdate.sh for each DomU and on Dom0
When patching Dom0, DomUs within the same db node will be unavailable.
Firmware/Bios is upgraded when Dom0 is upgraded

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 28

Health Checks and Monitoring
Exachk runs in Dom0 and DomU (cells and IB switches checks run with Dom0)
EM Monitoring support
EM Cloud Control requires Release 4 (12.1.0.4) or higher
Exadata Plug-in Version 12.1.0.6.0 or higher
Refer to MOS 1967701.1 for full details
Exawatcher support for Dom0 and DomU
Database/GI monitoring practices still apply
Dom0: XM top monitor all VMs
Currently Oracle VM Manager not supported on Exadata

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 29

Exadata MAA/HA
Exadata MAA failure/repair practices still applicable. Refer to MAA Best
Practices for Oracle Exadata Database Machine
OVM Live Migration is not supported use RAC to move workloads
between nodes

Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 30

Resource Management
Exadata Resource Management practices still apply
Exadata IO and flash resource management are all applicable and useful
Within VMs and within a cluster, database resource management practices
still apply
CPU count still needs to be set at the database instance level for multiple databases in
a VM. Recommended min = 2
No local disk resource management and prioritization.
IO intensive workloads should not use local disks
For higher IO performance and bandwidth, use ACFS or DBFS on Exadata or NFS.
All databases that share Exadata cells require unique DB_UNIQUE_NAME,
even across different clusters. Cloned test databases require new name.
Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 31
Copyright 2016, Oracle and/or its affiliates. All rights reserved. | 32