Oracle Cloud Reference Design-08!28!2015

Call 1 (415) 252-9164
Email:
sales@mokumsolutions.com
Web:
http://mokumsolutions.com
About the Author
The author of the Oracle Cloud Cookbook is none other than the owner of
Mokum Solutions, Inc., Roddy Rodstein. Roddy is one of the most respected
Oracle Cloud Computing experts, having designed and managed many of the
worlds largest and most complex Oracle clouds. Before establishing Mokum,
Roddy spent three years at Oracle on the Oracle VM and Linux team
designing and supporting Oracle's largest Oracle VM environments. Before
Oracle, Roddy spent six years at Citrix, designing and supporting Citrix's
largest and most complex customer environments, Including Oracle's.
About Mokum Solutions, Inc.
Founded in March 2011, Mokum Solutions, Inc. specializes in virtualizing
Oracle workloads in private and hybrid clouds. Mokum corporate
headquarters are located in San Francisco, CA
http://mokumsolutions.com
or call 1 415 252 9164.
Why Mokum?
Mokum has deployed hundreds of successful Oracle VM environments.
Mokum can save you months, and years of trial and error, and deliver an
enterprise-ready Oracle VM private cloud managed by Oracle Enterprise
Manager 12c or OpenStack, without having to overprovision hardware or
Oracle licenses. All of Mokums customers are Oracle VM success stories that
have saved MILLIONS in Oracle licensing and hardware costs.
Unlike Mokum, Oracle consulting and traditional Oracle partners have no
incentive for customers to succeed with Oracle VM, since Oracle VM is used
to reduce Oracle licensing costs, and eliminates the need to buy $$$$ Oracle
Engineered Systems. Our customers that have engaged with Oracle and
traditional Oracle partners for Oracle VM projects received junior engineers
with little or no Oracle VM experience that regularly reference our books and
blogs for their deployments. All of Mokums customers are Oracle VM
success stories that have saved MILLIONS in Oracle licensing and hardware
costs.
Copyright 2015 Mokum Solutions, Inc. All rights reserved.
Distribution of the Oracle Cloud Cookbook or derivative of the work in any
form is prohibited unless prior permission is obtained from the Copyright
holder.
Call 415-252-9164
Mokum Oracle Cloud Cookbook
http://mokumsolutions.com/
Document Created: 09/30/13

Last Update: 08/28/2015
This chapter of the Oracle Cloud Cookbook presents Mokum's Oracle private cloud reference design. The Mokum
Oracle private cloud reference designs encompass the software, hardware, storage, network, and
management components required to deploy a scalable, secure, and supportable Oracle private cloud.
Table of Contents
Mokum Oracle Private Cloud Reference Design Introduction
Mokum Oracle Private Cloud Reference Design Implementation Overview
Mokum Oracle Private Cloud Reference Design Support Infrastructure
Oracle Private Cloud Architectural Design
Oracle VM Hardware Architecture
..Oracle VM Hardware Sizing
Oracle VM Server Pool Design
Oracle VM Security Standards
Oracle VM Manager Security Controls
Oracle VM Server Security Controls
Virtual Machine Operating System Standards
Oracle VM Disaster Recovery
Oracle VM Application Integration
Call 415-252-9164
Mokum Solutions, Inc.
Call 415-252-9164
Mokum Oracle Private Cloud Reference Design Introduction

The Mokum Oracle private cloud reference design is a field-tested best-practice standard, designed with simplicity,
reproducibility, usability, scalability, supportability and security. The Mokum Oracle private cloud reference
designs represent a complete Oracle Private Cloud standard that can be leveraged as a vanilla solution or modified
to more accurately reflect organization-specific needs. The Mokum Oracle private cloud reference design includes
the following categories:
Cloud Delivery Model
Infrastructure
Management
Virtual Machines
Infrastructure as a Service (IaaS)
Oracle VM for x86

x86 64 Server Hardware
Oracle Enterprise Manager
Storage & Network
Note: A detailed explanation of each category and solution in the Mokum Oracle private cloud reference design is
presented in the architectural overview section.
Mokum Oracle Private Cloud Reference Design Implementation Overview

The Mokum Oracle private cloud reference design provides a well defined starting point for each Oracle private
cloud implementation. It also serves as a baseline upon which all solution additions, revisions, and tools will be
based. As such, there is an increasing value to Mokum Oracle private cloud reference design in keeping
implementations as close to the reference design as possible.
Prior to implementing an Oracle private cloud, its important that an infrastructure assessment (IA) and gap
analysis (GA) be performed. During the IA/GA, the architecture of the solution will match the customers business
needs while maintaining the integrity of the Mokum Oracle private cloud reference design. Implementation and
support will follow the analysis phase after careful consideration has been given to any specific design
modifications that deviate from the Mokum Oracle private cloud reference design.
This document outlines the decision points necessary for implementing the Mokum Oracle private cloud reference
design. For decisions that rely on preexisting factors or specific organizational needs, the appropriate best practice
will be discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be
analyzed carefully and decisions should be made based on organizational needs, existing architecture, and budget
resource availability.
The Mokum Oracle private cloud reference design is designed to be scalable and resilient for ease of
implementation, high availability, and ease of maintenance for private Oracle clouds. The complete solution is
made up of multiplearchitectural components that work together to provide flexibility and options for selfserviceInfrastructure as a Service withbroad network access, resource pooling, elasticity, measured service, high
availability, security and ease of maintenance.Infrastructure as a Service is the capability to provision and deliver
fundamental computing resources as a service to the consumer (consumer = end users). The Mokum Oracle private
Call 415-252-9164
Call 415-252-9164
cloud reference design outlines the decision points necessary for implementing an Oracle VM private cloud to
deliver self-serviceInfrastructure as a Service using pre-configuredvirtual machine templates from theOracle
Enterprise Manager Cloud Control 12cself service portal, or OpenStack.
Mokum Oracle Private Cloud Reference Design Support Infrastructure

Support is an integral part of any Oracle private cloud and includes a combination of Oracle support agreements
and on-site and off-site support from the implementing party. Administrators will have several options for support,
including live assistance, phone support, and web forums.
This table outlines the decision points for the support infrastructure for the Mokum Oracle private cloud reference
design. For decisions that rely on preexisting factors or specific organizational needs, the appropriate best practice
will be discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be
analyzed carefully and decisions should be made based on organizational needs, existing architecture, and budget
resource availability.
Decision Point
Decision
Justification
Oracle Support
Agreements
Oracle Support Agreements for the Oracle

technologies will be active and up to date.
Support is an integral part of every

successful IT project. Oracle support
agreements are necessary to be able to
create and manage service requests as well
as to be able toreceive software patches
and updates from Oracle Enterprise
Manager and My Oracle Support.
On-site and Off-site

support
On-site and off-site support from the

implementing party will be used for
maintenance, site reviews, upgrades, and
security audits.
On-site and off-site support from the

implementing party for problem
resolution, system maintenance, site
reviews, upgrades, and security audits
augments the Oracle support agreement
and internal IT operations staff.
Oracle Private Cloud Architectural Design

The following sections provides the decision matrices for the Mokum Oracle private cloud reference design.
Implementers of the Mokum Oracle private cloud reference design can use the decisionmatricesas quick
reference guide to identify settings and configuration decisions to be implemented in the environment. These
decisions should be carefully analyzed during a gap analysis phase.
Oracle VM Hardware Architecture

The server hardware for your Oracle VM environment is a critical component in the success of your Oracle private
cloud project. Oracles virtualization portfolio includes software only solutions that fall into the build your own
option with Oracle VM for x86 software and OpenStack on commodity x86hardware, and the buy option with
select Oracle VM enabled Oracle engineered systems such asOracle Exadata X5-2, Oracle Database Appliance,
Call 415-252-9164
Call 415-252-9164
Oracle Exalogic, and the Oracle Private Cloud Appliance.Oracle VM for x86 software is Oracle's leading edge
server virtualization software that was designed to make Oracle databases, middleware and applications easier to
deploy, support, and license, on commodity x86 hardware. With the Oracle VM for x86 software you can build
your own private cloud on commodity x86 hardware managed by Oracle software, or OpenStack. Oracle offers
commercial support for Oracle VM for x86 and OpenStack on certified third-party hardware, with Sun hardware,
and bundled with Oracle Exadata Database Machine X5-2, Oracle Database Appliance, Oracle Exalogic, and the
Oracle Private CloudAppliance.
Note: Without exception our customers elect tobuild thier own Oracle private clouds using their standard x86
server hardware and storage platforms.As of this writing, over 85% of our Oracle VM projects are being deployed
on Cisco UCS with EMC storage.Cisco UCS withEMC storage is an idealplatform for Oracle VM hosting the
most demanding Oracle workloads.
Oracle engineered systems are preassembled Oracle/Sun hardware and software solutions with targeted
functionality that addresses the buy versus build option.Oracle Exadata X5-2 is a preassembled hardware and
software solution engineered to be the highest-performance and most-available platform for the Oracle Database.
The Oracle Database Appliance is a fixed two node entry level preassembled hardware and software solution
engineered for the Oracle Database. Exadata and the Oracle Database Appliance are not general purpose
virtualization solutions. Exadata and the Oracle Database Appliance have been engineered specifically for Oracle
Database workloads running on Oracle Linux. Oracle Exalogic is a preassembled hardware and software solution
engineered for large-scale, performance-sensitive, mission-critical Oracle and Java applications. Exalogic is not a
general purpose virtualization solution. Exalogic has been engineered specifically for Oracle Applications and Java
workloads running on Oracle Linux. The Oracle Private CloudAppliance is Oracle' converged infrastructure
solution engineered for general purpose virtualization.
Figure 2 shows the Oracle private cloud build verses buy options.
Call 415-252-9164
Call 415-252-9164
Oracle approach to virtualization with the focus onsolving challenges at the application layeris indeed in stark
contrast to the other commercial virtualization software, converged, and hyper-converged solutions that solve
applicationchallenges at the virtualinfrastructure layer.For example, Oracle VM and the Oracle
VMenabledOracle engineered systemshave been built from the ground up to manage Oracle licenses, and to
support the most demanding Oracles workloads. Oracle Exadata, Exalogic, and theOracle Private Cloud
Appliancehave a 40G InfiniBand fabric to allow the virtual machines to meet the most demanding I/O and SQL
processing requirements for online transaction processing, data warehousing, and mixed applicationworkloads.
Since Oracle Database, Applications and Middleware software have application level fault tolerance, and data
replication functionality, Oracles virtualization portfolio does not have virtual machine fault tolerance, or storage
migration functionality. Oracles virtualization portfolio relies on Oracle application level fault tolerance, and data
replication, not virtual machine level fault tolerance, and storage migration functionality. The other commercial
virtualization software, converged, and hyper-converged solutions solveapplicationchallenges at
thevirtualinfrastructure layerwithvirtual machinefault tolerance, and live storage migration.Oracle approach to
virtualization has limited Oracle VM's adoption to exclusivelysupporting Oracle workloads.
If you elect to build your own Oracle VM private cloud, theOracle VM for x86 software can be deployed on any of
the hardware platforms listed on theOracle Linux and Oracle VM Hardware Certification List (HCL).
Oracle VM Hardware Sizing

The firststep in selecting an Oracle VM hardware platform isto size the server hardware, followed by calculating
the total number of servers required to be in each Oracle VM server pool. The formula to calculate Oracle VM
server sizing is: The total aggregate virtual machine CPU, RAM and Storage requirements plus your N+x
availability requirements provides the total server count along with the server hardware and storage requirements.
Call 415-252-9164
Call 415-252-9164
Oracle VM server sizing is calculated by adding the aggregate CPU, RAM and I/O requirements for all of the
virtual machines that could run on an Oracle VM server, and then selecting server hardware with ample CPU,
RAM and I/O resources. The exact number of network interfaces for an Oracle VM server entirely depends on
your organizations server hardware platform, and network and storage infrastructure capabilities. For example, an
Oracle VM server with four or more 10G NICs, configured with two 802.1Q bonds could support the most
demanding Oracle workloads.The amount of Fibre and HBAs for an Oracle VM Server entirely depends on your
Oracle I/O requirments. For example, an Oracle VM derver with at least two dual-channel8GBHBAs, EMC
PowerPath with an EMC VNX, or VMAX could support an Oracle Data Warehousewith minimumthroughput
requirments of3500MBPS.Once the server hardware has been selected, the number of servers in a server pool is
calculated by selecting enough servers to support the aggregate CPU, RAM and I/O requirements of all of the
virtual machines within a server pool, including the number of additional servers for availability, i.e. HA, Live
Migration and Distributed Resource Scheduling (DRS). Oracle VM server pools that use HA, Live Migration and
DRS must have excess CPU and RAM capacity for hardware failures and virtual machine migrations.Oracle VM
HA and Live Migration along with Distributed Resource Schedulingmust be considered to calculate the total
number of servers required to respond to hardware failures and virtual machine migrations.
Figure 2shows Oracle VM server pool designed with excess CPU and RAM capacity to be able to use HA,
Distributed Resource Scheduling and Live Migration. Excess CPU and RAM capacity is a requirement for HA,
Distributed Resource Scheduling and Live Migration.
The below image shows an
Oracle VM server pool with
excess capacity to be able to use
HA, Live Migration and DRS.
Call 415-252-9164
The below image showsan Oracle

VM server poolresponding to a
HA event, with DRS and/or Live
Migration moving running virtual
machines. Excess CPU and RAM
capacity is required to be able to
run the virtual machine from the
failed Oracle VM server.
The below image showsan Oracle

VM server poolmigrating running
virtual machines using DRS
and/or Live Migration.Excess
CPU and RAM capacity is
required to be able to run the
virtual machine from the failed
Oracle VM server.
Call 415-252-9164
Oracle VM HA automatically restarts virtual machines when an Oracle VM pool member fails or restarts. Live
Migration is used to eliminate planned downtime by migrating running virtual machines from one Oracle VM pool
member to another during a maintenance event, for example, for repairs or an upgrade. Distributed Resource
Scheduling provides policy based real-time utilization monitoring of Oracle VM servers with the goal to distribute
virtual machine loads across a server pool. Distributed Resource Schedulingautomatically migrates virtual
machines from heavily utilized Oracle VM servers to less utilized Oracle VM servers. Both HA, Live Migration
and Distributed Resource Scheduling require a server pool with at least three servers with excess CPU and RAM
capacity to be able to run and migrate virtual machines across the servers even if one Oracle VM servers fails.
Note:Virtual machines cannot aggregate CPU and memory resources from more than one Oracle VM server. That
is, a virtual machine consumes resources only from the Oracle VM server where the virtual machine is running.
Oracle VM server can be installed on an x86 64 bit server with up to 900CPU cores or threads, up to 6TB of
RAM, with no limit on the number of network ports. Oracle VM server can be installed on as little as a 4GB
partition or disk. Many customers install Oracle VMserver on flash storage modules or boot from SAN to reduce
costs and complexity.
The next table shows the maximum number of CPUs, RAM and NICs for Oracle VM server release 3.2.x, and
3.3.x.
Item
3.2 Maximums
3.3Maximums
CPU Cores or Threads
160
900
RAM
4 TB
6 TB
NICs
40
No limit
Call 415-252-9164
Call 415-252-9164
Before starting an Oracle VM project, we profile the existing Oracle systems to be able to size the Oracle VM
hardware, storage and network infrastructure.The next example shows our system profile format with an EBusiness Suite environment on physical Linux systems, followed by the same E-Business Suite environment hosted
on Oracle VM. The below example shows each E-Business Suitecomponentinstalled on a dedicated HP DL 380.
Hostname
OS
Hardware
CPUs/Cores
Processor
Factor
RAM
Total
Peek
Storage IOPS
Average
IOPS
dbprod
Oracle
Linux
6
DL 380
4 CPUs/16
Cores
8 Licenses
128
GB
1.5 TB
10493
2102
dbtest
Oracle
Linux
6
DL 380
4 CPUs/16
Cores
8 Licenses
128
GB
1.5 TB
14493
3802
obiprod
Oracle
Linux
6
DL 380
2 CPUs/8
Cores
4 Licenses
32
GB
300 GB
1534
91
obitest
Oracle
Linux
6
DL 380
2 CPUs/8
Cores
4 Licenses
32
GB
300 GB
2834
152
soaprod
Oracle
Linux
6
DL 380
2 CPUs/8
Cores
4 Licenses
32
GB
300 GB
2491
68
soatest
Oracle
Linux
6
DL 380
2 CPUs/8
Cores
4 Licenses
32
GB
170 GB
2903
92
approd
Oracle
Linux
6
DL 380
2 CPUs/8
Cores
4 Licenses
32
GB
300 GB
673
22
aptest
Oracle
Linux
6
DL 380
2 CPUs/8
Cores
4 Licenses
32
GB
300 GB
1200
35
8 Hosts
80
CPUs/Cores
40Oracle
Processor
Licenses
448
GB
RAM
4.5 TB
Storage
36621
Peek
IOPS
6364
Average
IOPS
The below example shows the virtualized foot print from the above E-Business Suiteenvironment on Oracle VM
with hyperthreading enabled. Oracle VM reduced the server count by 62.5%, and reduced the Oracle license count
by 65%.
Call 415-252-9164
Call 415-252-9164
Hostname
OS
Hardware
*CPUThreads
Processor
Factor
RAM
Total
Peek
Average
Storage IOPS IOPS
dbprod
Oracle
Linux
6
VM
16 vCPUs
4 Licenses
128
GB
1.5 TB
10493
2102
dbtest
Oracle
Linux
6
VM
16 vCPUs
4 Licenses
128
GB
1.5 TB
14493
3802
obiprod
Oracle
Linux
6
VM
4 vCPUs
1 Licenses
32
GB
300 GB
1534
91
obitest
Oracle
Linux
6
VM
4 vCPUs
1 Licenses
32
GB
300 GB
2834
152
soaprod
Oracle
Linux
6
VM
4 vCPUs
1 Licenses
32
GB
300 GB
2491
68
soatest
Oracle
Linux
6
VM
4 vCPUs
1 Licenses
32
GB
170 GB
2903
92
approd
Oracle
Linux
6
VM
4 vCPUs
1 Licenses
32
GB
300 GB
673
22
aptest
Oracle
Linux
6
VM
4 vCPUs
1 Licenses
32
GB
300 GB
1200
35
14Oracle
Processor
Licenses
448
4.5 TB
36621 6364
GB
Storage Peek
Average
RAM
IOPS IOPS
3DL 380s each 56 vCPUs

with:
2
CPUs/24Cores
192 GB RAM
2 120 GB
Disks RAID 1
2dualchannel8GB
HBAs
*Note:With hyper-threading enabled, a virtual machine with 4 CPUs is actually accessing 2 CPU cores, 4 CPU
threads, or 1 Oracle CPU license. With hyper-threading disabled, the same virtual machine with 4 CPUs is actually
Call 415-252-9164
Call 415-252-9164
consuming 4 CPU cores, or 2 Oracle CPU license. Oracle recognizes each CPU core as a separate CPU and each
CPU type with a different processor factor. The processor factor determines the processor count. The processor
count determines the number of processors required to license the Oracle product.
The majority of our customers order their standard VMware hardware for Oracle VM, and connect the Oracle VM
servers to their existing storage infrastructure. The majority of theVMware hardware we see has 2 multi-core
CPUs, up to 256 GB of RAM, 2 RAID 1 disks, 4 10G NICs, and 2 1-Port 8Gb HBAs. There are pros and cons to
using your standard VMware hardware and existing storage for Oracle VM. The pros are that your standard x86
server hardware is simple to procure and deploy, and your existing storage infrastructure is in-place and ready to
go. The cons are that production Oracle database workloads can easilyrequiremore I/O throughputthan your
standard VMwarehardware and storage infrastructurecan deliver.
A smaller number of our customers size and order server hardware and storage infrastructure specificallyfor their
Oracle VM environments. These folks typically order servers with 4 multi core CPUs, up to 4TB of RAM, 2 RAID
1 disks, 6 to 8 10G NICs, and 2 2-Port 8Gb HBAs, with dedicated EMC VNX, or VMAX storage. There are pros
and cons to using non standard x86 hardware and dedicated storage for Oracle VM. The pros are that your
production Oracle database workloads will have all of the resources they need. The cons are that a new x86
hardware will be introduced, with dedicated storage, at a higher cost.
A single Oracle VM 3.3server can support up to 900CPU cores or threads, 6TB of memory with local or remote
storage. An Oracle VM server with 6TB of RAM and 900CPU cores or threads could allocate the majority of the
6TB of RAM and more than 900CPU cores or threads to running virtual machines. Oracle VM server supports
CPU oversubscription. CPU oversubscription means that an Oracle VM server with 900CPU cores could
overallocate the total number of CPU cores to virtual machines. Oracle VM server does not support memory
oversubscription, which means that an Oracle VM server with 6TB of RAM cannot overallocate RAM to virtual
machines. By default, each Oracle VM server reserves at leat 512 MB of RAM for Oracle VM server (dom0). The
average memory overhead for each running virtual machine on an Oracle VM server is approximately 20 MB plus
1% of each virtual machine' memory allocation. The remaining RAM can be allocated to virtual machines.
Note: The lack of Oracle VM's memory oversubscription is by design. Memory oversubscription is actually in
Xen, and could easily be enabled in Oracle VM. If customers used CPU oversubscription with Oracle database
workloads, the virtual infrastructure including the Oracle databases would quickly slow down to a crawl with
memory starvation.
The next table outlines the various decision points to size and select Oracle VM for x86 hardware. For decisions
that rely on preexisting factors or specific organizational needs, the appropriate best practice will be discovered in
the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and
decisions should be made based on organizational needs, existing architecture, and budget resource availability.
Decision Point
Decision
Justification
Certification
The server hardware should be jointly

supported by the hardware vendor and
Oracle.
Only jointly supported hardware product

receive vendor support when problems
occur and service tickets are created. The
server hardware must be jointly supported
Call 415-252-9164
Call 415-252-9164
Note: The following link is the Oracle'

hardware certification page.
http://linux.oracle.com/pls/apex/f?
p=117:1:5773793518142288::NO:RP::
by the hardware vendor and Oracle.
CPU
Server hardware should be ordered with at

least two Intel multiple-core CPUs for
small and medium workloads and four
multiple-core CPUs for large Oracle
database workloads.
The Maximum Number of CPU cores or

threads an Oracle VM Release 3.3 server
can support is 900. Oracle VM server
maps a virtual CPU (vCPU) to a hardware
thread on a CPU core in a CPU socket.
Oracle VM Server supportsCPU

oversubscription. CPU oversubscription
allows an Oracle VM server with 900
CPU cores or threadsto overallocate the
total number of CPU cores to virtual
machines. For example, a server with an
Intel Xeon processor 5600-series CPU
with hyperthreading can have up to six
cores and twelve threads per socket. A two
socket server with an Intel Xeon processor
5600-series CPU could allocate twenty
four virtual CPUs without oversubscribing
the physical CPUs.
CPU-bound workloads, such as Oracle

Databases, should not be on Oracle VM
servers with oversubscribed CPUs.
RAM
Server hardware should be ordered with

the maximum amount of physical
memory.
Note: Oracle VM Release 3.3supports up

to 6TB of RAM.
Oracle VM Server does not support

memory oversubscription. For example,
an Oracle VM server with 1TB of RAM
cannot overallocate RAM to virtual
machines. By default, each Oracle VM
server reserves at least 512MB of RAM
for dom0. The average memory overhead
for each running guest on a dom0 is
approximately 20MB plus 1% of the
guests memory size. The remaining
physical RAM can be allocated to guests.
An Oracle VM server in a server pool with

Live Migration, DRS, DPM and/or HA
must haveexcessRAMcapacityto accept
virtual machines from a Live Migration,
Call 415-252-9164
Call 415-252-9164
DRS, DPM and/or HA operation. Oracle

VM pool members without available
RAM can not support Live Migration,
DRS, DPM and/or HA. Having available
RAM on each server provides flexibility
in terms of adding new virtual machines to
the server pool, and to allow Live
Migration, DRS, DPM and/or HA within a
server pool.
Local Storage
Unless the Oracle VM server is booting

from SAN, redundant internal hard drives
are recommended.
Virtual machine image and configuration
files are hosted on local, shared SAN,
iSCSI, or NFS repositories.
Network Interface
Cards
A minimum of one Ethernet network

interface (NIC) card is required just to
install Oracle VM server, although at least
four or more 10G NICs is strongly
recommended. NIC bonding withportbased VLANs and/or802.1Qtag-based
VLANs are supported and configured post
Oracle VM Server installation with Oracle
VM Manager or Enterprise Manager.
Oracle VM 3.0.1 through 3.1.1 supports
two NIC ports per network bond, and a
total of five network bonds per Oracle
VM server.
Oracle VM 3.2.x supports four NIC
ports per network bond,and a total of
ten network bonds per Oracle VM
server.
Oracle VM 3.3.x supports an unlimited
number of NICs, andbonds.
The exact number of network interfaces for
an Oracle VM server entirely depends on
your organizations business requirements,
server hardware, and network and storage
infrastructure. For example, there are no
Call 415-252-9164
Oracle VM Server requires 4GB of local

storage for the Oracle VM server
installation. The design goal for Oracle
VM is to support multiple node Oracle
VM server pools with shared fibre channel
SAN, iSCSI and/or NFS storage.
Oracle VM local storage does not support

HA or Live Migration.
Both 802.3AD NIC bonds, port-based
VLANs and/or 802.1Q tag-based VLANs
are supported and configured post Oracle
VM server installation with Oracle VM
Manager. Network redundancy, i.e.
802.3AD NIC bonding doubles the
number of required NICs.
Oracle VM uses a total of five discrete

networks; Server Management, Cluster
Heartbeat, live Migration, Storage and
Virtual Machines. All five networks can
be supported using 802.1Q tag-based
VLANs or using access ports.
Each Oracle VM server pool should have a

discrete network for the Server
Management, Cluster Heartbeat, Live
Migration, Storage and Virtual Machines.
Isolating theCluster Heartbeat, Live
Migration and Storage networks is
necessary to be able to protect
theserver'sheartbeat from interruptions
that could cause a pool member toreboot.
Call 415-252-9164
NIC limitations with a Cisco UCS

hardware, in contract to legacy hardware
with physical NICs. Cisco UCS supports
provisioning as many HA enabled vNICs
as necessary to meet the most demanding
Oracle VM network requirements, in
contrast to legacy hardware that could
require up to 6 10G NICs, or 12 or more
1G ports. It is hard to succeed without a
plan. Plan your Oracle VM project in
advance before ordering hardware, and
deploying Oracle VM.
Each Oracle VM server will be assigned a

unique IP address on the Server
Management, Cluster Heartbeat, live
Migration and Storage network.
NAME Rate(bit/s)
Gigabit Ethernet 1 Gbit/s
10 Gigabit Ethernet 10 Gbit/s
40 Gigabit Ethernet 40 Gbit/s
Infiniband DDR 16 Gbit/s
Tip: One thing to consider is NIC

firmware levels between bonded internal
NIC ports and PCI NIC ports. Consider
only bonding internal NICs with internal
NICs and PCI NICs with PCI NICs.
Host Bus Adapter
Cards
SAN Storage: At least 2 dual-channel8GB

Host Bus Adapter Cards (HBAs).
NAME Line-Rate Throughput

MBps
4GFC 4.25 800
8GFC 8.5 1600
10GFC 10.52 2550
16GFC 14.025 3200
20GFC 21.04 5100
At least 2dual-channel8GBHost Bus

Adapter Cards should beused to meet
Oracle Fibre Channel throughput
requirments and toeliminate a single point
of failure resulting in serverreboots due to
OCSF2 storage heartbeat interruptions .
Oracle VM Server Pool Design

Oracle VM uses the concept of a server poolto group together and centrally manage one or more server pools from
one or moreOracle VM Managers. If more than one location exists, Oracle VM server pools may be dispersed to
different locations and be managed via a local installation of Oracle VM Manager, or centrally managed from a
single Oracle VM Manager.Oracle VM server pools can be designed to accommodates organization-specific
needs, i.e., Oracle technology license management (hard and soft partitioning) , defense in depth, the principle of
least privilege, compartmentalization of information, and different applications and their performance,
authentication, and security requirements.
Call 415-252-9164
Call 415-252-9164
We often designOracle VM environments following FIPS PUB 199,theStandards for Security Categorization of
Federal Information and Information Systems to applydefense in depth, the principle of least privilege, and
compartmentalization of information.FIPS PUB 199 can be used to determine the security category of systems
and within which security domain systems should reside.FIPS PUB 199 uses the followingsecurity domains:
Controlled:A controlled security domain is used to restrict access between security domains. A controlled
security domain could contain groups of users with their network equipment or a demilitarized zone (DMZ).
Uncontrolled:An uncontrolled security domain refers to any network not in control of an organization, such as
the Internet.
Restricted:A restricted security domain can represent an organizations production, test and development
networks. Access is restricted to authorized personnel, and there is no direct access from the Internet.
Secured:A secured security domain is a network that is only accessible to a small group of highly trusted users,
such as administrators and auditors.
Figure 3shows fiveOracle VM server pools, managed by three Oracle VM Managersdeployed into the controlled,
restricted,and the secured security domains.
If more than one Oracle VM server pool exists, or locations with Oracle VM server pools, a single Oracle VM
Manager instance can be deployed, or each Oracle VM server pool and/or location could have a dedicated Oracle
VM Manager instance. Figure 3 shows fourOracle VM server pools, 2 oftheOracle VM server pools have a
dedicated Oracle VM Manager instance, and 3 of the server pools are managed by 1Oracle VM Manager instance.
Tip:Oracle Enterprise Manager can be used to aggregate multiple Oracle VM Managers and their resources in to a
single centralized console.
Oracle VM for x86 supports both clustered and non-clustered Oracle VM server pools. The primary difference
between clustered and non-clustered Oracle VM server pools is that clustered server pools use the Oracle Cluster
File System Version2 (OCFS2) with High Availability (HA), and OCFS2 automatic server reboot functionality.
Non-clustered Oracle VM server pools use NFS without OCFS2, High Availability, and OCFS2 automatic server
reboot functionality. Clustered Oracle VM server pools are managed using OCFS2. OCFS2 monitors the status of
each server within an OCFS2 cluster using a network and storage heartbeat. If an Oracle VM server within an
OCFS2 cluster fails to update or respond to network and/or storage heartbeats, by default OCFS2 reboots the
Call 415-252-9164
Call 415-252-9164
affected Oracle VM server. Non-clustered Oracle VM server Pools are not managed using OCFS2, and therefore
are not subject to OCFS2 automatic server reboots.
We have many customers that have been successfully running clustered Oracle VM server pools, each with
exceptional uptime of several years. Many of the Oracle VM server pools with exceptional uptime are static, with
very little change. Conversely, we have also supported many Oracle VM environments that have struggled to tune
out the storage or network interrupts that trigger OCFS2 automatic server reboots. If your clustered Oracle VM
server pools reboot due to storage or network hiccups, and a solution cannot be found, or if you simply cannot
tolerate Oracle VM Server, and virtual machine reboots, I highly recommend non-clustered Oracle VM server
pools.
The following table shows the high level architecture of a clustered and non-clustered Oracle VM server pool.
Figure 4 -Clustered Oracle VM server pool
Figure 5 -Non-clustered Oracle VM server pool
The following lists provides the pros and cons of clustered and non-clustered Oracle VM server pools.
Clustered Server Pool Pros:
Clustered Oracle VM server pools support Oracle VM HA.
Clustered Oracle VM server pools support virtual machine hot cloning.
Clustered Oracle VM server pools use Fibre Channel for virtual machine storage repositories, and virtual
machine RAW disks, and can also use NFS storage repositories, and shares.
Clustered Server Pool Cons:
Call 415-252-9164
Call 415-252-9164
If a clustered Oracle VM server fails to update or respond to network and/or storage heartbeats, the affected
Oracle VM Server is rebooted, then all HA-enabled virtual machines are restarted on a live Oracle VM Server.
Clustered Oracle VM server Pools require Fibre Channel storage.
Non-clustered Server Pool Pros:
Non-clustered Oracle VM servers will not reboot when network and/or storage interruptions occur.
Non-clustered Oracle VM server pools use NFS for the virtual machine storage repositories, and can use Fibre
Channel for virtual machine RAW disks, and NFS shares.
Non-clustered Server Pool Cons:
Non-clustered Oracle VM server pools do not support Oracle VM HA.
Non-clustered Oracle VM server pools do not support virtual machine hot cloning.
Non-clustered Oracle VM server pools require NFS for the virtual machine storage repositories.
Note:Both clustered and non-clustered Oracle VM server pools support local OCFS2 storage.
This table outlines the decision points foran Oracle VM server pool. For decisions that rely on preexisting factors
or specific organizational needs, the appropriate best practice will be discovered in the infrastructure assessment
(IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be made based
on organizational needs, existing architecture, and budget resource availability.
Decision Point
Decision
Justification
Oracle VM server
pool design
Prior to implementing an Oracle Cloud, its

important that an infrastructure assessment
(IA) and gap analysis (GA) be performed.
During the IA/GA, the architecture of the
solution will be matched to the customers
business needs.
Oracle VM server pool design is a

strategic, architectural security decision. It
is important that an infrastructure
assessment (IA) and gap analysis (GA)be
performed to be able to select an Oracle
VM server pool design thatmatchsthe
customers business needs.
Oracle VM
Manager
Oracle VM uses the concept of a server

poolto group together and centrally
manage one or more Oracle VM
serversfrom one or moreOracle VM
Managers. If more than one location
exists, Oracle VM server pools may be
dispersed to different locations and be
managed via a local installation of Oracle
VM Manager, or centrally managed from
a single Oracle VM Manager.
Oracle VM Manager design is a strategic,

architectural security decision. It is
important that an infrastructure assessment
(IA) and gap analysis (GA)be performed
to be able to select an Oracle
VMManager design thatmatchsthe
customers business needs.
If more than one Oracle VM server pool

exists, or locations with Oracle VM server
pools, a single Oracle VM Manager
instance can be deployed, or each Oracle
VM server pool and location could have a
Call 415-252-9164
Call 415-252-9164
dedicated Oracle VM Manager instance.
Oracle Enterprise Manager can be used to

aggregate multiple Oracle VM Managers
and their resources in to a single
centralized console.
Cluster verses Nonclustered Oracle
VM Server Pools
The primary difference between clustered

and non-clustered Oracle VM server pools
is that clustered server pools use the
Oracle Cluster File System Version2
(OCFS2) with High Availability (HA),
and OCFS2 automatic server reboot
functionality. Non-clustered Oracle VM
server pools use NFS without OCFS2,
High Availability, and OCFS2 automatic
server reboot functionality.
If a clustered Oracle VM server fails to

update or respond to network and/or
storage heartbeats, the affected Oracle VM
Server is rebooted, then all HA-enabled
virtual machines are restarted on a live
Oracle VM Server. Irecommendnonclustered Oracle VM server pools because
thayare not subject to OCFS2 automatic
server reboots. If you deployclustered
Oracle VM server pools,
werecommendsetting the maximum
storage and network heartbeat
timeoutvalue to 5 minutes, or 300in
Oracle VMManager, or 151 in the
/etc/sysconfig/o2cbfileto avoid or reduce
Oracle VMserver reboots.
OCFS2 Storage and

Network Heartbeat
Timeout Value
(Clustered Oracle
VM server pools)
Note that OCFS2 Storage and Network

Heartbeat Timeout Values areonly
applicable for clustered Oracle VM server
pools. Non-clustered Oracle VM server
pools do not have cluster heartbeats.The
storage and networkheartbeat time out
values areselected when the Oracle VM
server pool is created.
Network Heartbeat Timeout Value:

Once the cluster is on-line, each pool
member starts a process, o2net. The o2net
process creates TCP/IP intra-cluster node
communication channels on port 7777 and
sends regular keepalive packages to each
node in the cluster to validate if the nodes
are alive. The intra-cluster node
communication uses the Cluster Heartbeat
network channel. If a pool member loses
network connectivity the keepalive
connection becomes silent causing the
If a clustered Oracle VM server fails to

update or respond to network and/or
storage heartbeats, the affected Oracle VM
Server is rebooted, then all HA-enabled
virtual machines are restarted on a live
Oracle VM Server. Irecommendnonclustered Oracle VM server pools because
thayare not subject to OCFS2 automatic
server reboots. If you deployclustered
Oracle VM server pools,
werecommendsetting the maximum
storage and network heartbeat
timeoutvalue to 5 minutes, or 300in
Oracle VMManager, or 151 in the
/etc/sysconfig/o2cbfileto avoid or reduce
Oracle VMserver reboots.
Call 415-252-9164
Call 415-252-9164
node to self-fence,i.e.reboots.The
keepalive connection time out value is set
via Oracle VM manager and is managed
in each nodes /etc/sysconfig/o2cb file's
O2CB_IDLE_TIMEOUT_MS setting.
Storage Heartbeat Timeout Value:

Along with the network heartbeatthat
check for node connectivity, the cluster
stack also employs a disk heartbeat check.
o2hb is the process that is responsible for
the disk heartbeat component of cluster
stack that actively monitors the status of
all pool members. The heartbeat system
uses a file on the OCSF2 file system, that
each pool member periodically writes a
block to, along with a time stamp. The
time stamps are read by each pool member
and are used to check if a pool member is
alive or dead. If a pool members block
stops getting updated, the node is
considered dead, and self-fences, i.e.
reboots. The disk heartbeat time out value
isselectedwhenthe Oracle VM server
pool is created, and is maintainedin each
Oracle VM Server's/etc/sysconfig/o2cb
file's
O2CB_HEARTBEAT_THRESHOLD
setting.
Monitoring and
Alerting
Call 415-252-9164
The Oracle VM product family; Oracle

VM server, Oracle VM Manager, virtual
machines, Oracle VM Templates and
Assemblies can be managed and
monitored with Oracle VM Manager and
Oracle Enterprise Manager 12c. Unlike
Oracle VM 2.x, which could only be
managed by Oracle VM Manager or
Oracle Enterprise Manager, not both,
Oracle VM 3 and above can be managed
simultaneously by Oracle VM Manager
along with Oracle Enterprise Manager 12c
Cloud Control.
Oracle VM Manager is a stand-alone

When things go wrong with Oracle VM,

being able to quickly determine the root
cause of an issue can eliminate or reduce
down time. The most effective way to
identify problems with Oracle VM is to
analyze the Oracle VM Manager, the
Oracle VM servers, and the virtual
machines performance statistics, and log
files using alog file analytics
solution,Oracle Enterprise Manager or an
SNMP based monitoring solution.
Call 415-252-9164
management solution for Oracle VM, with

limited monitoring and alerting
functionality. Oracle VM is a
defaultOracle Enterprise Manager
12cfeature that provides Infrastructure as
a Service (IaaS), Database as a Service
(DaaS), Platform as a Service (PaaS) and
Testing as a Service (TaaS) provisioning
with a self-service portal. Oracle VM
should be enabled in Cloud Control by
installing an Oracle Management Agent
with the Virtualization plug-in on a
managed Linux target with Oracle VM
Manager. Once Oracle VM is enabled in
Cloud Control, Oracle VM Manager,
Oracle VM servers, and all the virtual
machines can be managed, and setup with
performance monitoring profiles and alerts
that can be used for root cause and
statistical analysis.IfOracle Enterprise
Manager is not an option,Oracle VM can
be monitored using anSNMP
basedsolution.
A log file analytics solution canbe

configured to capture the Oracle VM
server, the Oracle VM Manager, and the
virtual machine operating systemslog
files.
Network Time
Protocol (NTP)
Call 415-252-9164
With Oracle VM, accurate time is essential

to maintain system stability due to timesensitive cluster transactions between
Oracle VM servers. Without accurate
time, Oracle VM clusters can be brought
to a complete standstill.
By default, Oracle VM servers (up to

Release 3.1.1) that are discovered by
Oracle VM Manager are configured to use
the Oracle VM Manager host as the
upstream NTP time host.
A best practice is to have two internal NTP

servers on your local network to provide
time services for internal systems and
devices. Using internal time servers
normalizes system event time-stamps
across the Enterprise as well as reduces
NTP Internet bandwidth usage.
If internal time servers are not an option,

set-up the Oracle VM Manager hostsas
the upstream NTP time hostto
synchronize with upstream Coordinated
Universal Time (UTC) sources as well as
provide time services to Oracle VM
Call 415-252-9164
servers.
Oracle VM Server
Agent Roles
Oracle VM Manager facilitates centralized

management of server pools and their
resources using an agent-based
architecture. When an Oracle VM server is
added to a server pool, up to three Oracle
VM agent roles can be enabled. There are
a total of three Oracle VM agent roles; 1)
Master Server, 2) Utility Server and 3)
VM Server. When an Oracle VM server is
added to a server pool, it can be assigned
one, two, or all three of the agent roles.
Master Server Role

The Master Server is the principal
server pool role within a server
pool. TheMaster Serveris the
server that communicates with
Oracle VM Manager. TheMaster
Serverdispatches commands
received from Oracle VM Manager
to other servers within a server
pool. There can be only oneMaster
Serverin a server pool at any
instant.The Virtual IP feature is a
mandatory server pool property
that detect the loss of the server
pool master agent and responds
with automatic failover to the first
pool member that can lock the pool
file systsm. The server pool Virtual
IP feature removes the single point
of failure (SPOF) for the server
pool master agent role.
Master Server Role

By default each clustered server pool must
have one Master Server with the Virtual
IP feature enabled.
Utility Server Role

The Utility Server roleis responsible for
I/O-intensive operations such as virtual
machine creation and removal, as well for
as creating, deleting, modifying, copying
and moving virtual machine files.
Enabling the Utility Server agent role with
the VM Server role on the same Oracle
VM server will negativelyaffect running
virtual machines during Utility Server
operations.Server pools that are not static
and support theself service portal in
Oracle Enterprise Manager, or
OpenStackshould have more than one
dedicated Utility Servers to service I/O
intensive jobs.
VM Server Role
Unless a server pool is static, VM Servers
should only have the VM Server role
enabled to be able to dedicate CPU, RAM
and I/O resources to running virtual
machines, eliminating the effect of Utility
Server operations.
Utility Server Role

The Utility Server role is
responsible for I/O-intensive
operations such as virtual machine
creation and removal, as well for as
creating, deleting, modifying,
Call 415-252-9164
Call 415-252-9164
copying and moving virtual

machine files. TheMaster
Serverdispatches operations to
Utility Servers. There can be one
or more Utility Servers in a server
pool. When there are multiple
Utility Servers in a pool, the server
Master Server will select the least
loaded utility server to conduct a
task.
Tip:Oracle VM environments that

dynamically grow should have
more than one dedicated Utility
Servers to service I/O intensive
jobs. For example, an I/O intensive
job that runson an Oracle VM
server with the VM Server and
Utility Server role enabled will
impact the performance of all of
the virtual machines running on the
Oracle VM server.
VM Server Role
Servers with the VM Server role are
responsible for allocating CPU,
memory, and disk resources to the
virtual machines in a server pool.
There can be one up to 32 VM
Servers in a clustered Oracle
VMserver pool.
Storage
Oracle VM for x86 supports both clustered

and non-clustered Oracle VM server
pools. The primary difference between
clustered and non-clustered Oracle VM
server pools is that clustered server pools
use block storage with the Oracle Cluster
File System Version2 (OCFS2), with
High Availability (HA), and OCFS2
automatic server reboot functionality.
Non-clustered Oracle VM server pools use
file storage (NFS)without OCFS2, High
Call 415-252-9164
Oracle VM storage is a strategic,

architectural decision. It is important that
an infrastructure assessment (IA) and gap
analysis (GA)be performed to be able to
select an Oracle VM storage solution
thatmatchsthe customers business needs.
An Oracle VM storage solution consists of

three distinct layers. Each layer has its
own unique requirements, configurations,
dependencies and features.
Call 415-252-9164
Availability, and OCFS2 automatic server

reboot functionality.
Clustered Oracle VM server Pools use

OCFS2 and NFS storage, along with Raw
Device Mapping of SAN LUNs to
individual guests for data/database files.
Non-clustered Oracle VM server pools use

NFS storage andRaw Device Mapping of
SAN LUNs to individual guests for
data/database files.
The first layer is the storage array, which

is referred to as back-end storage. Oracle
VM supports Fibre Channel and iSCSI
SAN and NFS back-end storage.
Confirm that throughput and latency
needs of the databases can be meet
with the storage array, HBAs, or NICs.
Plan for excess I/O capacity for meeting
peak loads.
Plan the size and type of storage by
workload.
SAN Pre-deployment Prerequisites:
Confirm portconfigurations in the SAN
andswitch
Confirm the HBA slots
Confirm gbics
Confirm fiber cable length limits
Confirm that fiber is not bent
The second layer is the server data
transfer layerconsisting of the default
Oracle VM server's Device-Mapper
Multipath configurations, or EMC
PowerPath, and the connectivity toshared
Oracle Cluster File System 2 (OCFS2) or
NFS resources. Please note that we see
well over20%better read performance
with EMC PowerPathcompaired
toDevice-Mapper Multipath.
Tip: OCFS does not factor disk space

exhaustion including space for virtual
machine files as well as volume
metadata.OCFS2 metadata can consume
over6% of an OCFS2 volumes free disk
space. Plan accordingly or as soon as your
OCFS2 volumes become ~95% full they
will go read only.
The third layer is the virtual machine

front-end storage consisting of multiple
Call 415-252-9164
Call 415-252-9164
guest storageoptions such asfile and Raw

Device Mappings. Raw Device Mapping
of SAN LUNs to individual guests for
data/database files have the absolute best
performance of the two front-end storage
storage options. In most cases, Raw
Device Mappingsare the only option for
high I/O workloads like Oracle databases.
DM-Multipath
verses EMC
Powerpath
Multipathing is the technique of creating

more than one physical path between the
server CPU and its storage devices. It
results in better fault tolerance and
performance enhancement. By default
Oracle VM uses the Open Source solution
dm-multipath.
EMC Powerpath can provide as much as

20% better read and write performance
when compared to dm-multipath.
Networks
Oracle VM uses a total of five discrete

networks; Server Management, Cluster
Heartbeat, live Migration, Storage and
Virtual Machines.
The exact number of network interfaces for

an Oracle VM server entirely depends on
your organizations business requirements
and network and storage infrastructure
capabilities. For example, an Oracle VM
server with four 10G NICs, configured
with two 802.1Q bonds could support the
most demanding network and storage
requirements, with only four 10G NICs.
By contrast, an Oracle VM server
usingaccess ports/port-based VLANs
or802.1Q tag-based VLANSon a 1G
copper network, could easily use 8 or
more NIC ports to meet
theminimumnetwork requirements.
Each Oracle VM server pool must have a

discrete network for the Server
Migration, Storage and Virtual Machines.
Isolating the Cluster Heartbeat, live
Migration and Storage networks is critical
to protect the servers from OCSF2
heartbeat interruptions that can cause pool
members to fence from the pool and
reboot.
Each Oracle VM server should be assigned

a unique IP address on the Server
Migration and Storage network.
RAM
Oracle VM server pools should be

designed with excess RAM capacity to
accommodate the memory requirements of
virtual machines that could migrate or
start on any pool member.
Oracle VM server does not support

memory oversubscription, which means
that an Oracle VM server cannot accept a
DRS, Live Migration or HA requests
unless the server has available RAM for
the virtual machines. Having excess RAM
on each Oracle VM server is required for
growth, Distributed Resource Scheduling,
Call 415-252-9164
Call 415-252-9164
Live Migration and HA.

NUMA
Contemporary CPUs from Intel and AMD

have NUMA architectures. NUMA stands
for Non-Uniform Memory Access. With
NUMA each physical CPU (pCPU) will
be assigned its own local memory. An
assigned processor-memory pair is called
a NUMA node. Local memory access
from CPUs on the same socket will have
significantly lower latency than remote
memory access from CPUs on a different
socket.
Oracle VM supports NUMA using a Xen

feature called NUMA aware scheduling.
NUMA aware scheduling will assign a
virtual machine's vCPUs (virtual CPUs) to
a NUMA node as a NUMA client. If a
virtual machine has multiple vCPUs, the
NUMA scheduler will always assign the
virtual machine's vCPUs to a single
NUMA node to maintain memory locality.
For example, an Oracle Database virtual
machine with 32 vCPUs allocated to a
single NUMA node with 20 threads would
be oversubscribed. CPU-bound
workloads, such as Oracle Databases,
should not be on Oracle VM servers with
oversubscribed CPUs.
If your supporting virtual machines with more

vCPUs than its NUMA node, disable NUMA.Xen
NUMA aware scheduling will place a virtual
machine with 32 vCPUs on a single NUMA node,
even if the node does not have 32 cores or threads,
essentally oversubscribingthevirtual machines
vCPUs.
Oracle VM Security Standards

The security controls used to secure Oracle VM are similar to the security controls used to protect your existing
physical and virtual IT resources. As with physical and virtual IT resources, securing Oracle VM is dependent on
the security posture of each of its components, from the design, hardware, hypervisor, network, and storage to the
virtual machine operating systemsand installed applications. In short, if the organization has a security policy for
virtualization, networking, storage, operating systems and applications, the security policies could and should be
applied to Oracle VM.
Security controls should be employed using industry standard frameworks and standards in the context of the
organization's Enterprise Architecture (EA). Organizations turn to their Enterprise Architecture to understand how
Oracle VM fits within their information system. An Enterprise Architecture is articulated in diagrams and written
policies that define organizational standards and best practices to plan, build, run, and monitor technologies,
including Oracle VM.
Call 415-252-9164
Call 415-252-9164
Enterprise Architecture has well defined principles and processes and an approach that generates a comprehensive,
layered policy infrastructure used to communicate managements goals, instructions, procedures, and response to
laws and regulatory mandates. A policy infrastructure consists of written tier 1, tier 2, and tier 3 policies that
encompass people, systems, data, and information. Policies are broken down into high level policies and lower
level standards, procedures, baselines, and guidelines.
Oracle VM policies typically fall within the layered policy infrastructure of the platform architecture domain.
Platform architecture policies are the foundation used to manage the entire lifecycle of an Oracle VM environment.
This table outlines the decision points for Oracle VM Manager security controls. For decisions that rely on
preexisting factors or specific organizational needs, the appropriate best practice will be discovered in the
infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and
Oracle VM ManagerSecurity Controls

Decision Point
Decision
Justification
Oracle VM
Manager and
DMZs
The Oracle VM Manager application was

not designed to be an Internet facing
application.If Internet access is a
requirement for Oracle VM Manager,
VPN access should be used to access the
Oracle VM Manager GUI.
The Oracle VM Manager application was

not designed to be an Internet facing
application.
Network Time
Protocol (NTP)
With Oracle VM, accurate time is essential

to maintain system stability due to timesensitive cluster transactions between
Oracle VM servers. Without accurate
time, Oracle VM clusters can be brought
to a complete standstill.
A best practice is to have two internal NTP

servers on your local network to provide
time services for internal systems and
devices. Using internal time servers
normalizes system event time-stamps
across the Enterprise as well as reduces
NTP Internet bandwidth usage.
If internal time servers are not an option,

set-up the Oracle VM Manager hostsas
the upstream NTP time hostto
synchronize with upstream Coordinated
Universal Time (UTC) sources as well as
provide time services to Oracle VM
servers.
Virtual Machine
Console Access
Up to Oracle VM Release 3.2 usedthe

RAS proxy (Remote Access Service) java
applet to proxy virtual machine console
traffic from Oracle VM Manager to the
administrator's Client PC. An Oracle VM
Manager administrative account is a
All Oracle VM administrative users have

root access to all of the objects managed
by Oracle VM Manager. Virtual machine
end users such as DBAs and application
administrators should only have access to
thier virtual machines, notroot access to
Call 415-252-9164
Call 415-252-9164
requirement to access a virtual machine's

console.Any firewall between Oracle VM
Manager andthe administrator's Client
PCconecting to a virtual machine console
must have TCP port 15901 open for the
RAS proxy.
Oracle VM Manager does not support role

based access control. All administrative
users with access to the Oracle VM
Manager GUI have root administrative
access to all of the objects managed by
Oracle VM Manager, including all of the
virtual machine consoles.
Host firewall
all of the objects managed by Oracle VM

Manager.
If an Oracle VM Manager account is not an

option for a user, for example for DBAs,
Opertaions. or application
administators,Oracle VMrole based
access control can be configured
usingEnterprise Manager Cloud
Control.With Cloud Control, Roles can be
assigned to limit access to only select
virtual machines, or read only acces
totheOracle VM Manager objects.
The iptables service may be enabled on

each Oracle VM Manager host using a
ruleset managed in /etc/sysconfig/iptables.
In order to use Oracle VM Manager, the
Core API and the Oracle Management
Agent with iptables, it is necessary to open
tcp ports 7001, 7002, tcp-54321 or tcps54322, 15901 and 3872 as well as UDP
123.
Firewalls are the first line of defense in

network security. Firewallsshould beused
to filter network traffic between security
domains.Host firewalls, for example
iptables, are a fundamental part of
information security that protect hosts
from attacks and intrusions.
Host firewallfailed
connection logging
Iptables failed connection logging should

be enabled on each Oracle VM Manager
host.
The following two lines will be added prior

to the last REJECT line in the
/etc/sysconfig/iptables file:
-A RH-Firewall-1-INPUT -m limit --limit
15/minute -j LOG
--log-prefix "FW Drop:"
-A RH-Firewall-1-INPUT -j REJECT -reject-with icmp-hostprohibited
Failed connect logging is a fundamental

part of information security that allows
detection of attacks and intrusions.
Root ssh access and

sudo
By default, Oracle Linux and Oracle VM

permit ssh access using the root super user
account.
Systems administrators should access the

Oracle VM Manager and Servers
withnon-root individual user accountsand
use sudoto perform selected
One of the most important security

measure that can be taken with Oracle VM
is to prevent unauthorized access to the
root user account by disabling root ssh
access. A best practice is toonly allow
non-root individual user accounts
accessthat can be audited, disabled,
expired and managed using sudo.
Call 415-252-9164
Call 415-252-9164
administrative tasks.Sudo stands for

either "substitute user do" or "super user
do".
Note: All sudo user access will be tracked

and logged in the /var/log/secure file.
Root ssh access should be disabled on the

Oracle VM Manager host. Sudo should
should be used to configure fine-grained
permissions to allow administrative users
to perform selected administrative tasks
with logging.
Disables Root Access:

To disable root ssh access, edit the default
/etc/ssh/sshd_config file and uncomment
the
the #PermitRootLogin yes line and
change the yes to no; that is,
PermitRootLogin no. Next, restart the
sshd service by typing service sshd
restart to enable the change.
The visudo command is used to edit the

/etc/sudoers file.Consult the sudoers man
page for sudo configuration details.
SSH login banners
Call 415-252-9164
SSH login banners presents a definitive

warning or disclaimer to all users that
wish to access your systems using SSH.
SSH login banners should clarify which
types of activities are illegal as well as
advise legitimate users of their obligations
relating to the acceptable use of the
system.
Pre and post SSH login banners should be

configured on each Oracle VM Manager
host.
Pre-login banner:
Edit the /etc/ssh/sshd_config and add the
following directive:
Banner /etc/banner.net
Next, create the /etc/banner.net file and add
To be able to successfully prosecute

individuals who improperly use a
computer, the computer must have a
warning banner displayed at all access
points.
Call 415-252-9164
your login banner text, i.e.
This system is restricted to authorized

access only. All activities on this system
are recorded and logged. Unauthorized
access will be fully investigated and
reported to the appropriate law
enforcement agencies.
Once the file has been created and the

banner text is added and saved, restart the
sshd by typing:
# service sshd restart
Post login banner:

Edit /etc/motd and add your login banner
text, i.e.

Once the file has been edited and saved,

restart the sshd by typing:
Log file analytics
Log file analytics solutions can beused to

collect and centrallyanalize securityrelevant or operations-relevant events.
Log file analyticssimplifies security

management for the detection of attacks
and intrusions.
This table outlines the decision points for Oracle VM server security controls. For decisions that rely on preexisting
factors or specific organizational needs, the appropriate best practice will be discovered in the infrastructure
assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and decisions should be
made based on organizational needs, existing architecture, and budget resource availability.
Oracle VM Server Security Controls

Decision Point
Decision
Justification
Oracle VM Server
and DMZs
Oracle VM Servers hosting Internet facing

virtual machines can be placed in a DMZ
Oracle VM Servers in a DMZ should be

restricted from inbound and outbound
Call 415-252-9164
Call 415-252-9164
without connectivity to the Internet or

internal network segmentsto reduce the
attack surface.
Internet connectivity toreduce the attack

surface.
Build Process
Before any Oracle VM servers are placed

on the production network, a standard
build processes should be executed to
ensure that all Oracle VM servers are
installed, configured and maintained in a
manner that prevents unauthorized access,
unauthorized use and disruptions in
service.
An Oracle VM server build document

provides employees with an approved
procedure to install and configure Oracle
VM server. An Oracle VM server build
document is used with other IT
infrastructure policies to address
interoperability and security of Oracle VM
in the context of the entire information
system.
Patch Management
A key component of a successful Oracle

VM deployment is acquiring and vetting
new releases, patches and updates for
production systems. New Oracle VM
releases, patches and updates must be
researched to identify which release,
patches and updates are applicable to your
environment. Newly released versions,
patches and updates should be vetted
before being deployed into production.
Oracle VM Servers should be configured

to use local custom yum
repositories.Local yum repositories with
point-in-time static channel for each
supported Oracle VM release ensures all
likeOracle VM serverare patched in a
consistent manner across the organization.
All patches should be regression tested in

the lab environment before they are
deployed on production systems.Highpriority patches, security fixes, and
upgrades will be applied as needed in
accordance with your Change
Management Policy.
All production systems should undergo

security audits in accordance with your
Change Management Policy to validate
configuration and patch compliance.
A patch management program is an

integral component of an organization's
information security program used to
mitigate the risk from security
vulnerabilities (bugs) that are inherent in
all operating systems and applications.
Call 415-252-9164
Call 415-252-9164
Host firewall
The iptables service is installed by default

on Oracle VM servers, although it is
disabled to eliminate the high CPU load
that is created by implementing packet
filter rules. In earlier Oracle VM server
releases iptables was enabled and it
dramatically limited the number of virtual
machine that could be run on a host, and
regularly caused servers to crash.
A best practice is to use firewall devices,

not host firewalls,to filter network traffic
to Oracle VMservers.
Root ssh access and

sudo
By default, Oracle VM server permit ssh

access using the root super user account.
Systems administrators should access the

Oracle VM serverswithnon-root
individual user accountsand use sudoto
perform selected administrative
tasks.Sudo stands for either "substitute
user do" or "super user do".
Root ssh access should be disabled on the

Oracle VM servers. Sudo should should
be used to configure fine-grained
permissions to allow administrative users
to perform selected administrative tasks
with logging.
Disables Root Access:

To disable root ssh access, edit the default
/etc/ssh/sshd_config file and uncomment
the
the #PermitRootLogin yes line and
change the yes to no; that is,
PermitRootLogin no. Next, restart the
sshd service by typing service sshd
restart to enable the change.
The visudo command is used to edit the

/etc/sudoers file.Consult the sudoers man
page for sudo configuration details.
One of the most important security

measure that can be taken with Oracle VM
is to prevent unauthorized access to the
root user account by disabling root ssh
access. A best practice is toonly
allownon-root individual user accounts
accessthat can be audited, disabled,
expired and managed using sudo.
Note:All sudo user access will be tracked

and logged in the /var/log/secure file.
SSH login banners
SSH login banners presents a definitive

warning or disclaimer to all users that
wish to access your systems using SSH.
SSH login banners should clarify which
types of activities are illegal as well as
advise legitimate users of their obligations
relating to the acceptable use of the
system.
To be able to successfully prosecute

individuals who improperly use a
computer, the computer must have a
warning banner displayed at all access
points.
Call 415-252-9164
Call 415-252-9164
Pre and post SSH login banners should be

configured on each Oracle VM server.
Pre-login banner:
Edit the /etc/ssh/sshd_config and add the
following directive:
Banner /etc/banner.net
Next, create the /etc/banner.net file and add

your login banner text, i.e.

Once the file has been created and the

banner text is added and saved, restart the
sshd by typing:
Post login banner:

Edit /etc/motd and add your login banner
text, i.e.

Once the file has been edited and saved,

restart the sshd by typing:
Rootkit prevention
and monitoring
Call 415-252-9164
Wikipedia describes a rootkit as A rootkit

is software that enables continued
privileged access to a computer while
actively hiding its presence from
administrators by subverting standard
operating system functionality or other
applications.
Hypervisors like Oracle VM servermay be

Monitoring the Oracle VM serversfor

rootkits is fundamental part of information
security used to detect rootkits to prevent
attacks and intrusions. Each Oracle VM
server should have a rootkit prevention in
place, such as chkrootkit, that monitors the
host for rootkits.
Call 415-252-9164
one of the most sensitive operating

systems in the data center because it
controls the server hardware and the
virtual machines with Oracle workloads
on them. If the hypervisor is compromised
direct access to the hardware and all of the
virtual machines and Oracle workloads is
possible, and other code could be
monitored and controlled by the attacker.
Log file analytics
Log file analytics solutions can beused to

collect and centrallyanalizesecurityrelevant or operations-relevant events.
Log file analyticssimplifies security

management for the detection of attacks
and intrusions.
Virtual Machine Operating System Standards

This table outlines the decision points for thefor virtual machine operating systems hosted on Oracle VM. For
decisions that rely on preexisting factors or specific organizational needs, the appropriate best practice will be
discovered in the infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed
carefully and decisions should be made based on organizational needs, existing architecture, and budget resource
availability.
Decision Point
Decision
Justification
Virtualization Mode
- HVM vs PVHVM
vs PVM
Oracle VM supports two unique

virtualization modes, paravirtualization
mode (PVM mode) and hardware
virtualization mode, HVM. PVHVM is
also an available virtualization mode that
consists of HVM mode with PVM drivers.
Windows and Solaris x86 only support
HVM and PVHVM modes, Linux
supports PVM/HVM and PVHVM. Oracle
VM Servers can support both PVM mode
and HVM/PVHVM mode simultaneously
on a single x86_64 server that has either
Intel or AMD virtualization technologies.
Even with contemporaryLinux kernels and

driversPVM mode still hassignificantly
less overhead for timers, interrupts, I/O
traffic, and context switches, allowing
superior scalability under heavy loads,
when compared to all of the hardware
virtualization modes.
Virtual Machine
Operating Systems
A small number of virtual machine

operating systems should be be used. For
example, stadardizing on Oracle Linux 6
latest, in contract to supporting Oracle
Linux 5U2, 5U3, 5U4, 5U5, 5U6, 5U6,
5U8, 5U9, 5U10, 6, 6U1, 6U2, 6U3, 6U4,
etc...
Standardizing on a small number of virtual

machine operating systems streamlines
operations andreduces complexity by
limiting the number of supported operating
systems.
Virtual Machine
Each application has an operating system
In accordance with your Application
Call 415-252-9164
Call 415-252-9164
Operating System
Versioning
support matrix that lists the supported

operating systems, patch levels, and
software prerequisites.
Software Policy and Application Software

Standards, applications will determine the
operating system type and version.
Virtual Machine
Operating System
Deployments
All new virtual machine operating systems

should be deployed using a virtual
machine template or automated build
processin accordance with your Server
Policy, and Server Security Policy.
A virtual machine template is a selfcontained, preconfigured virtual machine

with an operating system and optionally an
application installed using corporate
standards.Each time a new virtual
machine is deployed using a virtual
machine template, standards are applied to
each new virtual machine.
Virtual Machine
Disk Partition
Alignment (OCFS2
Storage Only)
With Oracle VM, improper virtual

machine disk partition alignment only
affects virtual disk images stored on
OCFS2 repositories and does not apply to
NFS repositories.
Classic hard disks have a block size of 512

bytes. Depending on the guest operating
system of the virtual machine (Linux,
Windows or Solaris), one logical block of
the guest file system can use two blocks
on the storage. This is known as block
misalignment.
It is best to avoid block misalignment

because it doubles the IO on the storage to
access a block of the guest OS file system
(assuming a complete random access
pattern and no caching).All partitions on
OCFS2 repositories should be created
using an alignment scheme with multiples
of 8 (512 byte sectors), which aligns them
at 4KB, the OCFS2 block size, or retire
the VM with a new OCFS2 friendly OS.
Check for misaligned partitions in the

virtual disk images and, if any are found,
recreate the disk images with properly
aligned partitions. To determine whether
partitions are correctly aligned, use fdisk lu to find the starting sectors of the
partitions. Ensure that these are a multiple
of 8 (512 byte sectors), which aligns them
at 4KB, the OCFS2 block size.
Oracle Database
Files
High I/O disks such as Oracle database

files should be placed on RAW disks.
The virtual machinefront-end storage

consistsof multiple guest storageoptions
such asfile and Raw Device Mappings.
Raw Device Mapping of SAN LUNs to

individual guests for data/database files
have the absolute best performance of the
two front-end storage storage options. In
most cases, Raw Device Mappingsare the
only option for high I/O workloads like
Oracle databases.
Patch Management
A key component of patch management is

acquiring and vetting patches for
production systems. Patches must be
researched to identify which patches,

Call 415-252-9164
Call 415-252-9164
security fixes, and application updates are

applicable to your environment. Newly
released patches, security updates, and
application updates will be tested before
being deployed in to production using
time stamped local custom repositories.

All patches shouldbe regression tested in

application upgrades updates should be
applied as needed in accordance with your
Change Management Policy.Noncritical
fixes should be appliedin accordance with
yourChange Management Policy.
With Linux, local yum repositories should

be maintained for patch testing and
production using a point-in-time static
channel for each supported operating
system to ensure all like operating systems
are patched in a consistent manner across
the organization.
All production systems shouldundergo

security audits in accordance with your
Change Management Policy to validate
configuration and patch compliance.
Oracle VM Application Integration

This table outlines the decision points for an Oracle VM hosted application integration. For decisions that rely on
preexisting factors or specific organizational needs, the appropriate best practice will be discovered in the
infrastructure assessment (IA) and gap analysis (GA). The best practices should be analyzed carefully and
Decision Point
Decision
Justification
Application Support
Applications should be jointly supported

by the independent software vendor
(ISV)and Oracle for the deployed version
of Oracle VM.
Only jointly supported applications receive

vendor support when problems occur and
service tickets are created. The
applications hosted on Oracle VM
shouldbe jointly supported by the
independent software vendorand Oracle.
Application
Requirements and
Applications should be analyzed for

requirements and dependencies.
To reduce risk, and shorten implementation

time,applications should beanalyzed for
Call 415-252-9164
Call 415-252-9164
Dependencies
requirements and dependencies, then

deployed and tested on Oracle VM in
accordance with yourSoftware Installation
Standards.
Application
Installations
Applications should be installed using

vendor best practices.
One of the top application deficiencies we

encounter during health checks is Oracle
application installation inconsistencies
resulting in nonstandard installations,
excessive deployment errors, and high
support costs.
Applications should be installed following

vendor best practicesand Software
Installation Standards. We recommend
installing Oracle codeusing silent
installations to standardize the installation
routine, reduce deployment errors, and
minimize support costs.
Application
sunsetting
Applications that have reached the end of

their life cycle and are no longer supported
by a vendor should be given a sunset date.
The sunset date is when the product is
scheduled to be removed from
production.
Sunsetting applications that have reached

the end of their life cycle results in better
customer service and reduced costs.
Applications should be sunsetted in

accordance with yourSoftware Sunset
Policy.
Patch Management
A key component of patch management is

acquiring and vetting patches for
production systems. Patches must be
researched to identify which patches,
security fixes, and application updates are
applicable to your environment. Newly
released patches, security updates, and
application updates will be tested before
being deployed in to production using
time stamped local custom repositories.
All patches shouldbe regression tested in

application upgrades updates should be
applied as needed in accordance with your
Change Management Policy.Noncritical
fixes should be appliedin accordance with
yourChange Management Policy.

Call 415-252-9164
Call 415-252-9164
Call 415-252-9164

Oracle Cloud Reference Design-08!28!2015

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Oracle Cloud Reference Design-08!28!2015

Hochgeladen von

Copyright:

Verfügbare Formate

Call 1 (415) 252-9164

Mokum Oracle Cloud Cookbook

Document Created: 09/30/13

Mokum Solutions, Inc.

Mokum Oracle Cloud Cookbook

Mokum Oracle Private Cloud Reference Design Introduction

Oracle VM for x86

Oracle Enterprise Manager

Storage & Network

Mokum Oracle Private Cloud Reference Design Implementation Overview

Mokum Solutions, Inc.

Mokum Oracle Cloud Cookbook

Mokum Oracle Private Cloud Reference Design Support Infrastructure

Oracle Support Agreements for the Oracle

Support is an integral part of every

On-site and Off-site

On-site and off-site support from the

On-site and off-site support from the

Oracle Private Cloud Architectural Design

Oracle VM Hardware Architecture

Mokum Solutions, Inc.

Mokum Oracle Cloud Cookbook

Mokum Solutions, Inc.

Mokum Oracle Cloud Cookbook

Oracle VM Hardware Sizing

Mokum Solutions, Inc.

Mokum Oracle Cloud Cookbook

The below image showsan Oracle

Mokum Solutions, Inc.

The below image showsan Oracle

Mokum Oracle Cloud Cookbook

CPU Cores or Threads

Mokum Solutions, Inc.

Mokum Oracle Cloud Cookbook

Mokum Solutions, Inc.

Mokum Oracle Cloud Cookbook

3DL 380s each 56 vCPUs

Mokum Solutions, Inc.

Mokum Oracle Cloud Cookbook

The server hardware should be jointly

Only jointly supported hardware product

Mokum Solutions, Inc.

Mokum Oracle Cloud Cookbook

Note: The following link is the Oracle'

by the hardware vendor and Oracle.

Server hardware should be ordered with at

The Maximum Number of CPU cores or

Oracle VM Server supportsCPU

CPU-bound workloads, such as Oracle

Server hardware should be ordered with

Note: Oracle VM Release 3.3supports up

Oracle VM Server does not support

An Oracle VM server in a server pool with

Mokum Solutions, Inc.

Mokum Oracle Cloud Cookbook

DRS, DPM and/or HA operation. Oracle

Unless the Oracle VM server is booting

A minimum of one Ethernet network

Mokum Solutions, Inc.

Oracle VM Server requires 4GB of local

Oracle VM local storage does not support

Oracle VM uses a total of five discrete

Each Oracle VM server pool should have a

Mokum Oracle Cloud Cookbook