COLLABORATE 14 IOUG Forum

Database

Oracle RAC 12c and Policy-Managed Databases, a Technical

Overview
Ludovico Caldara, Trivadis AG

ABSTRACT
Oracle RAC Policy-Managed Database (PMD) is a powerful but so far rarely used feature introduced in Oracle Database 11g
Release 2 to automate the instance administration in a dynamic, multi-node cluster.
The aim of this presentation is to review how PMD works, how to implement and administer it successfully, and how to
benefit from this technology compared to the traditional administrator-managed deployment. During the session, the new
features of the 12c Grid Infrastructure related to PMD are highlighted.

TARGET AUDIENCE
This whitepaper is targeted at database administrators and architects who are familiar with Oracle Real Application
Cluster concepts and want to know more about Policy-Managed Databases.

EXECUTIVE SUMMARY
After reading this white-paper, you should be able to:
Evaluate the features of PMD
Describe how to plan and administer the solution
Identify the reasons to migrate to a PMD deployment

1 | P a g e O R A CL E 1 2 C D A T A G U AR D : F A R S Y N C , R E AL - T I M E C A S C AD E S T A N D B Y AN D O T H E R G O O D I E S
W h it e Pa p e r

COLLABORATE 14 IOUG Forum

Database

ADMIN VS POLICY MANAGEMENT

Oracle RAC admin-managed databases are so far the most common type of configuration used by Oracle customers to
administer databases, probably because the behavior is unchanged from Oracle 10g onwards and the DBAs feel more
confident with it. Admin-Managed databases provide high performance, high availability and scalability; however the process
of creating and configuring new instances is not completely straightforward: DBAs have to take care of creating the directory
structures, create the undo tablespaces dedicated to the instances, create the new redo threads and configure the instances to
be managed by the cluster:
srvctl add instance db dbname instance sid node server
There is a strong relationship between the instance and the server, so the DBAs must plan which instances will run on which
servers and prepare everything in advance; but where new nodes are added to the cluster, things become even more complex.
Policy-Managed databases dont have preferred servers. The databases are rather assigned to a pool of servers whose size can
change dynamically.

$ srvctl modify serverpool serverpool hr min 4

Instead of adding new instances to the database, the DBAs can just expand the server pool in order to have the database to
expand with it. The instance configuration is totally delegated to the cluster that will automatically add UNDO tablespaces and
redo logs as needed (provided that ASM or Oracle-Managed Files are in place), allowing having clusters that can grow or
shrink dynamically. This new behavior is an enabler for ease of consolidation and rapid provisioning, thus making a step
forward in the direction of elastic database clouds.
Another important improvement introduced by policy-managed databases resides in the resiliency of the clusters.
Imagine that you have two different databases, HR and CRM, the latter with higher importance in your company. If you lose
one node belonging to CRM database, with admin-managed databases you should do a manual intervention: stop one HR
instance and start another CRM instance, provided that it has been previously configured to run on the surviving server. If its
not the case, configuring from scratch the new instance can be time consuming and error prone, which is not recommended
during a resource shortage.

2 | P a g e O R A CL E 1 2 C D A T A G U A R D : F AR S YN C , R E AL - T I M E C A S C AD E S T A N D B Y AN D O T H ER G O O D I E S
W h it e Pa p e r

COLLABORATE 14 IOUG Forum

Database

Policy-Managed Databases solve this kind of problem automatically: DBAs can assign priorities to server pools and let them
resize automatically in order to meet resource requirements of most important databases.

Theres no requirement for an immediate DBA intervention (well, someone should at least solve the node failure!): theres a
higher level of availability for the higher prioritized databases.

SERVER POOLS
Server pools are the foundation of policy managed databases. They are used to partition a cluster into smaller groups of
servers. Because databases are assigned to server pools, their size directly affects the resources available to the databases.
There are always two predefined server pools. The Free server pool consists of new or unassigned nodes, whereas the
Generic server pool consists of servers running admin-managed databases.

Policy-managed databases cannot be assigned to the Generic server pool and the instances of admin-managed databases
cannot run in server pools other than the Generic. Running policy-managed databases requires then the creation of additional
server pools.

3 | P a g e O R A CL E 1 2 C D A T A G U A R D : F AR S YN C , R E AL - T I M E C A S C AD E S T A N D B Y AN D O T H ER G O O D I E S
W h it e Pa p e r

COLLABORATE 14 IOUG Forum

Database

You can have both admin-managed and policy-managed databases into the same cluster. This may be necessary in order to
consolidate pre-11gR2 admin-managed databases that cannot be policy-managed.
The conversion from admin-managed to policy-managed databases is achieved by:
Having an existing user-defined server pool, or creating a new one:
$ srvctl add srvpool -serverpool hr -min 1 -max 2 -importance 10

Moving the database to the server pool:

$ srvctl modify database db hr -serverpool hr
The clusterware will stop the database and restart it as policy-managed. Sadly, theres no way to convert it without service
disruption. If you have just two nodes (e.g. you want to convert an Oracle Database Appliance to policy-managed), you can
migrate by following these steps:
free up a node for usage in the new pool by stopping one instance and removing it from the cluster (srvctl
remove instance)
create the server pool
move the database to the newly created server pool (db is converted to policy-managed)
remove the last admin-managed instance from the cluster (that should be done automatically)

SERVER ASSIGNMENT
The size of server pools is driven by three main attributes:
MIN_SIZE: The minimum number of servers that compose the server pool
MAX_SIZE: The maximum number of servers
IMPORTANCE: The priority level (from 0 to 1000, higher numbers means higher priority)

The cluster uses these attributes when it has to decide for server assignment. Free and Generic server pools have by default all
attributes set to 0.
When the cluster starts:
The cluster starts by assigning servers to the most important server pools, up to MIN_SIZE:

4 | P a g e O R A CL E 1 2 C D A T A G U A R D : F AR S YN C , R E AL - T I M E C A S C AD E S T A N D B Y AN D O T H ER G O O D I E S
W h it e Pa p e r

COLLABORATE 14 IOUG Forum

Database

Once the first servers are available to the server pool, the corresponding databases are also started. This allows
DBAs to start most important databases first.
Then, in descending order of importance, the other server pool minimum sizes are fulfilled

When all minimum sizes are met, the same is done for the maximum sizes.

When all maximum sizes are met, eventual spare servers will belong to the Free server pool.

When a node fails or is removed from the cluster:

If a node fails in a server pool, but its size is still greater than its MIN_SIZE attribute, no action is taken, no
matter what the importance is.
If a node fails in a server pool causing its size to fall below MIN_SIZE, the cluster draws a server from a server
pool with more than MIN_SIZE servers, in ascending order of importance. This can actually be a server pool
with an importance higher than the failed server pool, because the MIN_SIZE will be still fulfilled.

If a node fails in a server pool and its size falls below MIN_SIZE but all server pool sizes are already equal or
below their respective MIN_SIZE, the server is taken from the pool with the lowest importance. This can lead to
having server pools with low importance and no available servers (thus, offline databases) and server pools with
higher importance with more than one server available. This is ok, but it is important that you consider it when
configuring your parameters.

5 | P a g e O R A CL E 1 2 C D A T A G U A R D : F AR S YN C , R E AL - T I M E C A S C AD E S T A N D B Y AN D O T H ER G O O D I E S
W h it e Pa p e r

COLLABORATE 14 IOUG Forum

Database

When a node is added or a server pool is modified, the cluster will recalculate and reallocate servers in order to satisfy the
requirements, using the same rules described above.
When a cluster is upgraded, the upgrade process will move resources gracefully and automatically in order to satisfy all the
requirements. E.g., the upgrade of a cluster from 11gR2 to 12c will let you organize the execution of the root.sh script in
batches that will stop and upgrade up to three nodes at time. With policy-managed databases, it doesnt matter how you group
servers in batches, because when the CRS is stopped on the nodes, the server pools size change depending on the rules
defined by the DBAs. This greatly increases the online upgrade capabilities of the cluster.

WHAT-IF CONDITIONS
Oracle Database 12c comes with a new feature called What-If that is handy to evaluate the result of a command without
actually executing it. You should use it whenever you have to modify server pool attributes and youre unsure of how your
services and databases will react.
When using crsctl, there is an eval command that triggers the evaluation:
$ crsctl eval modify serverpool ora.crm -attr "MIN_SIZE=4,MAX_SIZE=4" f
[]
Resource 'ora.hr.payroll.svc' (1/1) will be in
state [OFFLINE]
Server 'rac4' will be moved from pools [ora.hr]
to pools [ora.crm]
Resource 'ora.hr.db' (2/1) will be in state [OFFLINE]
When using srvctl, you can use the -eval flag:
$ srvctl add serverpool -serverpool bi -min 5 -max 5 -importance 10 eval
Service payroll will be stopped on node rac4
Database hr will be stopped on node rac4
Server rac4 will be moved from pool hr to pool bi

NESTED SERVER POOLS

Server pools can be nested; you can create child pools with crsctl by setting the attribute PARENT_POOLS. Child pools
creation using srvctl is not possible.
However, using nested server pools in conjunction with policy-managed database doesnt work as expected and its not fully
documented by Oracle. Because of that I do not recommend their usage.

6 | P a g e O R A CL E 1 2 C D A T A G U A R D : F AR S YN C , R E AL - T I M E C A S C AD E S T A N D B Y AN D O T H ER G O O D I E S
W h it e Pa p e r

COLLABORATE 14 IOUG Forum

Database

POLICY SETS
Sometimes Oracle customers need to have different server pool configurations at different times, depending on how workload
changes over time. As an example, a customer may want to have more servers (thus more resources) for the server pool HR
once a month in order to process payrolls and more servers for the server pool DWH for nightly reports.
Prior to version 12c, the only way to change the workload to fit the requirements was to execute several srvctl modify
serverpool. Oracle Database 12c has introduced the policy sets, a feature that allows preparing in advance all the different
workload scenarios and activating them with a single command.
The easiest way to create policy sets is dumping the current configuration into a file, edit it, and reload it into the cluster.
The following command will create a file containing the current configuration:
$ crsctl create policyset -file cluster_policies.txt
This is an example of content:
SERVER_POOL_NAMES=Free ora.dwh ora.hr
POLICY
NAME=Default
SERVERPOOL
NAME=ora.dwh
IMPORTANCE=5
MAX_SIZE=2
MIN_SIZE=1
SERVER_CATEGORY=ora.hub.category
SERVERPOOL
NAME=ora.hr
IMPORTANCE=8
MAX_SIZE=4
MIN_SIZE=3
SERVER_CATEGORY=ora.hub.category

You can append new policies to the files. As an example:

POLICY
NAME=monthly_payroll
SERVERPOOL
NAME=ora.hr
IMPORTANCE=10
MIN_SIZE=3
MAX_SIZE=3
SERVERPOOL
NAME=ora.dwh
IMPORTANCE=5
MIN_SIZE=1
MAX_SIZE=1
POLICY
NAME=nightly_reports
SERVERPOOL
7 | P a g e O R A CL E 12 C D A T A G U AR D : F AR S Y N C , R E AL - T I M E C A S C AD E S T A N D B Y AN D O T H ER G O O D I E S
W h it e Pa p e r

COLLABORATE 14 IOUG Forum

Database

NAME=ora.hr
IMPORTANCE=5
MIN_SIZE=1
MAX_SIZE=1
SERVERPOOL
NAME=ora.dwh
IMPORTANCE=5
MIN_SIZE=3
MAX_SIZE=4

Then you must reload the file by using the modify policyset command:
$ crsctl modify policyset -file cluster_policies.txt
The new policies can be activated by modifying the LAST_ACTIVATED_POLICY attribute of the policyset:
$ crsctl modify policyset attr "LAST_ACTIVATED_POLICY=monthly_payroll"
The command crsctl eval modify policyset is not implemented yet (as of release 12.1.0.1.2), you cannot benefit from it if you
want to do a what-if analysis prior to switch the active policy.
Theres no way to switch policies automatically from the Oracle side, policies are not integrated in the resource manager, you
have to script the switch and run it from a scheduler. The same kind of functionality however comes with the Quality of
Services Management product that complements policy-managed databases and provides inter-database resource management.

DATABASE BEHAVIOR
With admin-managed databases, as weve seen, you must set manually which instances run on which servers. This is
somehow an additional effort on one side, but on the other it allows DBAs to choose how many instances can run for each
database.

Policy-managed databases however span the entire server pool, so the number of instances is always equal to the number of
servers composing the server pool, and it is the same for each database in the server pool.

8 | P a g e O R A CL E 1 2 C D A T A G U A R D : F AR S YN C , R E AL - T I M E C A S C AD E S T A N D B Y AN D O T H ER G O O D I E S
W h it e Pa p e r

COLLABORATE 14 IOUG Forum

Database

The total number of instances is the result of (#DATABASES * #(SERVERS in the pool)). This can lead to instance
proliferation and the physical memory of each server can limit the number of databases running in the server pool
independently from the pool size. To work around this problem, you can rely on RAC One Node feature for less critical
databases, this can dramatically increase the number of running databases but the overcome is decreased high availability:

I recommend having a mix of both RAC and RAC One Node databases in the cluster in order to host both critical and noncritical databases.

When consolidating many database instances on the same server, make sure you use instance caging (CPU_COUNT) to
prevent one instance from consuming all the CPU of the server. In order to manage also other kind of resources, Oracle
recommends using Quality of Services management feature. Although QOS is quite complex and almost unfamiliar, its worth
to consider it for big consolidation projects.
Another viable way to reduce the instance footprint and manage resources between databases is to implement Oracle
Multitenant. The resources available to the different PDBs can be managed through the new PDB share management of
Resource Manager 12c.

9 | P a g e O R A CL E 1 2 C D A T A G U A R D : F AR S YN C , R E AL - T I M E C A S C AD E S T A N D B Y AN D O T H ER G O O D I E S
W h it e Pa p e r

COLLABORATE 14 IOUG Forum

Database

SERVICE BEHAVIOR
Like the databases, policy-managed services can either span the entire server pool or run on a single instance. You control the
behavior by specifying either UNIFORM or SINGLETON cardinality:
$ srvctl add service db CRM serverpool mixed service REPORTING cardinality SINGLETON
$ srvctl add service db CRM serverpool mixed service FIN_READ cardinality SINGLETON
$ srvctl add service db CRM serverpool mixed service SALES cardinality UNIFORM

There is no way to predict on which instance a SINGLETON service will run and its no more possible to specify the
preferred instance upon the service creation.

PRACTICAL TIPS
DONT MONITOR INSTANCES WITH CRSCTL
With policy-managed databases, the size of server pools and databases can expand or shrink depending on several factors, so it
is expected that in some situations instances shut down to honor a server pool shrink. After an instance shuts down, however,
the command crsctl will still show it, inducing the DBA to think that an instance is unexpectedly offline:
# crsctl status resource ora.CRM.db -t
---------------------------------------------------------------------Name
Target State
Server
State details
---------------------------------------------------------------------ora.CRM.db
1
ONLINE ONLINE
rac4
Open,STABLE
2
ONLINE ONLINE
rac3
Open,STABLE
3
ONLINE OFFLINE
STABLE
If you need to monitor the database instances status, you should rely rather on srvctl and count the number of instances
currently running and expect they dont fall below the MIN_SIZE you have set.:
# srvctl status database -db CRM
Instance CRM_1 is running on node node4
Instance CRM_2 is running on node node3

CENTRALIZE THE DIAGNOSTIC_DEST

Server maintenance, cluster restarts and other activities will make instances moving around and change often the server they
are running on. If you rely on local filesystems for your diagnostic_dest you may not be able to find which alert file on which
server actually contains the log information you need.

10 | P a g e O R A CL E 1 2 C D A T A G U AR D : F A R S Y N C , R E AL - T I M E C A S C AD E S T A N D B Y AN D O T H E R G O O D I E S
W h it e Pa p e r

COLLABORATE 14 IOUG Forum

Database

Its a good idea to point the diagnostic_dest to a common filesystem, ideally ACFS if its supported by your platform. Dont
forget to register the ACFS filesystem into the cluster configuration as a dependency of your databases.

MY CONCLUSION
Policy-management is an interesting piece of software. Its more resistant to server failures than the traditional admin-managed
databases; it allows very fast instance provisioning and has a proven agility and scaling. The administration becomes easier and
many tasks like upgrades and migrations are easier to accomplish. As of 12.1.0.1.1, policy-management is also a requirement
for Quality of Services management and Global Data Services. As a general suggestion, I recommend it for every RAC-based
database consolidation project.
There are however some fallbacks that require attention. The limited cardinality choice of services and databases (UNIFORM
or SINGLETON) doesnt allow manual, strategic instance placement. The Enterprise Manager, and the crsctl command itself,
are not always as reactive as one would expect when new instances appear, relocate or disappear. Some features dont work
together, in particular child pools and resource customization with crsctl may cause configuration errors that are sometimes
difficult to solve without service disruption.

REFERENCES

Oracle Clusterware Administration and Deployment Guide 12c Release 1 (12.1)

http://docs.oracle.com/cd/E16655_01/rac.121/e17886/toc.htm

Oracle Real Application Clusters Administration and Deployment Guide 12c Release 1 (12.1)
http://docs.oracle.com/cd/E16655_01/rac.121/e17887/toc.htm

Oracle Database 2 Day + Real Application Clusters Guide 12c Release 1 (12.1)
http://docs.oracle.com/cd/E16655_01/rac.121/e17616/toc.htm

11 | P a g e O R A C L E 1 2 C D A T A G U AR D : F A R SY N C , R E A L- T I M E C A S C A D E ST A N D B Y A N D O TH E R G O O D I ES
W h it e Pa p e r