Scalability of SAP utilizing Oracle Real

Application Clusters on IBM Power

. . . .Servers
. . . .

Joerg Droste
IBM Systems ISV Enablement
August 2009

© Copyright IBM Corporation, 2009. All Rights Reserved.

All trademarks or registered trademarks mentioned herein are the property of their respective holders
 Table of contents

Abstract........................................................................................................................................1
Introduction .................................................................................................................................1
Oracle Real Application Clusters ..............................................................................................1
The SAP Business Suite.............................................................................................................2
Running SAP applications with Oracle RAC ............................................................................2
Benefits of running SAP with Oracle RAC on IBM Power Servers ........................................................ 2
Configuration Overview .......................................................................................................................... 3
Hardware Configuration .................................................................................................... 3
Software Configuration...................................................................................................... 3
Implementation .................................................................................................................. 4
Performance Results ..................................................................................................................5
Scalability Aspects .....................................................................................................................6
Performance Considerations .....................................................................................................7
Hardware related Considerations........................................................................................................... 7
Operating System related Considerations ............................................................................................. 8
SAP related Considerations ................................................................................................................... 8
Oracle related Considerations................................................................................................................ 9
Network ............................................................................................................................. 9
Partitioning......................................................................................................................... 9
Summary......................................................................................................................................9
Resources..................................................................................................................................11
About the author .......................................................................................................................11
Acknowledgements ..................................................................................................................11
Trademarks and special notices..............................................................................................12

Scalability of SAP utilizing Oracle Real Application Clusters on IBM Power Systems
Abstract
This paper will demonstrate near-linear scalability of the SAP Sales and Distribution
benchmark when running on IBM System 570 Servers and IBM TotalStorage DS8000
when connected by an Oracle Real Application Cluster. It will document the scalability
of the environment when adding additional servers and provide considerations for the
implementation of this environment.

Introduction
Oracle Real Application Clusters (RAC) is an option to the award-winning Oracle Database
Enterprise Edition. Oracle RAC is a cluster database with a shared cache architecture that
overcomes the limitations of traditional shared-nothing and shared-disk approaches to provide a
highly scalable and available database solution for SAP business applications.

This document will demonstrate that the SAP software suite works and scales very well utilizing
multiple server nodes in a Oracle RAC cluster running on IBM Power 570 servers and IBM
TotalStorage DS8300 hardware.

Oracle Real Application Clusters

Oracle’s Real Application Clusters (RAC) option supports the transparent deployment of a
single database across a cluster of servers, providing fault tolerance from hardware failures or
planned outages.

Oracle Database 10g includes Oracle Clusterware, a complete, integrated clusterware

management solution. This clusterware functionality includes mechanisms for cluster
messaging, locking, failure detection, and recovery. Oracle Clusterware also provides a
complete and integrated SAP High Availability solution for the SAP Engueue and Message
Server.

Oracle RAC provides high availability for SAP applications by removing the single point of
failure with a single database server. If a node in the cluster fails, the Oracle Database
continues running on the remaining nodes. Individual nodes can be shutdown for maintenance
while SAP application users continue to work.

Oracle Real Application Clusters provide flexibility for scaling SAP applications. To keep costs
low, clusters can be built from standardized, commodity-priced processing, storage, and
network components. When you need more processing power, simply add another server
Scalability of SAP utilizing Oracle Real Application Clusters on IBM Power Systems

1
without taking users offline to gain horizontal scalability. Oracle Clusterware and Oracle RAC
support up to 100 nodes in the cluster.

The SAP Business Suite

Businesses large and small have discovered that SAP Business Suite is the world's most
comprehensive family of adaptive business applications, providing best-of-breed functionality
built for complete integration, industry-specific functionality, unlimited scalability, and easy
collaboration over the Internet.

Individually, SAP Business Suite applications help to manage the most critical business
processes. Collectively, they form a tightly integrated suite that adds value to every facet of the
organization – and the external value chain.

SAP Business Suite applications are based on the SAP NetWeaver platform, an integration and
application platform. This reduces total cost of ownership across the entire IT landscape and
supports the evolution of SAP Business Suite to a services-based architecture.

Running SAP applications with Oracle RAC

SAP AG has supported the Oracle RAC solution for many years and has integrated this solution
into the generic offering structure providing standardized tools for monitoring and configuration.
This document will focus on performance and scalability aspects of this solution documenting
the outstanding value of running SAP applications in an Oracle RAC clustered environment on
IBM Power hardware.

Benefits of running SAP with Oracle RAC on IBM Power Servers

There are multiple benefits to the customer when implementing Oracle RAC with SAP
applications on IBM Power servers:

• High Availability
The database for the SAP installation is distributed across multiple servers eliminating a
single point of failure in the hardware.

• Asset Protection
Individual database servers can be taken offline for maintenance and repair without service
interruption. Users continue to work during these periods.

Scalability of SAP utilizing Oracle Real Application Clusters on IBM Power Systems

2
• Scalability
An Oracle RAC database can scale very well over many nodes in a cluster providing
horizontal scalability to the application database. Each server node adds its full capacity with
only a small management overhead.

• Performance
The industry leading performance of IBM Power servers provides outstanding vertical
scalability and processing power to Oracle databases and SAP applications. IBM
TotalStorage subsystems provide incredible throughput rates and enable the whole
application stack to be exploited to its full potential.

• Reliability
Industry proven stability of SAP and Oracle software are complemented by IBMs
commitment to provide the most robust server and storage platform in the industry.

Configuration Overview

Hardware Configuration
The hardware configuration used in the following benchmark publications consisted of up to 5
IBM Power 570 servers. Each of them configured with:

• 8 processors / 16 cores / 32 threads

• 128GB Memory

• 4 Gigabit network adapters

• 2 fiberchannel I/O adapters

An IBM TotalStorage 8000 storage subsystem was used to provide leading performance,
reliability, flexibility, high availability, security and cost effective data storage for these
benchmarks.

Software Configuration
The tested configuration utilized the following Software components:

• SAP ERP 2005, Enterprise Core Component 6.0

• Oracle 10gR2

• AIX 5L Version 5.3

Scalability of SAP utilizing Oracle Real Application Clusters on IBM Power Systems

3
 Implementation
The chart below pictures the configuration of the full setup, each server was configured exactly
the same and could have been extended to even more nodes for additional capacity. To
achieve results with a lower number of nodes the corresponding number of servers was simply
removed from the environment. Each of the SAP instances connected to the local Oracle RAC
instance that efficiently coordinated the database accesses across all systems. Each of the
Power 570 Systems used 4 individual gigabit adapters for SAP and Oracle communication
providing outstanding network bandwidth and fault tolerance. The same was achieved for
access to the disk subsystem where 2 4Gbit/sec connections provided fast, reliable and multi-
pathed disk reads and writes.

Scalability of SAP utilizing Oracle Real Application Clusters on IBM Power Systems

4
Performance Results
To demonstrate the outstanding performance of this solution multiple individual tests were
performed and each scenario with 2, 3, 4 and 5 Oracle database nodes were measured and
certified by SAP.

Published benchmark results:

Result SAP Vendor Hardware Operating Process Process Cores Threads Number Database Publica
SD Version System or type ors of RAC Software tion
User nodes Number
37040 6.00 IBM Power 570 AIX 5.3 POWER 8 16 32 5 10gR2 200801
6 RAC 3
30016 6.00 IBM Power 570 AIX 5.3 POWER 8 16 32 4 10gR2 200801
6 RAC 2
22416 6.00 IBM Power 570 AIX 5.3 POWER 8 16 32 3 10gR2 200801
6 RAC 1
15200 6.00 IBM Power 570 AIX 5.3 POWER 8 16 32 2 10gR2 200801
6 RAC 0

Figure5 below graphically illustrates the outstanding scalability of the SAP application in this multi-node Real Application Clusters
environment supporting more than 37000 SD benchmark users on 5 IBM Power 570 systems.

Figure 1: Result Scalability: SD

Users

40000

35000

30000

25000

20000

15000

10000

5000

0
2-Node 3-Node 4-Node 5-Node

While the IBM System 570 servers where running at full CPU capacity the IBM TotalStorage
DS8000 easily processed the disk read and write operations providing extremely fast access

Scalability of SAP utilizing Oracle Real Application Clusters on IBM Power Systems

5
times for database operations, enabling the Servers to operate at very high utilization while still
scaling the application throughput near linearly.

Scalability Aspects
The SAP application suite can scale very well on a single server as many benchmark results
published on SAP's website document at http://www.sap.com/benchmarks. SAP has also
certified and published several 3-Tier benchmarks that demonstrate the ability to scale the
solution vertically across multiple servers. In 3-Tier benchmarks the database represents the
only factor that by default cannot be scaled vertically.

The Oracle RAC product eliminates this factor by distributing the database processes across
multiple servers and therefore eliminating a single point of failure while still providing near linear
scaling of the application across many physical servers.

The table below documents the scalability factors when adding mode database and application
instances to the configuration.

SD SD Users Scalability SAPS SAPS per Scalability

Users per server factor server factor

base: 2node

2-node 15,200 77,970

3-node 22,416 7,216 1.94 112,620 34,650 1.87

4-node 30,016 7,600 1.97 151,800 39,180 1.95

5-node 37,040 7,024 1.96 187,450 35,650 1.94

An average scalability factor of 1.95 demonstrates near linear scalability when adding additional
Servers. This small overhead is necessary to provide vertical scaling and a highly available
database environment to the SAP solution.

Scalability of SAP utilizing Oracle Real Application Clusters on IBM Power Systems

6
 Figure 2: SAP scalability in a Oracle RAC multi-node
environment
45000

30000

Measured
Linear
15000

0
2 Nodes 3 Nodes 4 Nodes 5 Nodes

SAP running on a Oracle RAC implementation scales very well even on large IBM Systems
when adding a large number of nodes. The tests demonstrate a 1.95 scaling average when
adding another 16-core Power 570 System to the cluster. The largest configuration utilized 80
POWER6 cores while still providing a high level of scalability.

Performance Considerations
IBM designs its servers with this environment in mind to ensure that the SAP and Oracle
software runs most efficiently. Because of this fact very little effort is necessary to make sure
software and hardware perform at highest level. This chapter lists specific considerations that
helped setup and run the benchmarks.

Hardware related Considerations

IBM Systems are well tested and tuned when they are delivered to customers. For the
benchmarks a single partition that contained all hardware resources was used on each
individual server. No additional hardware setup or modification was necessary on the server
hardware to achieve these results.

The IBM TotalStorage DS8000 used in these benchmarks demonstrates a similar situation. The
setup of the storage array was done using the included management tool that provided a simple
yet detailed overview of the array. The disk ranks were striped evenly over all disks and
Scalability of SAP utilizing Oracle Real Application Clusters on IBM Power Systems

7
volumes were allocated throughout these ranks. No tuning or special settings were required to
achieve the required throughput values.

Operating System related Considerations

AIX 5L Version 5.3 will run very well on an IBM System 570 without modifications. SAP
provides basic recommendations on Operating System specific settings document in an SAP
note. The benchmarks utilized most of these recommendations and no additional AIX specific
changes were implemented to achieve the results.

The environment made use of hardware specifics that provided additional performance
improvements. Multiple instances of SAP were used on each server to optimally utilize the SAP
architecture. Each SAP instance was bound to a subset of processing cores to avoid processes
migrating between processors and allocating the corresponding memory close the active CPU.
AIX resource sets were used to implement this functionality and the enablement of memory
affinity made sure the memory was allocated close to the processor. This setup is usually not
necessary since the IBM System 570 provides fast access to all memory regions and
processors but provided a small benefit during the benchmark.

Symmetric Multi-Threading (SMT) enabled in the POWER6 hardware improved the

performance of the application environment by feeding 2 processing threads into a single core,
efficiently utilizing the available processing cycles.

The use of 64kb pages for memory access significantly reduced operating system overhead in
managing a large number of memory pages providing more processing time to the application
and therefore improving performance.

SAP related Considerations

The SAP software features an asynchronous update mechanism. Any change during an online
transaction is not committed directly into the database. At completion of a transaction the dialog
workprocess writes the changed data into special database tables named vbtables consisting of
several individual tables. The asynchronous update workprocess reads this table and ultimately
commits the data into the corresponding database tables.

In an Oracle RAC scenario the frequent updates to these tables create a significant
synchronization overhead between the database nodes. To avoid a performance degradation
due to these effects SAP has implemented a mechanism to multiplex these tables and assign
an instance of the tables to a specific database node. Through this mechanism any updates
done by a specific SAP instance will result in an update to a set of tables that is only accessed

Scalability of SAP utilizing Oracle Real Application Clusters on IBM Power Systems

8
by one database node. Since each set of vbtables is only accessed by one database node there
is no synchronization necessary during writes to them.

The setup and maintenance of the vbtable multiplexing is documented in SAP note 52310.

Oracle related Considerations

The Oracle Database, especially with Oracle Real Application Clusters as a linear scaling, high
availability solution, works excellent on IBM Power Systems.

Network
In Oracle RAC configurations the database nodes require fast and redundant connections to
communicate and synchronize across the whole database implementation. The 1GBit Ethernet
network infrastructure with several segments provided reliable, low latency connections
resulting in a high level of concurrency between all the database nodes allowing the
environment to scale very well.

Partitioning
Oracle partitioning was used on several database tables used by the benchmark. Partitioning
allows the database to maintain specific parts of tables on a given node and prevents contention
on data used by all transactions. This distributes the load on database tables optimally across
all nodes. Partitioning does not need to be changed when adding or removing nodes.

The design of the RAC system enables the SAP application to move data that is cached in the
database buffer of one node to another node by simply copying it over the network. This
technology optimizes memory synchronization and avoids the overhead of multiple reads from
physical disk on the target node increasing the effectiveness of the parallel database
implementation.

Summary
As demonstrated by this paper the SAP software scales almost linearly on IBM system 570
servers connected by an Oracle Real Application Cluster. Although the simulated user count of
up to 38000 users heavily utilizes the up to 5 servers with a total of 72 processing cores, the
response time still averaged at less than 2 seconds on average. To achieve this result only very
little intervention from the system administrator was necessary to make full use of the server
processing power. The DS8000 storage array sustained the required i/o throughput while
providing very fast access times to the Oracle database enabling the environment to make full

Scalability of SAP utilizing Oracle Real Application Clusters on IBM Power Systems

9
use of it's resources and provide outstanding performance for the sales and distribution
workload.

These benchmark results are proof that IBM Systems with Oracle Real Application Clusters
provide an outstanding platform for deployments of SAP solutions. Additionally, these
benchmarks demonstrate near linear scalability as nodes are added to the cluster while
delivering a high level of performance and availability necessary for mission critical applications.

Scalability of SAP utilizing Oracle Real Application Clusters on IBM Power Systems

10
Resources
These Web sites provide useful references to supplement the information contained in this document:

• IBM Servers

http://www.ibm.com/servers

• SAP Service Marketplace

http://service.sap.com
Notes: 527843, 1171095, 581320

• SAP Developer Network

https://www.sdn.sap.com/irj/sdn/ora

• Oracle
http://ww.oracle.com/newsletters/sap/index.html

• IBM Redbooks
www.redbooks.ibm.com/

About the author

Joerg Droste is a Master Certified IT Specialist focusing on SAP performance on IBM Systems.

Acknowledgements
Many thanks for their review and contributions to this paper to:
Laurent Montaron – Manager SAP on IBM Systems

Scalability of SAP utilizing Oracle Real Application Clusters on IBM Power Systems

11
Trademarks and special notices
© Copyright IBM Corporation 2009. All rights Reserved.
References in this document to IBM products or services do not imply that IBM intends to make them
available in every country.
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business
Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked
terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these
symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information
was published. Such trademarks may also be registered or common law trademarks in other countries. A
current list of IBM trademarks is available on the Web at "Copyright and trademark information" at
www.ibm.com/legal/copytrade.shtml.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates.

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other
countries, or both.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.

Information is provided "AS IS" without warranty of any kind.

All customer examples described are presented as illustrations of how those customers have used IBM
products and the results they may have achieved. Actual environmental costs and performance
characteristics may vary by customer.
Information concerning non-IBM products was obtained from a supplier of these products, published
announcement material, or other publicly available sources and does not constitute an endorsement of
such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly
available information, including vendor announcements and vendor worldwide homepages. IBM has not
tested these products and cannot confirm the accuracy of performance, capability, or any other claims
related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the
supplier of those products.
All statements regarding IBM future direction and intent are subject to change or withdrawal without
notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller
for the full text of the specific Statement of Direction.
Some information addresses anticipated future capabilities. Such information is not intended as a
definitive statement of a commitment to specific levels of performance, function or delivery schedules with
respect to any future products. Such commitments are only made in IBM product announcements. The
information is presented here to communicate IBM's current investment and development activities as a
good faith effort to help with our customers' future planning.

Scalability of SAP utilizing Oracle Real Application Clusters on IBM Power Systems

12
Performance is based on measurements and projections using standard IBM benchmarks in a controlled
environment. The actual throughput or performance that any user will experience will vary depending
upon considerations such as the amount of multiprogramming in the user's job stream, the I/O
configuration, the storage configuration, and the workload processed. Therefore, no assurance can be
given that an individual user will achieve throughput or performance improvements equivalent to the
ratios stated here.
Photographs shown are of engineering prototypes. Changes may be incorporated in production models.

Scalability of SAP utilizing Oracle Real Application Clusters on IBM Power Systems

13