Put your integrated WebSphere environments into production fast

Best practices for performance-

tuning SAP R/3 and Oracle
database configurations
Part 2 — Tools for optimizing Oracle databases
by Kostas G. Gavrielidis
This article originally appeared in the September/October 2006 issue of SAP
Professional Journal and appears here with the permission of the publisher,
Wellesley Information Services. For information about SAP Professional
Journal and other WIS publications, visit www.WISpubs.com.

Performance-tuning your database configurations is an ongoing task that no

Kostas G. Gavrielidis
Master Technologist,
Hewlett-Packard Services
Kostas G. Gavrielidis is a master
technologist at HP with more than
25 years of experience in software
development and support. He is
currently involved with MSE
proactive consulting projects for
HP customer production database
management systems, and works
on the analysis and performance
enhancement of SAP, Oracle,
Rdb, Ingres, Sybase, SQL Server,
MySQL on Unix, Linux,
OpenVMS, and Windows
platforms. His other involvements
include the architecture, design,
and implementation of high-
availability and disaster-tolerant
customer database production
environments. You can reach him
at kostas.gavrielidis@hp.com.
DBA should take for granted. Left unattended, database performance issues
will gradually worsen over time. Monitoring and adjusting your database
configuration on a regular basis will circumvent larger problems and keep
your users happy.
This is the second installment of a three-part series on performance-
tuning best practices for SAP R/3 production systems running on Unix
with an Oracle relational database management system (RDBMS). In part
1,1 I described the tools available in SAP R/3 and the mySAP ERP Central
Component (ECC) for performance tuning. Here, I turn to the Oracle side
of the performance-tuning equation. I provide an overview of the perform-
ance characteristics of an Oracle-based production environment and
examine the Oracle tools available for tuning your databases. I also look
at the Oracle monitoring capabilities included with SAP R/3 for optimizing
your configuration, in particular the following transactions:
• SAP/Oracle Database Performance Monitor (ST04)
• Exclusive Lock Waits (DB01)
• Tables and Indexes (DB02)
SAP supports database-tuning tasks that can yield medium to high
database performance improvements. They include clustering of tables
with primary index, data buffer tuning, library cache tuning, and table
parameter tuning. For these tasks, the Oracle utilities STATSPACK and
UTLBSTAT/UTLESTAT help with statistics gathering and analysis.
For the Oracle database, you can address database caching efficiency,
database file I/O, database latch contention, and rollback segment
1
“Best practices for performance-tuning SAP R/3 and Oracle database configurations: Part 1 —
SAP tools for monitoring R/3 functionality” (SAP Professional Journal, July/August 2006).

No portion of this publication may be reproduced without written consent. 39

SAP Professional Journal • September/October 2006
Note!
While the suggestions and recommendations
I outline in this article series are based on HP
Tru64 Unix 5.x configurations, they apply to
non-HP configurations as well, since both SAP
R/3 and Oracle run on numerous other plat-
forms with very few differences. The primary
differences between the supported platforms
are in the operating system (OS) tuning areas
for seven of the major OS subsystems: VM,
RT, PROC, INET, RTG, SOCKET, and IPC.
Note!
This article applies to Oracle 9i 9.2 and
SAP R/3 4.6C. The concepts presented also
can be applied to mySAP ERP Central
Component (ECC).
Note!
SAP DBAs have less administrative freedom
than non-SAP DBAs. Many database-
related changes to an SAP R/3 production
environment that could improve database
performance require approval from SAP.
For example, an SAP DBA cannot allocate a
new index or database table structure without
permission from SAP. Additionally, since
ABAP dynamically creates SQL queries, you
cannot tune the queries directly. There is no
method for capturing generated SQL state-
ments and adding database hints to them to
improve execution performance. The most
you can do is note the SQL performance and
report it to SAP for resolution. SAP also does
not support database redesign or index tuning.
40
contention to improve performance. Before I discuss
these areas, it is important to understand that the
Oracle9i memory structures, which reside in the
Oracle System Global Area (SGA), play a significant
role in optimizing these areas of the Oracle database.
Understanding the Oracle9i
memory structures
Oracle9i stores database-related information for
both data and control, all of which is shared by the
Oracle9i database processes. It resides in the virtual
memory of the running server system in the form of
memory structures, which in combination make up
the Oracle System Global Area (SGA), which is
summarized in Figure 1.
The following are the memory structures relevant
to performance tuning:
• The shared pool contains the library cache, the
data dictionary cache, and the user global area
(UGA). The library cache stores the most recently
executed SQL statements. The dictionary cache
keeps information about dictionary objects. The
UGA stores information about shared server
connections when the large pool (more on this
in a moment) is not configured.
• The database buffer cache contains copies of
data blocks from the data files of the most recently
accessed data. To improve performance of the
database buffer cache, you may need to create
multiple buffer pools. Manually assign objects
to the KEEP buffer pool if they are likely to be
reused, and assign them to the RECYCLE buffer
pool if there is little chance of reuse; otherwise,
they will automatically go to the DEFAULT
buffer pool.
• The log buffer contains all transaction-related
information that can be later used for database
recovery.
• The large pool contains all the data for large
operations, such as backup and restore.
• The Java pool contains the most recent Java
applications and related Java objects.
www.SAPpro.com ©2006 SAP Professional Journal. All rights reserved.
 Best practices for performance-tuning SAP R/3 and Oracle database configurations: Part 2
Figure 1 The Oracle 9i System Global Area (SGA)
• The streams pool contains all the necessary data
for streams processing. This new pool has the new
Oracle10g parameter STREAMS_POOL_SIZE,
which specifies its size and must be accounted for
as part of the Oracle10g SGA.
• The least recently used (LRU) list monitors
buffer use. Buffers are sorted by the number of
times they have been used.
• The checkpoint queue contains all the buffers
found during the LRU scan that have been modi-
fied by a server process. The DB writer (DBWn)
server process writes these buffers to disk during
a checkpoint.
Beginning with Oracle9i, a dynamic SGA
implements an infrastructure that allows the SGA
configuration to change without shutting down the
instance. This then allows you to change the sizes of
the database buffer cache, shared pool, and large pool
without shutting down the Oracle instance. The size of
the SGA is determined by the following initialization
parameters, which are specified in the init<SID>.ora
Oracle initialization file:
• The SHARED_POOL_SIZE parameter specifies
the size (in bytes) of the area devoted to shared
SQL and PL/SQL (Oracle’s procedural language
extension to SQL). The default is 8MB for 32-bit
platforms and 64MB for 64-bit platforms. Other
No portion of this publication may be reproduced without written consent.
Oracle parameters that affect the shared pool
components include OPEN_CURSORS, SESSION_
CACHED_CURSORS, CURSOR_SPACE_FOR_
TIME, CURSOR_SHARING, and SHARED_
POOL_RESERVED_SIZE. Oracle recommends
setting SHARED_POOL_RESERVED_SIZE to
10% of the value of the SHARED_POOL_SIZE
parameter.
• The DB_CACHE_SIZE parameter specifies the
size of the DEFAULT buffer pool cache of stan-
dard blocks. The primary block size used for the
SYSTEM tablespace and the primary buffer caches
is specified by the DB_BLOCK_SIZE parameter.
• The DB_{2K, 4K, 8K, 16K, 32K}_CACHE_SIZE
parameter specifies the size of the 2K, 4K, 8K,
16K, and 32K buffers and is only allowed when
the DB_BLOCK_SIZE is not a multiple of kilo-
bytes in size.
• The DB_KEEP_CACHE_SIZE and DB_ RECYCLE
_CACHE_SIZE parameters specify the KEEP
and RECYCLE buffer pools, respectively. The
buffer size in both pools is specified by the
DB_BLOCK_SIZE parameter. The parameter
DB_KEEP_CACHE_SIZE replaces the Oracle8i
parameter BUFFER_POOL_KEEP. The parameter
DB_RECYCLE_CACHE_SIZE replaces the
Oracle8i parameter BUFFER_POOL_RECYCLE.
41
SAP Professional Journal • September/October 2006
• The LOG_BUFFER parameter specifies the
number of bytes allocated for the log buffer.
The default is the greater of 512KB or 128KB
× CPU_COUNT.
• The LARGE_POOL_SIZE parameter specifies
the size of the large pool. The default is zero
unless the init.ora Oracle parameter PARALLEL_
AUTOMATIC_TUNING is set to TRUE, in which
case the default is automatically calculated.
• The JAVA_POOL_SIZE parameter specifies the
size of the Java pool. The default is 24MB. Other
parameters that can be used to limit the amount
of memory used by each Java session include
JAVA_SOFT_SESSIONSPACE_LIMIT (the
default is 1MB) and JAVA_MAX_
SESSIONSPACE_SIZE (the default is 4GB).
• The STREAMS_POOL_SIZE parameter specifies
the size of the streams pool. If this parameter is set
to 0, then by default up to 10% of the shared pool
is allocated for the streams pool. If this parameter
is greater than zero, then the value is used to
specify the size of the streams pool.
As I mentioned earlier, the Oracle SGA plays
a role in database caching efficiency, database file
I/O, database latch contention, and rollback segment
contention. Let’s look at how you can optimize these
areas within the Oracle database, starting with data-
base caching efficiency.
Database caching efficiency
The most significant improvement you can make
to any Oracle database is to manage the buffers
containing database data that has been cached from
disk into system memory. The Oracle DBA speci-
fies the memory size for the cached data with the
Oracle init.ora parameter DB_CACHE_SIZE.
The DB_CACHE_SIZE parameter replaces the
DB_BLOCK_BUFFERS parameter of Oracle8i.
Since Oracle9i can support databases for different
block sizes, you can now change the size of all
Oracle caches such as the buffer cache, the shared
pool, the large pool, and the fixed size while the
instance is running, so you do not have to shut it
42
down. You do this using the new Oracle parameter
SGA_MAX_SIZE. It specifies the maximum size in
bytes that the Oracle SGA can reach for the life of the
particular instance.
The total size of the SGA cannot exceed the value
specified in the Oracle parameter SGA_MAX_SIZE
and can be computed with the following formulas for
both the Oracle9i and Oracle10g versions:
• Total SGA size for Oracle9i =
DB_CACHE_SIZE
+ DB_2K_CACHE_SIZE
+ DB_4K_CACHE_SIZE
+ DB_8K_CACHE_SIZE
+ DB_16K_CACHE_SIZE
+ DB_32K_CACHE_SIZE
+ DB_KEEP_CACHE_SIZE
+ DB_RECYCLE_CACHE_SIZE
+ SHARED_POOL_SIZE
+ LARGE_POOL_SIZE
+ JAVA_POOL_SIZE
+ LOG_BUFFER
+ 1MB
< SGA_MAX_SIZE
• Total SGA size for Oracle10g =
<Total SGA size for Oracle9i>
+ STREAMS_POOL_SIZE
< SGA_MAX_SIZE
Let’s now turn our attention to ways you can
improve the performance of database file I/O by
relocating files appropriately.
Database file I/O
Proper placement of database files can reduce
contention for resources. The Oracle utility
STATSPACK provides a list of all database files
and the associated amount of I/O. Verify the balance
of I/O activity across all available disks. If multiple
database files on the same disk cause contention,
move one of the files to a less active disk. Do not
move any of the database files to the application
disk, however, since this disk is usually quite active.
www.SAPpro.com ©2006 SAP Professional Journal. All rights reserved.
 Best practices for performance-tuning SAP R/3 and Oracle database configurations: Part 2
If I/O problems persist after attempts to reduce and
balance demand, consider enhancing system I/O
capacity. Options include using disk striping, adding
disks, and upgrading to faster disks, including solid-
state disks.
Note!
Take great care in distributing key datafiles
across the available file systems. Heavily
accessed tables should be located on file sys-
tems that are separate from their correspon-
ding indexes. The Oracle online redo log files,
control files, and archive log files should be
stored separately from the database datafiles
for recovery purposes. Store multiple copies
of the database control files on different disks
and under different disk controllers.
There are some specific steps and considerations
involved in relocating a datafile or relocating a table
to its own tablespace and datafile. We’ll take a look at
these next.
Relocating a datafile
When you have a large imbalance of datafile I/O
activity, you need to physically move the datafile to
another disk location to balance the datafile I/O
activity. After you have identified a particular datafile
that has contention issues, use the following procedure
to relocate the datafile:
1. Place the tablespace corresponding to the datafile
offline with the following SQL command:
SQL> ALTER TABLESPACE <tbl-space-name>
OFFLINE;
2. Use the Unix cp command to copy the datafile to a
new disk location:
No portion of this publication may be reproduced without written consent.
# cp /fromdisk/<tbl-space-datafile>.dbf
/todisk/<tbl-space-datafile>.dbf
3. Rename the datafile to point to the new disk
location with the following SQL command:
SQL> ALTER TABLESPACE <tbl-space-name>
RENAME DATAFILE '/fromdisk/<tbl-space-
datafile>.dbf'
TO '/todisk/<tbl-space-datafile>.dbf';
4. Bring the tablespace back online with the
following SQL command:
SQL> ALTER TABLESPACE <tbl-space-name>
ONLINE;
5. Delete the old datafile after verifying that the
moved datafile can be accessed with the Unix rm
command:
# rm /fromdisk/<tbl-space-datafile>.dbf
Relocating a table to its own
tablespace and datafile
You also need to balance table I/O activity when some
tables receive excessive amounts of read and write I/O
activities. To do this, you may have to move the tables
to their own tablespaces and datafiles. Performing
these steps could result in significant performance
gains due to reduced contention, reduced database
fragmentation, and balanced disk I/O. Consider the
following guidelines:
1. Verify the tables that Oracle updates or reads most
frequently.
2. Remove the tables through the Oracle export
utility.
3. Create new tablespaces for the tables and indexes
on separate disk drives.
4. Import the exported data into the newly created
tablespaces.
43
SAP Professional Journal • September/October 2006

Another area ripe for performance improvement is
database latches. Next we’ll look at how to identify
latches that are causing excessive resource consumption.
The cache buffer chains latch often experiences
contention, as does the cache buffer LRU chain and
the cache buffer handles latch. These latches typically
indicate that tuning the number of the SGA buffers is
required.

Database latch contention
Database latches are simple low-level serialization
mechanisms used to protect Oracle SGA shared data
structures. Their implementation is OS-dependent. If
a process cannot obtain a latch immediately, it spins
while it waits for the latch and thus consumes CPU
cycles.
Latches control access to the log buffer. The
Oracle statistical gathering tools — STATSPACK and
UTLBSTAT/UTLESTAT — provide a list of all
latches and the amount of contention for them. To
identify latch contention, you need to determine the
ratio between the columns TIMEOUTS and
IMMEDIATE. If the ratio of TIMEOUTS to
IMMEDIATE is greater than 15%, then latch
contention exists. If the value of latch_free for the
output from v$system_event seems to be causing
contention, then you can use the output from
v$session_wait to determine which latches are causing
the contention. The latch number is given in the p2
field, and you can identify it with this query form:
The last area we will look at is rollback segment
contention, and how you can improve performance by
correctly sizing the segments.
Rollback segment contention
Rollback segments are made up of contiguous multi-
block pieces known as extents, and they contain
rollback entries. The correct size of the rollback
segments depends on the normal transaction activity
of the database. Choosing the wrong size can affect
database performance.
The size is specified by the storage parameters of
the rollback segment itself. The best way to avoid
rollback segment contention is to create many rollback
segments large enough to contain the rollback entries
for the transactions, and to assign each transaction to
its own rollback segment with the use of the SQL SET
TRANSACTION statement. For example:

SQL> select latch#, name from v$latch SQL> set transaction use rollback
where latch#=8; segment prs_45;

A warning about Oracle parameters

SAP supports all versions of the R/3 release on 23 operating system (OS) platforms and on 17 persistent
storage managers. Now let’s reduce all this to just the HP-supported configurations with only the Oracle
storage manager. Oracle alone has n+1 versions supported in an SAP R/3 production environment. The
three most recent — Oracle8i, Oracle9i, and Oracle10g — have numerous important changes that range
from new parameters that are supported or declared obsolete to the way you can monitor database engine
performance.
So take into consideration that, depending on the Oracle version you are using, any Oracle parameter
mentioned in this article may have become obsolete, have a new name, may not be implemented on a
particular version of the OS, or is hidden and only Oracle support can authorize its use.

44 www.SAPpro.com ©2006 SAP Professional Journal. All rights reserved.

 Best practices for performance-tuning SAP R/3 and Oracle database configurations: Part 2

To find out which Oracle parameters are underscored (valid, but should be used only if absolutely
necessary) or obsolete, query the X$KSPPO view on your version of the Oracle RDBMS as follows:
SQL> SELECT kspponm,
2 DECODE(ksppoflg, 1,'Obsolete', 2, 'Underscored')
3 FROM x$ksppo
4 ORDER BY kspponm;

Executing this query on Oracle 9i (9.2.0.1.0), for example, produces the following results:
KSPPONM DECODE(KSPP)
----------------------------------------------------------------------------------
_average_dirties_half_life Obsolete
_db_no_mount_lock Obsolete
_lm_multiple_receivers Obsolete
_lm_statistics Obsolete
_seq_process_cache_const Obsolete
allow_partial_sn_results Obsolete
always_anti_join Underscored
always_semi_join Underscored
arch_io_slaves Underscored
b_tree_bitmap_plans Underscored
backup_disk_io_slaves Underscored
cache_size_threshold Obsolete
cleanup_rollback_entries Underscored
close_cached_open_cursors Underscored
compatible_no_recovery Underscored
complex_view_merging Underscored
cpu_count Underscored
db_block_checkpoint_batch Obsolete
db_block_lru_extended_statistics Obsolete
db_block_lru_latches Underscored
db_block_lru_statistics Obsolete
db_block_max_dirty_target Underscored
db_file_simultaneous_writes Obsolete
delayed_logging_block_cleanouts Obsolete
discrete_transactions_enabled Underscored
distributed_recovery_connection_hold_time Underscored
distributed_transactions Obsolete
fast_full_scan_enabled Underscored
freeze_DB_for_fast_instance_recovery Obsolete
gc_defer_time Underscored
gc_latches Underscored
gc_lck_procs Obsolete
gc_releasable_locks Obsolete
gc_rollback_locks Obsolete
hash_multiblock_io_count Underscored

Continues on next page

No portion of this publication may be reproduced without written consent. 45

SAP Professional Journal • September/October 2006

Continued from previous page

instance_nodeset Obsolete
job_queue_interval Obsolete
job_queue_keep_connections Obsolete
large_pool_min_alloc Underscored
lgwr_io_slaves Underscored
lm_locks Underscored
lm_procs Underscored
lm_procs Underscored
lm_ress Underscored
lock_sga_areas Underscored
log_block_checksum Obsolete
log_files Obsolete
log_files Obsolete
log_simultaneous_copies Underscored
log_small_entry_max_size Obsolete
max_transaction_branches Obsolete
ogms_home Underscored
ops_admin_group Obsolete
ops_interconnects Obsolete
optimizer_percent_parallel Underscored
optimizer_search_limit Underscored
parallel_broadcast_enabled Underscored
parallel_default_max_instances Underscored
parallel_min_message_pool Underscored
parallel_server_idle_time Underscored
parallel_transaction_resource_timeout Obsolete
push_join_predicate Underscored
row_cache_cursors Underscored
sequence_cache_entries Obsolete
sequence_cache_hash_buckets Obsolete
shared_pool_reserved_min_alloc Underscored
snapshot_refresh_interval Obsolete
snapshot_refresh_keep_connections Obsolete
snapshot_refresh_processes Obsolete
sort_direct_writes Obsolete
sort_multiblock_read_count Underscored
sort_read_fac Obsolete
sort_spacemap_size Obsolete
sort_write_buffer_size Obsolete
sort_write_buffers Obsolete
spin_count Underscored
standby_preserves_names Obsolete
temporary_table_locks Obsolete
text_enable Obsolete
use_ism Underscored

46 www.SAPpro.com ©2006 SAP Professional Journal. All rights reserved.

 Best practices for performance-tuning SAP R/3 and Oracle database configurations: Part 2
79 rows selected.
SQL>
SQL> exit
Disconnected from Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.1.0 - Production
Now that I’ve shown you ways to optimize the
performance of your Oracle databases using Oracle
utilities, let’s see what you can do with the tuning
tools available with SAP R/3.
SAP/Oracle Database
Performance Monitor
The SAP/Oracle Database Performance Monitor
shows the most important Oracle database perform-
ance indicators. To display the monitor, from the main
SAP R/3 screen follow the menu path Tools →
Administration → Computing Center → Management
System → Control → Performance Menu → Database
→ Activity, or use transaction ST04.
The information on the main screen is subdivided
into eight sections — Data buffer, Shared Pool, Log
buffer, Calls, Time statistics, Redo logging, Table scans
& fetches, and Sorts — which each contain various
subsections, as shown in Figure 2 on page 48. In the
next sections, we will examine the most important
information sections and subsections on this screen.
Data buffer
The database buffer cache, also known as the data
buffer or Oracle data buffer, is the area of the Oracle
SGA that contains copies of data blocks read from the
datafiles of the most recently accessed data. With the
exception of table sorting and table scans, Oracle user
No portion of this publication may be reproduced without written consent.
processes generally cannot read data directly from
datafiles, which is why all data must first be read into
this buffer cache.
When a user process requests a data block that is
already in the data buffer, it can be read without
having to access the disk again, assuming that the
block has not been changed since it was last read into
the buffer. This saves considerable processing time. In
this situation, the user process has made a “hit” on
that data block. A user process request on a data block
that is not in the data buffer is called a “miss.” The
relationship between hits and misses is known as the
“hit ratio.” The hit ratio also can be thought of as the
quality of the database buffer cache.
To improve performance of the database buffer
cache, you may need to create multiple buffer pools.
Manually assign objects to the KEEP buffer pool if
they are likely to be reused. Assign them to the
RECYCLE buffer pool if there is little chance of
reuse. Otherwise, they will automatically go to the
DEFAULT buffer pool. Objects in the KEEP buffer
pool will be reused rather than reloaded from disk.
This SQL example specifies the INDEX Cust_idx
to be kept in the KEEP buffer pool at the time of
index creation by using the STORAGE clause:
CREATE INDEX Cust_idx ...
STORAGE (BUFFER_POOL KEEP ...);
This SQL example alters the TABLE customer to
be kept in the RECYCLE buffer pool at runtime by
using the STORAGE clause:
47
SAP Professional Journal • September/October 2006

Database P11 Database summary Day, Time 04.05.2004 07:31:31

DB Server kgg0064c Start up at 20.03.2004 13:14:52
Release 9.2.0.4.0 Elapsed since start (s) 3.867.400
Data buffer --------------------------------------------------------------------------
Size kb 4.096.000 Reads 142120468.130
Quality % 96,7 Physical reads 4.650.531.039
writes 419.344.855
Buffer busy waits 233.966.522
Buffer wait time s 1.283.242

Shared Pool ------------------------------ Log buffer -----------------------------

Size kb 2.048.000 Size kb 1.304
DD-Cache quality % 96,2 Entries 3.144.468.028
SQL Area getratio % 99,5 Allocation retries 334.757
pinratio % 100,4 Alloc fault rate % 0,0
reloads/pins % 0,1370 Redo log wait s 12.545
Log files (in use) 8( 8)
Calls --------------------------------------------------------------------------------
User calls 7.602.211.045 Recursive calls 563.043.522
commits 105.102.848 Parses 155.472.916
rollbacks 112.153 User/Recursive calls 13,5
Reads/User calls 18,7

Time statistics ----------------------------------------------------------------------

Busy wait time s 33.060.492 Sessions busy % 7,54
CPU time s 4.108.898 CPU usage % 6,64
Time/User call ms 5 CPU count 16
Redo logging -------------------------------------------------------------------------
Writes 99.540.081 Write time s 224.341
OS-Blocks written 1.876.349.507 Mb written 1.259.075
Latching time s 405

Table scans & fetches -------------------- Sorts ----------------------------------

Short table scans 26.414.777 Memory 366.379.089
Long table scans 42.223 Disk 4.402
Fetch by rowid 205350340.086 Rows sorted 13468.625.297
by continued row 481.402.761

Figure 2 Database performance summary

init<SID>.ora Oracle parameter file. By default, SAP

ALTER TABLE customer
STORAGE (BUFFER_POOL RECYCLE);
You’ll find the most useful information for tuning
purposes in the Size, Quality, Reads, Physical reads/
writes, Buffer busy waits, and Buffer wait time
subsections. We’ll take a closer look at these next.
uses a value of 8192 bytes for the DB_BLOCK_SIZE
parameter for most Oracle databases. A database that
has been created by whatever value used for this
DB_BLOCK_SIZE parameter cannot be later changed
for the life of the database. You can change the value
for DB_BLOCK_BUFFERS, however. Note that
Oracle10g can automatically size the blocks.

Size
Data buffer size is determined by the product of the
block size (Oracle parameter DB_BLOCK_SIZE)
and the number of database block buffers (Oracle
parameter DB_BLOCK_BUFFERS) specified in the
Quality
The data buffer quality indicates how successful the
Oracle RDBMS is at finding data blocks in memory
that have been requested by a database server process.
Every time this activity is successful, the quality
improves, and every time it is not successful, the

48 www.SAPpro.com ©2006 SAP Professional Journal. All rights reserved.

 Best practices for performance-tuning SAP R/3 and Oracle database configurations: Part 2
quality degrades. SAP recommends that you maintain
a data buffer quality of at least 96%.
Note!
When the database instance has just been
started, the hit ratios shown may not be accu-
rate. The database should be running for at
least several days in the production system to
get accurate indications of the hit ratios. The
Elapsed since start (s) line at the top of the
display shown in Figure 2 gives this time.
Reads, Physical reads/writes
The read and write operations statistics provide a
quick indication of the level of read/write I/O activity
in the database since it was started. In a newly created
database, expect a large amount of write I/O because
of the batch updates to populate the database. If you
find the number of physical writes to be considerably
higher than the number of reads after this time, then
you need to monitor the activities of the database
writer, rollback activities, and redo log writer.
Buffer busy waits, Buffer wait time
When an Oracle server process attempts to access a
block that is being modified or is in an inconsistent
state, a wait situation occurs in the buffer. The value
displayed here is the average number for all the
Oracle block classes. A number of Oracle block
classes may play a part in the occurrence of wait
situations, but only four are commonly found when
monitoring the SAP system: data block, segment
header, undo header, and undo block classes.
When the total number of wait situations exceeds
5% of the total number of reads, you need to closely
analyze the situation. To do this, go to the Detail
Analysis menu (transaction ST06) and click on Wait
Situations. If the number of waits on any one of
the block classes specified exceeds 1% of the reads,
No portion of this publication may be reproduced without written consent.
it might indicate excessive contention for this
particular class.
You can reduce undo header waits and undo block
classes waits by adding more rollback segments to the
database. Data blocks waits may be due to the data
buffer size not being large enough. Segment headers
waits often indicate contention for free lists.
Shared Pool
The shared pool contains the library cache, the data
dictionary cache, and the UGA. The library cache
stores the most recently executed SQL statements.
The data dictionary cache keeps information about
dictionary objects. The UGA keeps information about
shared server connections when the large pool is not
configured. Significant performance improvements
result when the issued SQL statements can be found
in the shared pool and are reused without reparsing
them. This is one of the most important storage struc-
tures in an Oracle database system.
You’ll find the most useful information for tuning
purposes in the Size, DD-Cache quality, and shared
SQL Area subsections. We’ll take a closer look at
these next.
Size
The size of the shared pool is determined by the
Oracle parameter SHARED_POOL_SIZE specified in
the init<SID>.ora file. The value of this parameter is
set to 20MB in SAP R/3. Depending on the system
workload, it may be necessary to increase this value
by taking into account the total amount of memory
available. SAP recommends that the value set for this
parameter does not exceed 50MB. This is based on
SAP’s benchmarks, but your production system may
have other needs.
DD-Cache quality
The data dictionary cache contains information
about users, database schema objects, and database
49
SAP Professional Journal • September/October 2006
storage structures. Modification to the data dictionary
cache are made every time a data definition language
statement is issued. Oracle accesses the data in the
data dictionary cache frequently, so it is important to
retain most of this needed data in memory, and specif-
ically in the Oracle SGA. The data dictionary cache
quality, then, is a reflection of the overall average hit
ratio for the various Oracle dictionary caches. SAP
recommends that the hit radio should be more than
90% on production systems. You do this by moni-
toring and adjusting the SHARED_POOL_SIZE
parameter.
As with the other caches and their state at database
startup time, the data dictionary cache is empty and
will grow with data as the database is being used. The
statistics will be useful to look at after the database
has been running in the production environment for
several days.
Shared SQL Area
The shared SQL Area contains the parse tree and
execution plans for all SQL statements. The process
of parsing and loading an SQL statement in memory
requires valuable system resources. SQL statements
that have been through this process are stored in the
shared SQL areas along with other identical SQL
statements. Even a single poorly written SQL state-
ment can have a negative effect on the performance
of the system, so the goal is to reuse SQL statements
that have already been parsed. The success rate for
accessing SQL statements in the shared SQL area is
given by the values of the measurement fields under
SQL Area getratio and pinratio. The pinratio is the
most important indicator and should be as close to
100% as possible.
The goal is to reduce the amount of parsing by
identifying the SQL statements that can be improved
so they can be placed in the shared SQL areas. If
this is not possible, then the only alternative is to
increase the size of the shared pool by increasing the
Oracle SHARED_POOL_SIZE parameter in the
init<SID>.ora file.
50
Log buffer
The log buffer holds information about all changes
made to the database before the Oracle Log Writer
(LGWR) process writes the buffer contents in the
online redo log files. Each change generates a “redo
entry,” which is needed to reconstruct these changes
during the database recovery process.
Storing information in memory first reduces the
disk I/O activity. It is important to flash the log buffer
from memory to disk. Otherwise the database
recovery cannot be guaranteed.
You’ll find the most useful information for tuning
purposes in the Size, Allocation retries, and Allocation
fault rate subsections. We’ll take a closer look at
these next.
Size
The log buffer size is set by the Oracle initialization
parameter LOG_BUFFER in the init<SID>.ora file.
On SAP systems, the default value of this parameter is
320KB. For non-SAP configurations, the default value
of this parameter is the greater of 512KB or 128KB
times the value of the CPU_COUNT parameter.
Allocation retries
This field indicates the number of unsuccessful
attempts by the Oracle service process to allocate
space in the log buffer. This value should be as close
to zero as possible. Every time the LGWR is not
successful, it waits for a redo log file switch to
perform this needed operation. If the allocation retries
field increases while you monitor it, you may need to
increase the Oracle LOG_BUFFER parameter.
A redo log file switch is the activity when Oracle
stops writing to one of the online redo log files and
begins writing to another online redo log file. This
can happen by specifying a time manually, or auto-
matically when the redo log file is full. If the switch
is based on the size of the redo log file, then you may
want to have larger-size files to store the redo log
entries. This reduces the frequency of redo log file
www.SAPpro.com ©2006 SAP Professional Journal. All rights reserved.
 Best practices for performance-tuning SAP R/3 and Oracle database configurations: Part 2
switches because it allows more entries stored in
each file.
Allocation fault rate
This field shows the ratio of allocation retries to the
total redo entries in the log buffer. The desired rate for
this ratio should be close to zero. Otherwise, you may
need to investigate the cause of the errors and identify
areas for improvement.
Calls
This portion of the report contains the number of calls
made to the Oracle database engine from SAP-related
processes since the start of the database instance.
Reducing the number of calls to the Oracle kernel alle-
viates the database workload. However, you should
expect the value of this field to be high in a production
environment and to increase while monitoring.
You’ll find the most useful information for tuning
purposes in the Commits, Rollbacks, Recursive calls,
and Parses subsections. We’ll take a closer look at
these next.
Commits
The Commits field indicates the total number of
committed transactions since database instance startup.
Rollbacks
The Rollbacks field indicates the total number of
transactions that were rolled back since the database
system was started. Rollbacks can be caused by a
number of reasons, including failing programs, appli-
cation deadlocks, or abnormal application termination.
If a large number of rollbacks are reported relevant
to what’s been observed in the past, check the Oracle
alert and trace files for possible problems. The Oracle
alert and trace files for the Oracle background
processes are usually found in the /oracle/<SID>/
saptrace/background directory area. The trace files
for Oracle user processes are found in the
/oracle/<SID>/saptrace/usertrace directory area.
No portion of this publication may be reproduced without written consent.
Recursive calls
Recursive calls occur when Oracle itself must issue
an SQL statement in addition to the SQL statement
issued by a particular user process. The most common
causes of recursive calls are misses in the data dic-
tionary cache, dynamic storage extension, execution
of DDL statements, the enforcement of referential
integrity constraints, and the use of PL/SQL (Oracle’s
procedural language extension to SQL).
Recursive calls can impair the performance of the
database system and should be minimized wherever
possible. The recursive call ratio is calculated as
recursive calls/user calls. Ideally, this ratio should
be 4:1. If it is not, then check the data dictionary
cache hit ratio and average parse ratio. Increasing
SHARED_POOL_SIZE should help. Dynamic storage
extension occurs when a database object (a table or
index) must extend beyond its allocated space (that is,
a new extent is allocated).
I recommend that you always create the original
extent in the INITIAL parameter and the following
extents of the NEXT parameter large enough to mini-
mize dynamic extent assignment. It is only possible to
change the INITIAL storage parameter for an object
through reorganization. You should adjust the NEXT
parameter to SAP requirements on a regular basis
using the SAP program SAPDBA with the -next
option as in this example:
sapdba –next <tablespace/s>
Note that table reorganization is generally not
necessary; but index reorganization can sometimes
prove helpful.2
As in the case of the cache hit ratios, the value for
recursive calls will be high after database instance
startup. Since the data dictionary cache is at first
empty, all calls needed to load information into
working memory will be recursive.
2
For more on database reorganizations, see the article “Boost SAP
R/3 Performance by Reorganizing Your Oracle Database: A Proven
Reorganization Strategy” in the July/August 2005 SAP Professional
Journal.
51
SAP Professional Journal • September/October 2006
Parses
This field shows the total number of times an SQL
statement was parsed. To calculate the average parse
ratio, divide parses by user calls. If this ratio is above
25%, there may be a problem with retaining cursors in
the shared cursor cache. Check the hit rates discussed
in the shared SQL area statistics; it may be necessary
to increase the Oracle SHARED_POOL_SIZE
parameter.
Table scans & fetches
A full table scan occurs when a user process queries
data from the database table without using an index.
The entire table must be read to retrieve the requested
information. Sometimes this is desirable, for example,
if the table is short. Often, it is more efficient to use
an index.
You’ll find the most useful information for tuning
purposes in the Short table scans, Long table scans,
Rows accessed, Data blocks read, Fetch by rowid, and
Fetch by continued row subsections. We’ll take a
closer look at these next.
Short table scans
This field shows the total number of full table scans
performed on short tables. Before Oracle10g, short
tables were those that had fewer than five Oracle data
blocks. By default, a table in Oracle10g is considered
small when it is defined as cached or it is smaller than
2% of the DB_BLOCK_BUFFERS parameter or the
DB_CACHE_SIZE parameter. It is generally more
efficient to perform full table scans on short tables
rather than access the data using indexes.
Long table scans
This field shows the total number of full table scans
done on long tables. Before Oracle 10g, long tables
were those that contained five or more Oracle data
blocks. In Oracle10g, a long table is one that is larger
than 2% of the DB_BLOCK_BUFFERS parameter or
the DB_CACHE_SIZE parameter. It is usually better
to access long tables using indexes.
52
The sum of the values for short and long tables
gives the total number of full table scans performed
since database instance startup. A high number of full
table scans on long tables might indicate missing table
indexes. Check whether indexes are missing via the
Missing Indexes button on the Tables and Indexes
screen (from the SAP R/3 main menu, choose Tools →
Administration → Computing Center → Management
System → Control → Performance Menu → Database
→ Tables / Indexes, or use transaction DB02).
Use the Explain one SQL request option for an
SQL trace (select Test → Performance Trace from the
ABAP Workbench screen, or use transaction ST05) to
examine the optimizer access path of expensive state-
ments. Once you enter the SQL statement and save it,
the system analyzes the SQL statement and displays
all relative information about the strategy used to
access the database. Based on this information, you
need to determine whether to add a new index or
reorder an existing one.
Rows accessed
This field shows the total number of rows accessed
during full table scans. Table scan rows gotten plus
table fetch by rowid gives the total number of rows
accessed since database instance startup. Figure 2
does not show this field because there was no activity
in that area in this case. Rowid is the fastest path to a
row, so you should use it wherever possible. You
might have to make changes to the applications to
take advantage of using the rowid feature.
Data blocks read
This field shows the total number of Oracle data
blocks read during full table scans. Again, this field
does not appear in Figure 2 because there was no
activity in that area in this case. You can measure data
block selectivity as the ratio of data blocks read during
full table scans to rows read during full table scans. It
could indicate excessive empty space in the table being
scanned, possibly due to large delete operations. You
can check fully scanned tables with poor data block
selectivity individually using the State on Disk func-
tion in the Detail Analysis menu (transaction ST06).
www.SAPpro.com ©2006 SAP Professional Journal. All rights reserved.
 Best practices for performance-tuning SAP R/3 and Oracle database configurations: Part 2
Fetch by rowid
This field shows the number of rows that were
accessed either by index lookup or by specifying a
distinct rowid as part of the SQL statement. Rowid is
the fastest path to a row and you should use it when-
ever possible. A high value in this field is a good
indication that indexes are being used. Taking care to
place the index fields first in the table structure will
improve performance. Generally it is better to perform
a full table scan when more than 20% of the rows in a
table are the result of an SQL query, rather than to
make use of rowid.
Fetch by continued row
This field shows how often the database has accessed
chained data records. Row chaining occurs when the
current block size is too small to insert data. The
entire row is migrated to another block, causing
increased I/O to read the row and subsequent perform-
ance degradation.
The number of table fetches by continued rows
should be near zero. If this becomes an issue, use the
ANALYZE command to see the number of chained
rows in frequently accessed tables as well as in the
data dictionary. First, connect internally and execute
the utlchain utility to create the temporary table
CHAINED_ROWS. Then issue the following SQL
commands:
SQL> ANALYZE TABLE <table-name> LIST
CHAINED ROWS;
SQL> SELECT * FROM CHAINED_ROWS;
Use ANALYZE to list chained rows in the data
dictionary tables TAB$ (tables), IND$ (indexes), and
CLU$ (clusters). Another method to identify the objects
with chained records is to use the SAPDBA utility.
Chaining cannot be avoided for tables in which the
row length exceeds the block size. To resolve this,
export, drop, and import the table, or delete and rein-
sert the chained rows. If the row length exceeds the
block size, export the full database and re-create it
with a larger block size.
No portion of this publication may be reproduced without written consent.
Sorts
Sorting is done in Oracle for SQL statements that use
ORDER BY, GROUP BY, and SORT MERGE JOIN operations,
and for index creation. To increase the efficiency of
large sorts, stripe the temporary tablespace. Use OS
striping whenever possible.
You’ll find the most useful information for tuning
purposes in the Memory, Disk, and Rows sorted
subsections. We’ll take a closer look at these next.
Memory
This field indicates the number of sorts that have
been done in memory, which are generally much
faster than those done on disk. The init<SID>.ora
Oracle parameter SORT_AREA_SIZE determines the
amount of process memory, which can be used for all
sort operations. This parameter is yet another Oracle
RDBMS version-dependent implementation. With
Oracle9i, Oracle introduced a new parameter called
PGA_AGGREGATE_TARGET. If this new parameter
is set and you are using dedicated Oracle connec-
tions, Oracle9i ignores all PGA parameters in the
init<SID>.ora file, including the SORT_AREA_
SIZE and SORT_AREA_RETAINED_SIZE
parameter settings, and it will use the common
PGA area for sorting as defined by the PGA_
AGGREGATE_TARGET parameter.
A large sort area increases the chance that sort
operations are performed in memory, and it does
not use temporary segments on disk. If the hard-
ware configuration has a large amount of available
memory, increase the init<SID>.ora Oracle para-
meter SORT_AREA_SIZE to increase the amount
of memory available in the SGA for sort operations.
Disk
This field indicates the number of sorts written to
temporary segments on disk to be sorted. SAP uses
the tablespace PSAPTEMP to hold these segments.
The disk-to-memory ratio should be no more
than 5%. If it is higher, consider increasing the
Oracle SORT_AREA_SIZE parameter in the
init<SID>.ora file.
53
SAP Professional Journal • September/October 2006

10.05.2004 07:59:46 P11 p11dbsrv

Fast Access: Exclusive session-lock situations

Level = 1 : Lock Situation detected

Level > 1 : Lock Holder is waiting on one or more Locks itself
Level = -1: Deadlocks detected

Object Holder (Oracle-SID, -SPID; Client-Host, -PID) Time (s) Level

Waiters (Oracle-SID, -SPID; Client-Host, -PID) Time (s) Level

No exclusive Lockwaits found

Object Waiters (Oracle-SID, -SPID; Client-Host, -PID) Time (s)

No Waiters without Holders found

Figure 3 Transaction DB01 report

Figure 4 Tables and Indexes screen

54 www.SAPpro.com ©2006 SAP Professional Journal. All rights reserved.

 Best practices for performance-tuning SAP R/3 and Oracle database configurations: Part 2

Database system
Refresh RSORATDB is started immediately to update all statistical data.
Checks Up-to-date statistics are created, and some statistics already generated with RSORATDB are
displayed. Details on extents and missing indexes are always up-to-date.
Space statistics Shows the history of the database size. These values are read from table MONI. Click on
Refresh if you require more up-to-date status data; this starts RSORATDB again. Use this
option only if necessary, as it can take considerable time to determine all the information
depending on the size of the database.
Tablespaces
Current sizes These details are determined during the last RSORATDB run.
Space statistics History of tablespace size. These values are read from table MONI.

Freespace statistics These details are determined during the last RSORATDB run.

Tables and indexes

Detailed analysis These details are determined during the last RSORATDB run.

Missing indexes RSORATDB examines the database for missing indexes.

Space critical objects These details are determined during the last RSORATDB run.
Space statistics History of table/index size. These values are read from table MONI.

Figure 5 Transaction DB02 buttons and their functions

Rows sorted present in the environment. The report identifies all

This field indicates the total number of rows sorted. exclusive lock waits and all lock waiters without lock
Sorting is expensive. Application developers should holders. Figure 3 shows an example of a transaction
aim to reduce the amount of sorting wherever possible. DB01 report with no exclusive locking issues reported.

Let’s now turn our attention to ways you can

improve the performance of your Oracle databases
by identifying exclusive locks that are holding up
transactions.
Exclusive Lock Waits
Lock waits, which occur when multiple users are
trying to update the same information, are yet another
source of performance degradation. In particular,
exclusive lock waits can hold up transactions for
unreasonably long times. Fortunately, you can identify
troublesome exclusive locks so that you can notify the
owners and take appropriate action.
Transaction DB01 (Analyze Exclusive Lock Waits)
generates a report of all the lock situations that may be
Tables and Indexes
To ensure the smooth operation of your database
system, you must monitor tablespace growth in your
database and the amount of freespace left for the table-
spaces to grow. If a tablespace overflows, no entries
can be made in this tablespace until freespace is added.
No tablespace that SAP R/3 performance statistics
show as growing should be more than 90% full.
To find tablespaces that are too full, from the main
SAP R/3 screen follow the menu path Tools →
Administration → Computing Center → Management
System → Control → Performance Menu → Database
→ Tables / Indexes, or use transaction DB02, shown
in Figure 4. As you can see, the information on this
screen is subdivided into various sections (Figure 5).

No portion of this publication may be reproduced without written consent. 55

SAP Professional Journal • September/October 2006
Figure 6 The Database Checks screen
Click on the Space critical objects button in the Tables
and indexes frame to search for tablespaces that are
too full.3
Three different functions are provided for checking
tablespace data consistency between the ABAP
Dictionary and the database:
• Missing indexes displays indexes that are not
known in the database or ABAP Dictionary. To
use this check, click on the Missing indexes
button on the Tables and Indexes screen (Figure 4).
• Database <-> ABAP Dictionary checks the
existence of all database objects defined in the
3
For more on handling tablespace freespace, see the article “Boost SAP
R/3 Performance by Reorganizing Your Oracle Database: A Proven
Reorganization Strategy” in the July/August 2005 SAP Professional
Journal.
56
ABAP Dictionary. To use this check, click on the
Checks button on the Tables and Indexes screen
(Figure 4), which takes you to the Database
Checks screen (Figure 6).
• Missing unique indexes displays database tables
without a unique index. To use this check, click
on the Checks button on the Tables and Indexes
screen (Figure 4), which takes you to the Database
Checks screen (Figure 6).
Check results are displayed in a top, middle, and
lowest hierarchy, as described in Figure 7.
Extents affect the way the tables and indexes are
growing, and they can pose performance issues. To
analyze the extent structure, click on Checks in the
Database system section of the Tables and Indexes
screen (Figure 4), which takes you to the Database
www.SAPpro.com ©2006 SAP Professional Journal. All rights reserved.
 Best practices for performance-tuning SAP R/3 and Oracle database configurations: Part 2

Hierarchy level Result

Top Distinction between the objects defined in the ABAP Dictionary and the objects defined in the
database.

Middle Subdivision by object type (for example by table, view, or index) with the number of missing
objects displayed.

Lowest The names of the individual objects displayed.

Figure 7 Consistency check results hierarchy

Reorganizing the database

An additional area to consider when performance tuning is reorganizing your database. Reorganization is
needed when you find database storage issues such as:
• Internal database fragmentation leads to performance degradation. You need to defragment the database
and merge all the small freespace segments into larger ones.
• Your configuration has reached the limit for the maximum number of extents per table or per index. You
need to merge all the data from the object and compress all the extents into a single extent.
• You have reached the maximum number of files supported per database. You need to consolidate your
tablespaces in to a smaller number of files.
A detailed look at reorganizing Oracle databases is provided in the article “Boost SAP R/3 Performance by
Reorganizing Your Oracle Database: A Proven Reorganization Strategy” in the July/August 2005 SAP
Professional Journal.

Checks screen (Figure 6). The Check for reorganiza- job. The MAXEXTENTS value for SAP objects
tions section provides the following options: is usually set to 300. This is a soft limit and not
necessarily what is allowed by Oracle. Any table
• Extents of tables and indexes displays all objects
or index whose MAXEXTENTS value comes
with more than 10 extents. Alongside the size of
close to this limit may result in a problem such as
the objects (in kilobytes and blocks), the number
abnormal transaction termination. The solution is
of used extents and the value defined for the object
to monitor and increase the MAXEXTENTS value
for MAXEXTENTS are displayed. The DBA
for the particular table or index. If you reach
should check these details regularly to avoid
the maximum value allowed by Oracle for the
possible problems with Maximum Number of
MAXEXTENTS, your only option is object
Extents. The value for the NEXT storage param-
reorganization.
eter is also listed. If you want, you can start a
detailed storage space analysis as a background • Extents per tablespace allows you to analyze the

No portion of this publication may be reproduced without written consent. 57

SAP Professional Journal • September/October 2006

Figure 8 Tablespace history

extent structure for all tablespaces. In addition to reaching the limit for the number of extents set
this list output, a second overview displays all in the MAXEXTENTS parameter. You can also
tables and indexes with more than four extents. display a detailed history of an object (Figure 8)
Alongside the size of the objects (in kilobytes and by selecting the Detailed analysis button (on the
blocks), the number of used extents and the value DB02 transaction screen) in order to examine the
defined for the object for MAXEXTENTS are also selected tablespace in more detail.
displayed. If you want, you can start a detailed
storage space analysis as a background job.
• Check next extent size displays all objects that
have exhibited critical growth within the last four
weeks. This allows you to trace the growth in
number and size of extents. You can immediately Putting the database analysis
see the size of the first extent and the NEXT value
defined for the object. You will also find details on
techniques to use
how close the number of extents has come to Figure 9 gives you an idea of how much you can do

58 www.SAPpro.com ©2006 SAP Professional Journal. All rights reserved.

 Best practices for performance-tuning SAP R/3 and Oracle database configurations: Part 2
Figure 9 Database Performance Analysis: Oracle Database Overview
with the database analysis techniques I’ve just demon-
strated. You can see most of the useful options for
getting more detailed information on the database
activity, resources consumed, analysis of exception
conditions, and database configuration and perform-
ance. To access the screen in Figure 9, follow the
menu path Tools → Administration → Monitor →
Performance → Database → Activity.
No portion of this publication may be reproduced without written consent.
Summary
Parts 1 and 2 of this series have given you enough
information to plan and implement your own data-
base performance-tuning activities and routines.
I’ve described the available SAP and Oracle tools,
provided configuration advice for both systems,
and explained the types of performance issues you
59
SAP Professional Journal • September/October 2006
are likely to encounter in your production environ-
ment. In the final installment of this three-part
article series, I’ll review potential issues you may
encounter when running SAP R/3 in an HP Tru64
Unix environment and how they might affect database
performance.
60
Acknowledgements
I am grateful to my colleagues George W. Anderson,
Laurence Oberman, Rick Heile, Jeff T. Parker, Claus
Vestergaard, and Richard White for their valuable
review and input on this article.
www.SAPpro.com ©2006 SAP Professional Journal. All rights reserved.
Answers at your fingertips.
Extend your learning throughout the year with a license to SAP Professional Journal Online, where developers
and administrators are never more than an arm's length away from thousands of best practices and step-by-step
tutorials to implement, maintain, and extend SAP R/3 and SAP NetWeaver technology.

www.sappro.com

To learn about licenses for individuals, teams, and entire sites,

visit www.sappro.com.