BRSPACE Online Reorganizations

Top Ten Pitfalls and Problems

Martin Frauendorfer, SAP Active Global Support

martin.frauendorfer@sap.com

November 2009
 Agenda

1. Overview
2. Pitfalls and Problems

© SAP 2009 / Page 2

 Overview (1)

Reorganizations in general

A reorganization in a database context is the recreation or

cleanup of one or several segments.
Several types of reorganization exist:
Index reorganization (e.g. via REBUILD or COALESCE), mainly used to
reduce index fragmentation
Table reorganization with tools like:
– BRSPACE (online, offline, export / import)
– Oracle tools (DBMS_REDEFINITION, MOVE, EXP / IMP, EXPDP /
IMPDP, …)
– SAP functionality (ICNV, SE14, …)
Tablespace reorganization (reorganization of all tables in the tablespace
and recreation of the tablespace)
In this presentation we concentrate on the BRSPACE online
reorganization of tables (which also includes tablespace
reorganizations).

© SAP 2009 / Page 3

 Overview (2)

In which situations can a table reorganization be useful?

Reduction of table size

Cleanup of table fragmentation
Change of table storage parameters (e.g. INITRANS)
Move of table to a different tablespace
Reduction of chained and migrated rows
Reduction of number of allocated extents
Transition from LONG to LOB columns (Oracle >= 10g)
Reduction of hot spots on disk level
Activation of Transparent Data Encryption
Change of table structure

© SAP 2009 / Page 4

 Overview (3)

In which situations can a tablespace reorganization be useful?

Activation of LMTS
Activation of ASSM
Cleanup of freespace block corruption
Reduction of database size
Reduction of backup size
Reduction of filesystem size
Reduction of fragmentation
Reduction of number of datafiles
Increase of number of datafiles
Transition to new tablespace layout
Other change to tablespace layout
Optimization of filesystem layout

© SAP 2009 / Page 5

 Overview (4)

How does a BRSPACE online reorganization work?

Determination of CREATE commands for target table and indexes based
on DBMS_METADATA.GET_DDL
Creation of target table with naming convention <source_table>#$
Export of source table CBO statistics based on
DBMS_STATS.EXPORT_TABLE_STATS
Copy of source table data to target table based on
DBMS_REDEFINITION.START_REDEF_TABLE. This package will implicitly
create a materialized view log MLOG$_<source_table> to track all
changes that are performed on the source table while the reorganization
is running.
Creation of target indexes with naming convention <source_index>#$
Import of source table CBO statistics to target table based on
DBMS_STATS.IMPORT_TABLE_STATS
Finalizing of the online reorganization using
DBMS_REDEFINITION.FINISH_REDEF_TABLE. In this step all changes in
the materialized log are applied to the target table and the names of
source and target table are exchanged.
The source table is dropped.
The „#$“ suffix of the target table indexes is removed.
© SAP 2009 / Page 6
 Overview (5)

What are the advantages of a BRSPACE Online Reorganization?

It uses the advantages of DBMS_REDEFINITION:

The reorganization can be performed in parallel to production operation.
This means the table is accessible and records can be inserted / updated /
deleted.
If anything goes wrong during the reorganization the source table still
exists unchanged and no restore / recovery is necessary.
Compared to an offline reorganization via export / import all data has to be
copied only once and not twice
It provides possibilities for client and server side parallelism in
order to speed up the reorganization if sufficient system
resources exist and the reorganization should finish within a
specific time window.
It has already proven to be a reliable reorganization tool at many
customers.
It performs a lot of useful activities implicitly that would require a
lot of manual efforts and impose the risk of errors if
DBMS_REDEFINITION is used directly.
© SAP 2009 / Page 7
 Overview (6)

What are the restrictions of a BRSPACE Online Reorganization?

It is only available as of Oracle 9i.

It cannot handle tables with LONG or LONG RAW columns. As of
10g it is possible to convert them during an online reorganization
on the fly to LOB columns (if SAP >= 6.40 is used).
Temporarily twice the space is needed because both source and
target table and indexes exist in parallel.
If a table has no primary key constraint (or no unique index with
NOT NULL columns) a ROWID based online reorganization is
necessary that requires overhead (ROWID index, ROWID column).
The setup of the materialized view is an overhead that can
negatively impact the performance of the reorganization of many
very small tables.
Structure changes on the source table and its indexes must not
be performed during the reorganization (as they would be lost).

© SAP 2009 / Page 8

 Overview (7)

How is a normal BRSPACE Online Reorganization started?

Table reorganization:
brspace -f tbreorg -t <table_name>
Tablespace reorganization:
brspace -f tbreorg -s <source_tsp> -t "*" -n <target_tsp>
Additional useful options exist, e.g.:
-p <degree>
BRSPACE parallelism degree
-e <degree>
PX parallelism degree
-l <category>
Initial extent size category
-i <target_ind_tsp>
Definition of target index tablespace (if different from target table
tablespace)
For more detailed information see note 646681 and the BRSPACE
online documentation.

© SAP 2009 / Page 9

 Pitfalls and Problems - Overview

Top Pitfalls and Problems

1. What is a good parameter and redo log configuration?
2. What has to be considered for PSAPTEMP?
3. What about PSAPUNDO?
4. And why is the DEFAULT tablespace sometimes used significantly?
5. Why do tables sometimes grow during reorganization?
6. What about SAP technical settings?
7. Is it useful to convert from LONG to LOB?
8. What is a good way of parallelism?
9. Why should I look for invalid objects?
10. Why does a DROP sometimes take ages?
© SAP 2009 / Page 10
 Pitfalls and Problems - 1. Configuration

1. Database Configuration
A good online reorganization performance needs a slightly
different database configuration than normal system operation.
Oracle parameters:
DB_FILE_MULTIBLOCK_READ_COUNT >= 128
Fast full table scan on the source table when creating the target table
PGA_AGGREGATE_TARGET as large as possible
Fast index creations in PGA with minimum PSAPTEMP accesses
If Oracle parallel execution is used:
– PARALLEL_EXECUTION_MESSAGE_SIZE >= 16384
– PARALLEL_MAX_SERVERS >= 2 * BRSPACE parallelism degree *
Oracle parallelism degree
– PARALLEL_ADAPTIVE_MULTI_USER = FALSE
Otherwise Oracle may reduce the desired parallelism degree
– SESSIONS, PROCESSES: Sufficiently large to cope with the
increased PARALLEL_MAX_SERVERS value
DB_CACHE_SIZE >= 1 GB
Large buffer pool doesn't provide significant advantage
© SAP 2009 / Page 11
 Pitfalls and Problems - 1. Configuration

1. Database Configuration (2)

Redo Log Configuration
Usually all changes of an online reorganization are recorded in the redo
logs, so there can be a significant amount of additional redo logs
generated
With current BRSPACE patches you can use the option "-NBR" to create
the target indexes with NOLOGGING
No redo log information is created for the indexes
Theoretically it is possible to switch off logging for the target table, too.
Almost no redo log information created at all
Dangerous, because in case of recovery across the reorganization time
the table data is lost!
Only useful during a dedicated reorganization time window and with a
good backup after the reorganization
Redo logs should be created large enough to avoid log switches within less
than one minute
Avoids unnecessary checkpoint and log switch load
ARCHIVELOG mode can be disabled in case of a dedicated reorganization
time window and a good backup after the reorganization.
© SAP 2009 / Page 12
 Pitfalls and Problems - 2. PSAPTEMP

2. PSAPTEMP
Problems with PSAPTEMP can result in a long reorganization
runtime or terminations.
PSAPTEMP recommendations:
Define PSAPTEMP as LMTS/T tablespace (see SAP Note 659946) in
order to avoid delays due to space transactions
Make sure that PSAPTEMP is assigned as temporary tablespace to the
SAP user and to SYS.
Make sure that there are no I/O bottlenecks when accessing PSAPTEMP.
PSAPTEMP must be large enough to hold the size of the largest
reorganized index.
If BRSPACE parallelism is used ("-p <degree>" option), PSAPTEMP must
be large enough for the <degree> largest indexes that are reorganized in
parallel.
In case of a sorted reorganization ("-r <index>" option), PSAPTEMP must
be large enough to hold the table data.
Consider AUTOEXTEND or RESUMABLE to be on the save side

© SAP 2009 / Page 13

 Pitfalls and Problems - 3. PSAPUNDO

3. PSAPUNDO
PSAPUNDO is only used to a minor extent:
The data transfer into the target table works with APPEND, so no undo is
generated.
The creation of the indexes also doesn't create undo.
Concurrent DML operations logged in the MLOG$ table and finally applied
to the target table generate undo.
PSAPUNDO recommendations:
Use Automatic Undo Management (AUM, SAP Note 600141)
Usually no increase of PSAPUNDO necessary
Larger PSAPUNDO may be required if reorganization is done in parallel to
significant DML operations (in general not recommended).

© SAP 2009 / Page 14

 Pitfalls and Problems - 4. Default Tablespace

4. Default Tablespace
The default tablespace is used in two situations:
The MLOG$ table with the recorded DML operations is created in the
default tablespace.
If a table has no primary index, a ROWID index (I_SNAP$...) is created in
the default tablespace.
Default tablespace recommendations:
Make sure that a default tablespace with sufficient free space or
AUTOEXTEND is assigned to both the SAP user and SYS.
Use LMTS or make sure that NEXT and MAXEXTENTS is set sufficiently
high in order to avoid max extents errors.

© SAP 2009 / Page 15

 Pitfalls and Problems - 5. Table Growth

5. Table Growth
Normally a reorganization will reduce the table size or at least
keep it on the same level.
Sometimes the size reduction is not as high as expected or there
is even a size increase
The most typical reasons are:
The reorganized table has no primary index.
– In this case a ROWID based reorganization rather than a PK
constraint reorganization is done and a ROWID column has to be
added to the target table. This column is hidden after the
reorganization, but it still allocates space ( table increase up to 30
%)
– Typically happens with SAP conversion tables (QCM tables) and BW
fact tables
– Should be avoided whenever possible (e.g. cleanup of QCM tables,
MOVE based reorganization of BW fact tables)
A large INITIAL_EXTENT value is used
– E.g. because of a former SAPDBA reorg with Compress Extents
– Use BRSPACE option "-l 2" to set the initial extent to 64K.
© SAP 2009 / Page 16
 Pitfalls and Problems - 6. Technical Settings

6. Technical Settings
The SAP technical settings contain a data class (TABART)
(TABART) that
maps a table to a tablespace:
Table TABART Tablespace

DBSTATC SPROT PSAPPROTD

DD09L TAORA /
IAORA
The TABART is used whenever a table or index is newly created
(e.g. during a transport of a new table or new index or during a
system copy). If an existing table or index is modified, the current
tablespace is used regardless of the TABART.
Whenever you move tables and indexes to different tablespaces,
you should consider the TABART settings.
Otherwise segments may be created in wrong tablespaces.
BRSPACE is TABART aware.

© SAP 2009 / Page 17

 Pitfalls and Problems - 6. Technical Settings

6. Technical Settings (2)

Recommendations:
Use the "-l" option to define an appropriate existing or new TABART
whenever creating a new tablespace using "brspace -f tscreate":
– "-l <tabart>": New tabart <tabart>
– "-l all": All standard TABARTs
– "-l <source_tsp>": TABART of tablespace <source_tsp>
Maintain TABARTs also in the other systems of the system landscape.
TABART can point to different tablespaces in different systems.
Follow SAP Notes 778784 and 777615 in order to avoid problems during
system copies.
In BW TABARTs are also maintained in special BW tables like RSDCUBE,
RSDODSO, RSTS, RSDS and RSDCHABAS. These entries also have to
be adapted. See SAP Note 771191 for more information.

© SAP 2009 / Page 18

 Pitfalls and Problems - 7. LONG2LOB

7. LONG2LOB Conversion
Tables with LONG and LONG RAW columns can't be reorganized
online - unless an implicit LONG2LOB conversion is performed
(BRSPACE parameter "-a long2lob")
In general it is useful to convert to LOB columns, but there are
some restrictions:
Oracle >= 10g, SAP >= 6.40
ORA-00932 errors possible on SAP side (for bugfix see SAP Note
1223716)
ORA-00600 errors possible if PX is used (for bugfix see SAP Note
1292525)
High water mark enqueues possible with LOBs in ASSM (for bugfix see
SAP Note 1166242).
Several LOB related DBSL problems in SAP 7.1x (for bugfixes see SAP
Notes 1329293, 1269704, 1257979, 1226314 and 1171043)
Small performance impact due to LOBs reported (see SAP Note 835552),
can usually be neglected

© SAP 2009 / Page 19

 Pitfalls and Problems - 8. Parallelism

8. Parallelism Settings
Two possibilities of parallelism:
"-p <degree>": BRSPACE parallelism degree, reorganization of multiple
different tables in parallel
"-e <degree>": Oracle parallel execution (PX) degree, several Oracle PX
slaves work on the reorganization of one table in parallel
Reorganization in parallel to production operation
System resources (I/O, CPU) mustn't be exceeded.
Time is less critical
Use no or a rather small parallelism setting (e.g. "-p 2 - e 2")
Reorganization in a dedicated time window with low or no parallel
production operation
Use the system resources (I/O, CPU) as good as possible.
Often I/O is the bottleneck and not CPU, but I/O capacities are often hard
to determine.
Starting point can be e.g. "-p 6 -e 4" for tablespace reorganizations or "-e
8" for individual table reorganization

© SAP 2009 / Page 20

 Pitfalls and Problems - 9. Invalid Objects

9. Invalid objects
Different kinds of invalid objects can result in errors and
terminations during online reorganizations.
Therefore the following should be checked in advance:
Make sure that no index or index partition of the tables to be reorganized is
in status UNUSABLE. Perform a rebuild if required.
Make sure that no invalid triggers exist on the tables to be reorganized
(e.g. /BIC/05… BW triggers). Drop or validate these triggers.
Make sure that no SYS objects exist in the Oracle DDIC with status =
INVALID. Particularly important are packages like DBMS_REDEFINITION,
DBMS_METADATA, DBMS_STATS and underlying objects.
Make sure that the source tables are consistent (SAP Note 23345). As an
example the REBUILD ONLINE bug described in SAP Note 1413928 can
result in ORA-01452 („cannot CREATE UNIQUE INDEX; duplicate keys
found“) errors when creating the unique index on the target table and the
whole table reorganization is terminated.

© SAP 2009 / Page 21

 Pitfalls and Problems - 10. DROP Performance

10. DROP Performance

Usually it is a matter of seconds (or less) to drop a table or an
index.
But: Under certain circumstances dropping the source table (or
the source tablespace) can take longer than the whole
reorganization. In real life we have already seen situations where
a DROP of a single table took several days!
The reason are typically space transactions in dictionary
managed tablespaces - every formerly allocated extent needs to
be freed individually. Due to an inadequate Oracle index design
on the DDIC tables FET$ and UET$ the cleanup has a square
complexity (4 times higher runtime for twice the number of
extents).
Even worse: As space transactions require the unique space
transaction (ST) enqueue, other extent operations in the system
may be blocked for a long time.

© SAP 2009 / Page 22

 Pitfalls and Problems - 10. DROP Performance

10. DROP Performance (2)

Recommendations
Check for critical segments in dictionary managed tablespaces with more
than 5000 extents. The higher the number of extents, the more critical.
More than 100.000 extents is definitly a big problem.
If segments are returned, you have several options:
– Reduce the number of extents in smaller portions in advance (e.g. by
copying and truncating the table with REUSE SPACE and doing
ALTER TABLE … DEALLOCATE UNUSED KEEP <size> with <size>
decreasing over time)
– Reorganize the critical tables individually during certain defined
timeframes.
– Test how long the DROP takes and provide a sufficient time window
for the reorganization in production.
– In extreme situations it can even make sense to recreate the database
with LMTS using R3load.
See SAP Note 745639 (section TYPE = ST) for more information.

© SAP 2009 / Page 23

 Thank you!

© SAP 2009 / Page 24

 Copyright 2008 SAP AG
All rights reserved

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed
without prior notice.
Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.
SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, Duet, Business ByDesign, ByDesign, PartnerEdge and other SAP products and services mentioned herein as well as their
respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned and
associated logos displayed are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

The information in this document is proprietary to SAP. This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document
contains only intended strategies, developments, and functionalities of the SAP® product and is not intended to be binding upon SAP to any particular course of business, product strategy,
and/or development. SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or
other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of
merchantability, fitness for a particular purpose, or non-infringement.
SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation
shall not apply in cases of intent or gross negligence.
The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in these
materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages

Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch
SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden.
Einige von der SAP AG und deren Vertriebspartnern vertriebene Softwareprodukte können Softwarekomponenten umfassen, die Eigentum anderer Softwarehersteller sind.
SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, Duet, Business ByDesign, ByDesign, PartnerEdge und andere in diesem Dokument erwähnte SAP-Produkte und Services
sowie die dazugehörigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und in mehreren anderen Ländern weltweit. Alle anderen in diesem Dokument erwähnten
Namen von Produkten und Services sowie die damit verbundenen Firmenlogos sind Marken der jeweiligen Unternehmen. Die Angaben im Text sind unverbindlich und dienen lediglich zu
Informationszwecken. Produkte können länderspezifische Unterschiede aufweisen.

Die in diesem Dokument enthaltenen Informationen sind Eigentum von SAP. Dieses Dokument ist eine Vorabversion und unterliegt nicht Ihrer Lizenzvereinbarung oder einer anderen
Vereinbarung mit SAP. Dieses Dokument enthält nur vorgesehene Strategien, Entwicklungen und Funktionen des SAP®-Produkts und ist für SAP nicht bindend, einen bestimmten
Geschäftsweg, eine Produktstrategie bzw. -entwicklung einzuschlagen. SAP übernimmt keine Verantwortung für Fehler oder Auslassungen in diesen Materialien. SAP garantiert nicht die
Richtigkeit oder Vollständigkeit der Informationen, Texte, Grafiken, Links oder anderer in diesen Materialien enthaltenen Elemente. Diese Publikation wird ohne jegliche Gewähr, weder
ausdrücklich noch stillschweigend, bereitgestellt. Dies gilt u. a., aber nicht ausschließlich, hinsichtlich der Gewährleistung der Marktgängigkeit und der Eignung für einen bestimmten Zweck
sowie für die Gewährleistung der Nichtverletzung geltenden Rechts.
SAP übernimmt keine Haftung für Schäden jeglicher Art, einschließlich und ohne Einschränkung für direkte, spezielle, indirekte oder Folgeschäden im Zusammenhang mit der Verwendung
dieser Unterlagen. Diese Einschränkung gilt nicht bei Vorsatz oder grober Fahrlässigkeit.
Die gesetzliche Haftung bei Personenschäden oder die Produkthaftung bleibt unberührt. Die Informationen, auf die Sie möglicherweise über die in diesem Material enthaltenen Hotlinks
zugreifen, unterliegen nicht dem Einfluss von SAP, und SAP unterstützt nicht die Nutzung von Internetseiten Dritter durch Sie und gibt keinerlei Gewährleistungen oder Zusagen über
Internetseiten Dritter ab.
Alle Rechte vorbehalten.

© SAP 2009 / Page 25