IBM

IBM InfoSphere MDM Server

Best Practices
MDM System Performance Testing
- Guidelines and Instructions

Yongli An Manager, MDM Server Performance and Benchmarks

IBM InfoSphere MDM Server Table of Contents

Table of Contents
Table of Contents ................................................................................................. 2 Executive Summary............................................................................................. 3 Introduction Understand and Plan Performance Testing........................... 4 Recommendations for a Repeatable Performance Test Process.................... 7 High Level Recommendations..................................................................... 7 High Level Test Process ................................................................................ 9
Pre Test Steps ................................................................................................................9 Test Execution Steps ..................................................................................................10 Post Test Execution Steps ..........................................................................................11

Performance Data Collection Instructions ..................................................... 12 MDM / WebSphere Application Server(WAS) ........................................ 12 DB2 for z/OS Server ..................................................................................... 14 DB2 for AIX/Linux ....................................................................................... 14 References ........................................................................................................... 15 Acknowledgments ............................................................................................. 16 Appendixes ......................................................................................................... 16 Appendix #1 Dynamic SQL Statement Statistics.................................. 16 Appendix #2 Sample Approach for DB2 Snapshot Data Collection . 18 Appendix #3 Sample Run README Document Template ................ 20 Notices ................................................................................................................. 22 Trademarks ................................................................................................... 23

IBM InfoSphere MDM Server Executive Summary

Executive Summary
An MDM implementation usually faces strict performance and scalability requirements because it is a mission critical system in the center of the enterprise level IT systems. Being the cornerstone of such a MDM system, InfoSphere Master Data Management (MDM) Server has been designed with high performance and scalability requirements since its inception and has been thoroughly tested as part of the rigorous development cycle in each release. However, as with any other software project, the performance characteristics of the implementation using MDM server should also be ascertained by following a similar rigorous performance engineering process and this needs to be done as part of the implementation project. MDM server can be used and customized in so many different ways that these customizations may have an overall affect on the overall performance of the system. Customizations may be required for a number of reasons and are based on a set of specific customer business requirements including business functional requirements, integration requirements with other systems, and different data characteristics of customer data. Taken together these may all impact the behavior and performance of the system differently and may require different implementation design decisions. For any MDM implementation, performance must be forethought and not an afterthought. While there are many areas and opportunities to improve performance, you should follow best practices during the design phase long before starting the actual performance testing of your implementation. Please see the resources section for further reading on this particular topic. This paper focuses on the best practices for performance testing and this is necessary throughout the implementation cycle and is critical to the success of your MDM system in production. At the end of the implementation cycle, you will still always need to do the final performance testing and gather performance metrics properly to ensure the implementation performs to meet your SLA (Service Level Agreement) before going into production. Ongoing regression testing is also needed as the implementation evolves. While this document is an MDM Server best practices document, many of the best practices presented are also more general performance testing best practices.

IBM InfoSphere MDM Server Introduction Understand and Plan Performance Testing

Introduction Understand and Plan Performance Testing

InfoSphere MDM Server was originally architected with performance and scalability as a key requirement given the demands of the various styles of MDM implementations and specifically the centralized (or transactional) style. It is also important that implementations also address performance proactively. This article details the best practices on how to conduct performance testing with your MDM system with a consistent and repeatable process. The field of performance testing has several sub-genres covering different aspects of performance testing and serving different purposes. The most common type of performance testing is load testing - either for the purpose of characterizing the performance behavior of the system under test or for the purpose of understanding the current bottlenecks in the system under test or both. In addition, other forms of performance testing such as configuration testing (to validate the performance impact of a particular configuration change) or isolation testing (to reproduce a problem to confirm or investigate the cause) are also important. Please refer the links in the references section for additional reading on these topics. Regardless of the motivation for performance testing, it is important to have a rigorous performance test process that provides consistency and repeatability. Doing so will provide the implementation team with a high level of confidence in the data that has been collected that will be subjected to further analysis. For a test that will be used as an important baseline or reference point down the road, it is also important to have a clearly defined set of minimum required performance data that needs to be collected from each performance test. The objectives of any planned performance tests must be clearly stated and documented. Not only will this avoid confusion and unnecessary debate but it also to helps clarify the set of performance data to be collected. It is strongly recommended to start performance testing early in the development cycle to reduce costly remedies of performance issues that are identified late in the cycle.

Best Practice: Clarify and document the performance objectives and performance test data collection for key usage scenarios, and start performance testing early in the implementation cycle

For MDM systems, performance goals may be defined based on a few key performance metrics that include:

IBM InfoSphere MDM Server Introduction Understand and Plan Performance Testing a) throughput - for example, the number of Transactions Per Second (TPS), b) MDM service response time, and c) resource utilization of the involved servers (CPU, RAM and I/O). These performance goals are only valid in the context of a specific workload on a specific system configured in a certain way. Consequently, it is important to clearly document a matrix that identifies the hardware being used for a particular workload as part of the performance testing tasks. In the context of performance testing a MDM system, this document provides guidelines and instructions on the following topics 1. 2. How to establish a consistent and repeatable performance test process What data, and with what process data should be collected

These are general best practices recommended both to our customers and services people in the field working on MDM server solutions. First, a repeatable performance testing process should be established in the early stage of the MDM solution implementation phase. With this repeatable testing process, major code changes should be tested and evaluated against a baseline in order to determine their performance impact during the journey to get the system into production. It has been proven many times that proper planning for performance testing and thorough execution of such a performance test plan early in the cycle is a must to ensure good performance and scalability of any implemented MDM system. Equally important is having proper data collection in order to ensure that the right performance metrics are collected and compared. A repeatable process and the right set of data are required to properly evaluate the impact of the changes and accurately quantify and characterize the level of performance of the system. Good source code and change control is also required. Using source control will allow the team to go back to the performance tests for a particular version of the implementation thereby facilitating the performance problem investigation process. Recognize that performance testing is an iterative process that requires multiple iterations of the same, or similar, steps until the defined performance goals are met. Change control applies to all elements of the system, including the servers, storage, network, Operating System and database software, as examples.

Best Practice: Establish a consistent and repeatable performance testing process that includes proper and sufficient performance data collection, a dedicated or controlled performance testing environment, and good source code and change control

IBM InfoSphere MDM Server Introduction Understand and Plan Performance Testing

Using a disciplined approach, as described above, also means better efficiency and productivity by avoiding unnecessary test repetition allowing the project to progress more quickly. This document provides a set of commonly used data collection instructions which can be tailored to serve a specific purpose. With minor exceptions you can consider the list as a mandatory minimum set of data to collect. Even if you dont think you have the need of a particular piece of data we strongly encourage you to keep them as part of data collection because more often than you think, later on you will come back to check certain things and find the collected data has become a reference point. The suggested data collection typically has a very minimal performance impact. It is also strongly recommended to have a performance data repository to keep the performance data from all the milestone deliveries of the implementation with properly labeled names for the tests. This historical performance information becomes very handy when trying to understand performance behavior changes or when investigating a performance problem.

Best Practice: Always collect the recommended minimum set of data required for monitoring each layer of the whole MDM stack

IBM InfoSphere MDM Server Recommendations for a Repeatable Performance Test Process

Recommendations for a Repeatable Performance Test Process

This section will cover high level recommendations and a step by step description to help you achieve a repeatable performance test process.

High Level Recommendations

At a very high level, the following is required in order to get comparable and consistent results (i.e. repeatability): 1) Dedicated test environment During the test period, a dedicated test environment is strongly recommended. The resources allocated to LPARs must be dedicated and fixed. In cases where this requirement can not be fully met, the implications need to be fully understood by repeating the same tests and quantifying the run variation. The more variation that is observed the greater the number of runs required to understand what is significant. 2) Same starting point or starting state The system under test should be able to be restored back to the same state before starting next test (e.g. by restoring the DB, restarting the application server, etc. ). Be sure to verify that the database backup contains current and optimal indexes and statistics to facilitate good access path selection. 3) Same test execution (steps) To establish comparability and consistency before evaluating any changes, the same test should be done using the same input data, running the same test cases, going through the exactly the same steps (e.g. warm up, and steady phase) with the same data collection traces or logging enabled. Once such repeatability is established, the system is good to evaluate the targeted changes with all the other parameters and variables being kept the same 4) Real or realistic data in enough volume Using production data or data that is derived from production data and has very similar characteristics is strongly recommended. The size of the test data volume for preproduction performance validation purpose should be at least one third of production data volume. The closer you get to using the production data is the lower the risk of seeing unexpected, undesirable performance behavior in production. If completely synthetic data is being used then you need to verify that that the data characteristics of

IBM InfoSphere MDM Server Recommendations for a Repeatable Performance Test Process the synthetic data set are similar to production data, while at the same time ensuring that primary key, foreign key and domain constraints are met. 5) Sufficient warm up Performance measurements should only be taken after a sufficient warm up phase has been completed. This phase needs to be long enough to allow the workload to reach a stable steady state (a nearly flat throughput curve). A warm up of 20 minutes is a good starting point; a bigger system and higher throughput will require a longer warm up. The best way to determine a suitable warm-up period is to do a set of long runs and look for a repeatable inflection point where throughput has risen to a flat or nearly flat line. 6) Long enough steady state The number of transactions or data to drive the test should be sufficient to last long enough for stable and consistent performance. Once in steady state, a performance measurement phase of 30 minutes is a good starting point. In other cases, especially when there is high workload variation, longer measurement phase is required for better consistency and comparability. 7) A run README document A run README document/directory should be created for each test describing the test with some basic run information. This document records the key run information including the time of the run, duration of the run, purposes of the run, software settings and hardware configurations, application version being used, performance metrics (e.g. average Response Time, Transaction Per Second), and so on. A template of such document should be developed and ready for use prior to the start of testing (please see Appendix #2 below for an example) 8) Test repeatability confirmation The repeatability of the tests in the performance test environment should be established and confirmed. To establish repeatability, or to understand the run to run performance variation in your test environment, its strongly recommended to run the same test multiple times at different times of the day or on different days, without any changes, then compare the key performance metrics. The greater the variability the more runs need to be made but there should be at least 3 runs and as many as 7-9 so the high/low results can be discarded. The collected performance metrics should include average transaction Response Times, Transactions Per Second, CPU and I/O utilization on both database and MDM server machines, and MIPS consumption in case of any databases on System z. Repeatability confirmation may need to be done again when there are changes in the test environment causing any doubt about such repeatability. 9) Proper source and change control

IBM InfoSphere MDM Server Recommendations for a Repeatable Performance Test Process Source code control is important to allow a performance test to be able to go back to a particular version of the application code and the performance test. So both the application and the test harness need to be managed. Change control (in the test environment) is also necessary to help ensure the tests are valid and performance impact is from the targeted changes only.

Best Practice: Allocate a dedicated or well controlled environment Follow exactly the same test steps Start from the same state Use production data or alike in a volume close to production Ensure sufficient warm up before take measurements Take long enough performance measurements during steady stage Confirm repeatability of the tests Enforce proper source and change control Document each test with a run README file

High Level Test Process

Before a test can start or later be repeated whenever needed, several key steps should have been completed. Such steps include identifying the performance goals, arranging a dedicated test environment, and creating/verifying the test workload. Once these prerequisites are satisfied, next comes the actual test execution. This includes pre-test preparation, test execution, and post-test execution. Consider the following as a concrete example of how a performance test may be conducted.

Pre Test Steps

1) Stop the MDM server instance 2) Restore the MDM server database The MDM database is restored from a backup that has been done from a healthy and tuned database that is considered as a baseline database. 3) Verify the key settings Make sure the settings (logging and/or tracing) are set at the desired levels in the property files or configuration and ensure sufficient disk space to record all events even when detailed level of logging is enabled. For example, in the case of a typical stack, the following settings need to be checked: a) MDM server settings

IBM InfoSphere MDM Server Recommendations for a Repeatable Performance Test Process b) WAS settings c) DB2 settings d) OS level monitoring 4) Clear or reset files, directories, metric counters and tables used for logging purpose Examples are: a) Application server side WAS/MDM log and trace directories or files should be reset or cleared for each new run. Old files or content should be backed up if needed) b) Any tables in DB2 used solely for logging purpose for each run should be cleared by issuing delete * from so the table will only contain the needed data for the coming test; one example is the table created for dynamic statement cache statistics collection on DB2 for z/OS c) Some resettable DB2 snapshot counters should be reset using db2 reset command 5) Restart the MDM application instance 6) Set up the driver tier The driver tier should be set up with the same chosen test case (including the right input data) with the same chosen load level (e.g. Same # of submitters as specified in batch processor, unless this is the variable you are trying to test) 7) Prepare the monitoring scripts/tools Get the monitoring scripts or tool ready for any required monitoring and data collection Note: for detailed data collection instructions, please refer to the data collection instruction section later in this document.

Test Execution Steps

1) Start monitoring Start any monitoring scripts or tool that require a manual start right before the warm up run. For any OS level monitoring (e.g. AIX or Z/OS when using DB2 for zOS), we recommend to start monitoring about 3 minutes before starting the warm up phase so we have a baseline of OS level performance metrics when the system is considered 100% idle (or as a way to show how idle the system is) before starting the test 2) Start the warm up Run a fixed number of transactions or run transactions for a period of time to reach steady state. A warm up of 20 minutes should be a good starting point; a bigger system and higher throughput will need a longer warm up. The required warm up period should already have been determined during the preparation phase by plotting the throughput curve during trial runs. 3) Start the steady state measurement run Run or continue to run a fixed number of transactions or alternatively run transactions

10

IBM InfoSphere MDM Server Recommendations for a Repeatable Performance Test Process for a specified period of time. Once in steady state, establish a performance measurement phase. Thirty minutes is a good starting point. In some cases, especially if you have a highly variable workload, a longer measurement phase may be required for better consistency and comparability. Notes: a) You dont need to have a stop between warm up and steady state if you are using the same workload as long the time is recorded correctly in the run README document to tell the start and end time for each phase. b) If different techniques (other than running the same workload) are used to warm up the system, document clearly how it was done in the run README document and explain why you think such warm up is sufficient.

Post Test Execution Steps

1) Record the time and duration of the test, as well as any performance metrics (eg. TPS) reported from the test driver tool used in the test 2) Collect all the log, trace and monitor data files from various locations 3) If any software specific data processing reports need to be generated, generate the reports per the software instructions. E.g., DB2 account reports on Z/OS need to be generated based on the run duration 4) Complete the run README document for this test run. A template of this document should already be available and maintained. 5) Package all of the above assets in a directory and/or zip file. The packaged data collection is now available to be sent to the requester or the right team for analysis, or stored in the performance data repository as a historical test record. Ideally a performance summary report should be prepared after the analysis is done and filed to the same performance data repository.

Best Practice: Carefully follow all pre-test, execution and post-test steps, and consciously ensure no other workloads on the system that may impact the performance tests. This will ensure accuracy, repeatability and comparability and guarantees that the data collected is from and only from the monitored run.

11

IBM InfoSphere MDM Server Performance Data Collection Instructions

Performance Data Collection Instructions

This section details the critical performance data that typically needs to be collected from load testing - also known as a performance measurement run, or from configuration/isolation testing - a run for the purpose of investigating a particular problem. It is not always necessary to collect all of the data listed below. Some items need to be collected only when being specifically requested. Those items are marked so accordingly below. For those not marked so, its strongly recommended to collect them for each test run. Please note that the instructions do not currently cover all the supported platforms and associated software stacks. In fact, the document only covers most common combinations. For other supported software not covered here, similar concepts should apply. Please refer to the product manuals of that particular software product.

MDM / WebSphere Application Server(WAS)

Software stack version information List of transactions used for the run and their frequency of execution MDM and application logs collected (e.g., all logs generated in MDM and WAS log directories) Java Virtual Machine (JVM) Garbage Collector (GC) trace Ensure verbose GC is enabled to be able to analyze JVM memory usage e.g. when WAS is used, GC trace can be enabled using the WAS admin console. CPU, I/O, and memory utilization on AIX nmon can be used with a measurement interval of 5 seconds. This should be collected from the all MDM instances if multiple MDM instances are used. nmon needs to be installed on the system. If not yet, please install first. The following sample gives 1 hour with snapshots every 5 seconds: nmon fdt s5 c720 or with nohup and & if you intend to log out of the console: nohup nmon fdt s5 c720 &

MDM Performance Monitor data (only when requested for problem determination) Performance Monitor data is suggested to be enabled and set to level 2. This

12

IBM InfoSphere MDM Server Performance Data Collection Instructions should be done only when requested. Normally this is not needed for pure performance measurements. Rather, its more often needed for performance investigation. Enabling MDM Performance Tracking To make use of MDM Performance Tracking you must explicitly enable it. Like most product features in MDM Server, this is accomplished by modifying the Configuration and Management tables, specifically the CONFIGELEMENT table. Use the following commands to enable MDM Performance Tracking at the recommended level for problem diagnosis and tuning:
UPDATE CONFIGELEMENT SET VALUE = 'true', LAST_UPDATE_DT = CURRENT_TIMESTAMP WHERE NAME = '/IBM/DWLCommonServices/PerformanceTracking/enabled'; UPDATE CONFIGELEMENT SET VALUE = '2', LAST_UPDATE_DT = CURRENT_TIMESTAMP WHERE NAME = '/IBM/DWLCommonServices/PerformanceTracking/level'; COMMIT;

Since we are recommending the use of the log4j output as a starting point for problem investigation, it helps to configure log4j optimally to get the right size and roll-over window for the performancemonitor.log. To ensure proper capture of an elusive problem or to simply get a large statistical sample, we recommend dedicating at least 2 GB to the rolling performancemonitor.log files. This is typically enough to capture an hour of workload on an MDM Server in complicated and high-volume implementations. Find the following entries in the log4j.properties file in the MDM Server properties.jar archive, as an example:
log4j.appender.performanceLog=org.apache.log4j.RollingFileAppender log4j.appender.performanceLog.Encoding=UTF-8 log4j.appender.performanceLog.Threshold=ALL log4j.appender.performanceLog.layout.ConversionPattern=%d : %m%n log4j.appender.performanceLog.layout=org.apache.log4j.PatternLayoutlog4j.appender.performanceLog.File=/opt/IBM/MDM 85/MDMlogs/perflogs/performancemonitor.log log4j.logger.com.dwl.base.performance.internal.PerformanceMonitorLog=ALL,performanceLog log4j.appender.performanceLog.MaxBackupIndex=20 log4j.appender.performanceLog.MaxFileSize=100MB...

After both the CONFIGELEMENT and log4j changes, a WebSphere Application Server recycle for the MDM Server instance will be required to enable MDM Performance Tracking. Once the server has recycled, any transactions sent to that MDM Server will be logged by MDM Performance Tracking. WAS Performance Monitoring Infrastructure (PMI) data (when requested for problem determination) PMI data normally needs to be collected on all the MDM instances if more than one is used. Current settings (default/BASIC) are fine. Please start logging just before your test and then stop it afterwards. For more details, see following instructions: WAS PMI should be left at the default level (this has a negligible impact)

13

IBM InfoSphere MDM Server Performance Data Collection Instructions Enable Logging and watch the Tivoli Performance Viewer during the run. Specifically: DWL/Customer datasource pool size DWL/Customer datasource connections ORB Thread count For details on the Tivoli Performance Viewer, please refer to chapter 14.3 in the following IBM Redbook: http://www.redbooks.ibm.com/abstracts/sg246392.html?Open

DB2 for z/OS Server

Local Buffer Pool (BP) size , BP hit ratio, and other db2 info basic accounting class 1,2,3,7,8 and statistics class 1,3,4,5,6 table space, table and index definitions and disk layout description I/O details and CPU utilization RMF Summary Report Dynamic SQL statement statistics (only when requested for problem determination) Please see Appendix 1 for details If you are using Data Sharing, record how many members are defined/active Group BP size

DB2 for AIX/Linux

CPU, I/O, and memory utilization nmon can be used with a measurement interval of 5 seconds. This should be collected from the DB2 server where the MDM database resides. nmon needs to be installed on the system. If not yet, please install first. The following sample gives 1 hour with snapshots every 5 seconds: nmon fdt s5 c720 or with nohup and & if you intend to log out of the console: nohup nmon fdt s5 c720 & Database configuration, database manger configuration and other basic information db2support <output path> -d <db_name> -c m type db2support help to learn about other options.

14

IBM InfoSphere MDM Server References DB2 snapshot data using db2 commands or other monitoring tools capture the following information: Buffer Pool sizes, Buffer Pool hit ratios, Database Configuration and Database Manager Configuration settings, table space, table and index definitions and disk layout description. Please find one approach as an example (in Appendix #2) that works for both old and new DB2 versions. You can either follow the steps to do exactly the same, or learn the techniques and ideas and apply them when using different tools. Event monitor log for deadlocks (only when requested for problem determination) Event monitors are used to collect information about the database and about any connected applications when specified events occur. One of such event monitors is deadlock event monitor. When a deadlock occurs, the deadlock event monitor collects information about the applications involved including the SQL statements participating in the deadlock and the locks causing high contention. Deadlock event monitor log can help you to understand and resolve excessive deadlocks that sometimes happen in an MDM Server database. By default, all databases have an event monitor defined named DB2DETAILDEADLOCK, which records detailed information about deadlock events. The DB2DETAILDEADLOCK event monitor starts automatically when the database starts. After formatting the event monitor files, typically you should be able to find all the SQL statements participating in the deadlocks. Please refer to DB2 information center for more details.

References
o o o o o o o o o o o o Best Practices: Implementing InfoSphere MDM Server for high performance Best Practices: Achieving high availability and scalability with IBM InfoSphere Master Data Management WebSphere Customer Center: Understanding performance Loading a large volume of Master Data Management data quickly, Part 1: Using RDP MDM Server maintenance services batch Monitoring and tuning InfoSphere Master Data Management Server, Part 1: Set goals and tune each layer of the infrastructure Monitoring and tuning InfoSphere Master Data Management Server, Part 2: Monitor the DB2 layer and learn about different monitoring tools Master Data Management: IBM InfoSphere Rapid Deployment Package More MDM best practices publications MDM Server 9: Information Center DB2 tuning tips for OLTP applications Patterns & Practices: Performance testing guidance Software performance testing

15

IBM InfoSphere MDM Server Acknowledgments

Acknowledgments
The author would like to thank Berni Schiefer, Distinguished Engineer & Technical Executive of IBM, for his encouragement and guidance on creating this document, David Zhang from DB2 for z/OS performance team for his contribution to the DB2 for z/OS related items, Stephanie Hazlewood from MDM server development, for her thorough review and great editorial work, and the global MDM performance team members for continuously maturing this team development topic and for the feedback on improving this article.

Appendixes

Appendix #1 Dynamic SQL Statement Statistics

Dynamic statement caching feature was introduced in DB2 for z/OS V5. Whenever DB2 prepares an SQL statement, it creates a control structure that is used when the statement is executed. When dynamic statement caching is in effect, DB2 stores the control structure associated with a prepared dynamic SQL statement in a storage pool. If that same statement is executed again, DB2 can use the cached control structure, avoiding the expense of re-preparing the statement. DB2 maintains statement caching performance statistics records when dynamic statements are cached. The statistics include cache hit ratio and other useful data points that help you evaluate the overall performance of your statement caches and statement executions. You may follow the following steps to externalize the statement cache statistics information for your performance analysis: 1) If not created yet, create DSN_STATEMENT_CACHE_TABLE to hold the statistics data by referring to sample job DSNTESC in HLQ.SDSNSAMP. Otherwise skip to next step.
CREATE TABLE DSN8!!0.DSN_STATEMENT_CACHE_TABLE ( "STMT_ID" INTEGER NOT NULL, "STMT_TOKEN" VARCHAR(240) , "COLLID" VARCHAR(128) NOT NULL, "PROGRAM_NAME" VARCHAR(128) NOT NULL, "INV_DROPALT" CHAR(1) NOT NULL, "INV_REVOKE" CHAR(1) NOT NULL, "INV_LRU" CHAR(1) NOT NULL, "INV_RUNSTATS" CHAR(1) NOT NULL, "CACHED_TS" TIMESTAMP NOT NULL, "USERS" INTEGER NOT NULL, "COPIES" INTEGER NOT NULL, "LINES" INTEGER NOT NULL, "PRIMAUTH" VARCHAR(128) NOT NULL, "CURSQLID" VARCHAR(128) NOT NULL, "BIND_QUALIFIER" VARCHAR(128) NOT NULL, "BIND_QUALIFIER" VARCHAR(128) NOT NULL, "BIND_ISO" CHAR(2) NOT NULL,

16

IBM InfoSphere MDM Server Appendixes

"BIND_CDATA" CHAR(1) NOT NULL, "BIND_DYNRL" CHAR(1) NOT NULL, "BIND_DEGRE" CHAR(1) NOT NULL, "BIND_SQLRL" CHAR(1) NOT NULL, "BIND_CHOLD" CHAR(1) NOT NULL, "STAT_TS" TIMESTAMP NOT NULL, "STAT_EXEC" INTEGER NOT NULL, "STAT_GPAG" INTEGER NOT NULL, "STAT_SYNR" INTEGER NOT NULL, "STAT_WRIT" INTEGER NOT NULL, "STAT_EROW" INTEGER NOT NULL, "STAT_PROW" INTEGER NOT NULL, "STAT_SORT" INTEGER NOT NULL, "STAT_INDX" INTEGER NOT NULL, "STAT_RSCN" INTEGER NOT NULL, "STAT_PGRP" INTEGER NOT NULL, "STAT_ELAP" FLOAT NOT NULL, "STAT_CPU" FLOAT NOT NULL, "STAT_SUS_SYNIO" FLOAT NOT NULL, "STAT_SUS_LOCK" FLOAT NOT NULL, "STAT_SUS_SWIT" FLOAT NOT NULL, "STAT_SUS_GLCK" FLOAT NOT NULL, "STAT_SUS_OTHR" FLOAT NOT NULL, "STAT_SUS_OTHW" FLOAT NOT NULL, "STAT_RIDLIMT" INTEGER NOT NULL, "STAT_RIDSTOR" INTEGER NOT NULL, "EXPLAIN_TS" TIMESTAMP NOT NULL, "SCHEMA" VARCHAR(128) NOT NULL, "STMT_TEXT" CLOB(2M) NOT NULL, "STMT_ROWID" ROWID NOT NULL GENERATED ALWAYS, "BIND_RO_TYPE" CHAR(1) NOT NULL WITH DEFAULT, "BIND_RA_TOT" INTEGER NOT NULL WITH DEFAULT ) IN DSN8D!!A.DSN8S!!X CCSID UNICODE;

--------------------------------------------------------------------CREATE THE AUXILIARY TABLE FOR THE SAMPLE DSN_STATEMENT_CACHE_TABLE CREATE LOB TABLESPACE DSN8L!!X IN DSN8D!!A BUFFERPOOL BP32K; CREATE AUX TABLE DSN8!!0.DSN_STATEMENT_CACHE_AUX IN DSN8D!!A.DSN8L!!X STORES DSN8!!0.DSN_STATEMENT_CACHE_TABLE COLUMN STMT_TEXT; CREATE INDEX DSN8!!0.DSN_STATEMENT_CACHE_IDX1 ON DSN8!!0.DSN_STATEMENT_CACHE_TABLE ( "STMT_ID" ASC ); CREATE INDEX DSN8!!0.DSN_STATEMENT_CACHE_IDX2 ON DSN8!!0.DSN_STATEMENT_CACHE_TABLE ( "STMT_TOKEN" ASC ) CLUSTER; CREATE INDEX DSN8!!0.DSN_STATEMENT_CACHE_IDX3 ON DSN8!!0.DSN_STATEMENT_CACHE_TABLE ( "EXPLAIN_TS" DESC );

17

IBM InfoSphere MDM Server Appendixes

CREATE INDEX DSN8!!0.DSN_STATEMENT_CACHE_AUXINX ON DSN8!!0.DSN_STATEMENT_CACHE_AUX;

2) If above table was created in the past, ensure to do the following: delete the content of DSN_STATEMENT_CACHE_TABLE table to clean up any data collected from the previous runs: DELETE FROM DSN_STATEMENT_CACHE_TABLE 3) Stop and start tracing with IFCID 316, 317, 318 to reset all the counters dynamic statement cache statistics so that we don't pick up stats from previous runs STOP TRACE(P) CLASS(30) IFCID(316, 317, 318) START TRACE(P) CLASS(30) IFCID(316, 317, 318) IFCID 316 contains the first 60 bytes of SQL text and execution statistics; IFCID 317 captures the full text of the SQL statement; IFCID 318 is a monitor trace instrumentation identifier; DB2 begins to collect statistics and accumulates them for the length of time when the trace is on. 4) Run the workload Issue statement EXPLAIN STMTCACHE ALL in a DSNTEP2 job to extract all statements from the global cache and to dump the statistics information to DSN_STATEMENT_CACHE_TABLE, begin your evaluation of the statement cache performance by selecting on the inserted rows from the DSN_STATEMENT_CACHE_TABLE table. Important: Stopping the trace will reset all statistics. It is important to complete the above step before turning off the monitor trace.

Appendix #2 Sample Approach for DB2 Snapshot Data Collection

Obviously there are a variety of tools you can use to collect DB2 performance metrics available. For example, you might traditionally rely on snapshot. This approach is still supported and working in recent DB2 versions. However, starting in DB2 Version 9.7, table functions are available as a light-weight alternative to the traditional system monitor. There are also various commercial tools that leverage the same underlying monitoring technology in DB2 for end users to achieve the same monitoring and data collection purpose. Alternatively, the key information above is also available from the reports such as DBSUMMARY, CURRENTSQ, CURRENTAPPS, CONNECTION, and LOCKWAIT, which are generated by the MONREPORT module in DB2 Version 9.7 or later.

18

IBM InfoSphere MDM Server Appendixes Below is one approach as an example in the appendix section that works for both old and new DB2 versions. You can either follow the steps to do exactly the same, or learn the techniques and ideas and apply them when using different tools. For a deeper understanding of a database workload during optimization or problem resolution, try the low-impact investigative technique of taking snapshots in intervals, with monitor resets between each capture. For example, to monitor a one-hour period for a production workload, a script can be easily constructed to do the logic of thread 1 and thread 2 as shown in the thread in Listing 1. Thread 1 in Listing 1 is for taking multiple interval snapshots of 60 seconds each with a reset monitor at the beginning of each interval snapshot.

Listing 1: Thread 1 pseudo code

db2 v update monitor switches using bufferpool on sort on lock on \ statement on table on uow on db2 -v get monitor switches count = 0 while (count < 10) sleep 300 ## not being monitored db2 -v reset monitor all ## clear the counters sleep 60 ## collect data for only 60second interval db2 -v get snapshot for all on <db_name> 2>&1 > snapshot_${count}_withReset.out end while db2 -v update monitor switches using bufferpool off sort off \ lock off statement off table off uow off db2 -v terminate

Thread 2 in Listing 2 is for taking multiple snapshots over the period without issuing the reset monitor command after the initial reset monitor command.

Listing 2: Thread 2 pseudo code

db2 v update monitor switches using bufferpool on sort on lock on \ statement on table on uow on db2 -v get monitor switches db2 -v reset monitor all ## clear the counters once count = 0 while (count < 10) loop sleep 360 ## taking a snapshot every 6 minutes db2 -v get snapshot for all on <db_name> 2>&1 > snapshot_${count}_NoReset.out end while db2 -v update monitor switches using bufferpool off sort off \ lock off statement off table off uow off db2 -v terminate

19

IBM InfoSphere MDM Server Appendixes

Appendix #3 Sample Run README Document Template

This is a sample run README document template. Often, it can be tailored and customized based on your specific need. Table 1 Sample Run README Document Template
Run ID: Start Time Warm up start time Measurement start Application version: Test Environment Warm up approach Date of Test: End Time Warm up end time Measurement end Owner of Test

Test Purpose: o Notes / Key observations: o Average TPS: o Average Response Time o Error/Failure Count: o Type of Failures/Errors o Concerns / Issues

Key Application Server Settings: o Number of application server instances: o JDBC o Prepared statement cache: o EJB cache: o KeepDynamic (enabled?): o JVM heap setting: Min: Max: GC policy: o Database Connection pool setting Min : Max : o Log4J debug settings: o PMI setting: o MDM session count: o DB Connection timeout setting Timeout: Min: Max: Reap: Unused: Aged: o Any other performance monitoring agents / profilers:

Key DB2 Settings: o Local Buffer pool settings/sizes:

20

IBM InfoSphere MDM Server Appendixes o o o o o o Group Buffer pool size: KeepDynamic enabled (yes/no): Number of table partitions (if applicable): Trace /debug setting: Number of connections: Key zParms:

Key MDM Configuration: o Logging level (error, warning, etc): o Performance monitor logging (off, 1, 2, ): o SDP (on/off): o Tail (on/off): o History (on/off): o Householding (yes/no): o Event Manager (yes/no):

System Information: Application Server Configuration LPAR A Hardware Model Number Enabled / Active (yes/no) Max Cores Memory GB Entitlement: Capped (yes / no) MDM version AIX version Database Server Configuration (for AIX on Power) LPAR C Hardware Model Number Enabled / Active (yes/no) Max Cores Memory GB Entitlement: Capped (yes / no) DB2 version AIX version Database Sever Configuration (for z/OS on mainframe) LPAR C Hardware Model Number Physical CEC (mainframe) Enabled / Active (yes/no) Logical Processors (GP) ZIIP Processors ZAAP Processors Memory GB zOS Version DB2 Version Data Sharing (yes/no)

21

IBM InfoSphere MDM Server Appendixes

Notices
This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. Without limiting the above disclaimers, IBM provides no representations or warranties regarding the accuracy, reliability or serviceability of any information or recommendations provided in this publication, or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information contained in this document has not been submitted to any formal IBM test and is distributed AS IS. The use of this information or the implementation of any recommendations or techniques herein is a customer responsibility and depends on the customers ability to evaluate and integrate them into the customers operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Anyone attempting to adapt these techniques to their own environment do so at their own risk. This document and the information contained herein may be used solely in connection with the IBM products discussed in this document. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurements may have been estimated through extrapolation. Actual

22

IBM InfoSphere MDM Server Appendixes

results may vary. Users of this document should verify the applicable data for their specific environment. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs.

Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at Copyright and trademark information at www.ibm.com/legal/copytrade.shtml Windows is a trademark of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.

23