Yphise portfolio of valuable projects

Independent research since 1985

Software Product Assessment

EXECUTIVE VOLUME

SELECTED BUSINESS CASE

PTF - PERFORMANCE TESTING FACTORY

For guaranteeing the service level of operational services and applications

GM Governance

IS Information Systems

PR Projects

SO Service Management

Abstract
EXECUTIVE OVERVIEW

3 4 9 15 19

Benefits Yphise shortlist Yphise ranking

APPENDIX

Assessment overview

Yphise ISO 9001:2000-certified independent quotations

of software products

Yphise ISO 9001:2000-certified ongoing research program since 1985

Yphise independent research continuously monitors the strategies and solutions of all successful software vendors in order to select valuable and high-priority projects for improving the business performance. Each year, Yphise research draws up the portfolio of valuable projects that the executives should consider when making investment decisions. Yphise transforms high-level strategies or complicated problems in concrete projects with accurate benefits and valuable software solutions. Yphise puts forth investment recommendations on software solutions based on accurate assessment according to valuable business cases. Yphise judgment relies on a detailed assessment of each solution. Yphise has unique experience in identifying the useful functional and technical requirements in order to score solutions according to benefits expected by large companies. The Yphise ongoing research program has helped executives plan, prioritize, conduct and control the useful software projects in order to increase business performance since 1985. Our independence since 1985 proves our unparalleled expertise in helping the executives stand back from operational pressure and arrive at sound decisions focused on the issues of large companies. ISO 9001:2000-certification of our research program is evidence of this expertise. This unique distinction states independence and robust methodology for assessing products according to issues and priorities of large companies.

The portfolio of valuable software projects aimed at IT steering committee

SIPP report Software Investment & Project Planning

Each selected business case The project qualified at executive level: benefits, scope, key requirements, final short-list of valuable solutions, ranking of the solutions

EXECUTIVE vol

The detailed assessment of the project The shortlist of solutions of interest, the detailed assessment of each shortlisted solution, our final ranking of the solutions The results in Excel format The business and IT benefits The useful set of requirements

ASSESSMENT vol and selection ASSISTANT

REQUIREMENTS vol

The business cases that represent valuable investments and lead to pragmatic projects according to issues of interest to executives Continuous monitoring of the marketplace

Selected Business Cases

Ongoing monitoring

yphise@yphise.com PO BOX 142, Southbury, CT 06488 - USA 6 rue Beaubourg - F-75004 PARIS

T 1 203-405-2549 - F 1 203-405-2569 T (33) 1 44 59 93 00 - F (33) 1 44 59 93 09

This report is produced and edited by Yphise. Yphise is an independent analysis company. Yphise cannot be held responsible for the use made of the opinions and results provided. No part of this document may be reproduced or transmitted in any form or by any means, without the express written consent of Yphise. Yphise. November 2008.

ABSTRACT
We are pleased to provide you with our opinion on the Performance Testing Factory (PTF) business case. Our research demonstrates that Performance Testing Factories represent a priority investment. Executives can invest in all confidence because this business case leads to concrete projects with strong benefits, accurate scope and valuable software solutions, backed by the commitment of vendors and by useful strengths. Consequently, we select the Performance Testing Factory business case in the Yphise portfolio of valuable projects that executives should consider when making investment decisions. Performance Testing Factories are a key asset for ensuring successful business activities. Each company needs to keep its customers satisfied. Customer satisfaction is tied to the performance of services delivered by the IT. PTFs check the availability, capacity and continuity of these services before the deployment in operations. They deliver warranties on the performance of business applications and services. They help comply with service level agreements. Our research also shows that Performance Testing Factories mitigate the risks of failed projects. They reduce the complexity of performance testing. The ability to conduct performance tests faster and easier increases business agility. This is key to deliver reliable applications and services while complying with business requirements in terms of budget and project duration. This business case is selected in the Yphise portfolio of project opportunities for the first time. We highlight the value of Performance Testing Factories for implementing continuous and industrialized management of application performance. We put forth this investment recommendation based on our independent ongoing research. We remind you that, since 1985, this independent research has been continuously monitoring the strategies and solutions of all successful software vendors in order to select high-priority projects for improving the business performance. This research is ISO 9001:2000-certified (since 1999). This unique distinction is evidence of independence and robust methodology for assessing software solutions according to issues and priorities of large companies. We were very pleased to closely work with the experts of each shortlisted vendor during this research. We are sure that you will be excited by this investment opportunity and by launching the corresponding project. Yphise research team November 2008

Yphise

BENEFITS
Performance testing factories (PTF) help validate and improve the performance of services and applications during the development process. They facilitate the design and execution of load-testing scenarios. They measure application response time and track performance bottlenecks through load and stress tests. They analyze the root causes of performance problems. They ensure regression testing and continuous improvement of the services technical quality. They help plan capacity. Service Validation and Testing is part of Service Transition in ITIL Version 3 (December 2007). It is aimed at ensuring that new or changed services are aligned on service-level requirements and are ready to run in the operations environment. The ITIL V3 process of Service Validation and Testing validates that a service is fit for use, i.e. able to deliver the required service level, before deployment in operations. PTFs are a key asset in order to implement a successful service validation and testing process. They address both service-level testing and operational testing. They provide confidence in service availability, capacity and continuity. They also help check the operational performance of services. Benefits for the business Increasing the customer satisfaction Successful companies rely on satisfied customers and internal users. Customer satisfaction is tied to the performance of services delivered by the IT. Unavailable applications and long response times lead to noncompliance with service level agreements, degraded productivity, loss of customers and loss of revenue. The trend to open applications and services to partners and customers increases the need for reliable performance in operations. The risk represented by an insufficient service level in operational services must be assessed and managed proactively. Detecting bad performance in operations is too late. Customers need warranties on the service delivered by IT. Performance and load tests help deliver these warranties. PTFs measure and check the service scalability, capacity, availability or the response time according to the load. They highlight performance issues and bottlenecks. They help identify the root cause of such problems. They contribute to quality assurance of IT services. Last but not least, PTFs provide forecasts and trend indicators useful for capacity planning. This helps IT departments anticipate a rise in business activity and allocate sufficient resources to maintain the level of service delivered over time. Increasing the business agility Companies require agility in order to stay competitive in a changing environment. The cost and duration involved in implementing the IT projects and changes have a strong impact on business agility. Information systems in large companies are complex and heterogeneous. This usually makes performance testing long and costly. Such tests 4 Yphise

are often bypassed due to delays or budget. This leads to failed projects due to insufficient performance. Performance testing itself must be agile in order to integrate more easily into the projects. The capability to conduct performance and load testing faster, at lower costs and sooner in projects is a key factor in increasing the business agility. This requires industrializing the performance testing. PTFs address this need to industrialize. They facilitate the creation of test scenarios. They reduce the complexity of test design. They support various sets of technologies and protocols, thus reducing the time needed to build performance tests in heterogeneous environments. This leaves more time to run tests, identify bottlenecks and improve application performance. Performance testing industrialization also means building reproducible test processes. The ability to reuse test assets is a key factor in regression testing. This requires capitalizing on previous test campaigns. This also requires sharing the information on test results and assets. PTFs help capitalize on test assets and reuse them. PTFs help implement a continuous improvement cycle of performance. Benefits for the IT processes

Effectiveness
E s s e n t i a l

Service-level

Industrialization

Testing

Conducting the projects Product & infrastructure Development

. .

Maintenance

Operations

Incident management

U s e f u l

Productivity
Useful Essential

Benefits for the IT processes

Yphise

Testing The test process for each project must guarantee both the functional and technical adequacy of the result when budgets and time frames are tight. Testing the performance of new or changed services is a complex task. Such tests are often neglected due to time constraints, or they fail due to a lack of adequate tools. PTFs ensure a valuable and productive performance testing process. They manage performance testing and analysis during development. They help align tests on business requirements. They industrialize the testing process. PTFs facilitate the design of load tests for effective and efficient testing. They reduce the time required to build test campaigns. This allows more time for validating various performance scenarios. The productivity provided by PTFs also mitigates the risk of performance testing cancellation. Industrialization The industrialization process must ensure that the information system can be operated according to the service level expected by the business. In each application project, the industrialization process develops the operational tools and instructions required for deployment and operations. It designs and manages the technical solutions in order to guarantee effectiveness in operations. Effectiveness in operations includes the capability to deliver the expected performances. Industrialization must demonstrate this capability before deployment in operations. PTFs validate the service level delivered by the new or changed services before the deployment in operations. They ensure that these services are operational. They mitigate the risks associated with deployment of new applications. Conducting the projects The mission of the project manager is to conduct projects according to the target value and the expected cost and duration. Performance testing is a key factor of success in development projects. Implementing an efficient performance testing process is essential to ensuring that this step is not neglected during the projects. PTFs reduce the cost and duration of load and performance testing. They help align the test campaigns on business priorities. They accelerate the design and implementation of performance test scenarios. This allows more time to run detailed performance analysis on new or changed applications. Last but not least, PTFs can run performance analysis at various stages of a project, which helps build efficient schedules. Service level management The service level management process guarantees the service levels of the IS in operations. Service level agreements (SLAs) are particularly aimed at business operations, and there is a need to demonstrate compliance with these agreements. The service level manager must guarantee that the service level delivered to users complies with the negotiated SLAs. 6 Yphise

PTF solutions help estimate and validate the service level that applications can effectively deliver before deployment in operations. They prevent service level managers from defining SLAs impossible to fulfill. Maintenance The maintenance process guarantees the definition of scope-limited changes. These changes must bring added value to the business, quickly and at low cost. A key challenge of maintenance is the guarantee of non-regression of existing services when budgets and time frames are tight. IT managers must ensure that the various maintenance projects have no impact on the service level of existing applications. They need to implement performance regression testing. Such tests must be easy to define and run in order to comply with project time and financial constraints. PTFs ensure an efficient regression testing process during maintenance. They help capitalize on existing test assets in order to reduce the time required for regression testing. They compare successive performance test results in order to track and identify degradation over time. Development A key development challenge is productivity, i.e. releasing the correct applications on time and within budget. The development process must be optimized. Accurate processes improve the quality and quickness of software development. PTFs include tools to analyze code performance. This facilitates early identification of performance issues such as memory leaks. PTFs help development teams pinpoint the code where a performance issue arises. They improve the technical quality of new and changed services. Product and infrastructure management The product and infrastructure management process guarantees the expertise and knowledge required to manage and support the various products used in a company, i.e. hardware or software, middleware, software packages or applications. PTFs test the performance of new or changed services before deployment in operations. They help predict the performance of these services on the operational infrastructure. They help capacity planning identify when additional resources will be required to ensure service level continuity. Incident and problem management The incident management process guarantees the resolution of incidents as soon as possible, in order to ensure activity recovery. The problem management process identifies the root cause of an incident, in order to avoid any reoccurrence. PTFs help identify and correct performance issues in new or changed services before deployment in operations. They improve the technical quality and the robustness of 7 Yphise

applications. They reduce the number of incidents and problems detected in operations. Operations The IT department must guarantee correct business operations according to service levels agreements. PTF solutions help validate that new or changed applications can deliver the performance required by the business before deployment in operations. They perform load and stress testing in order to identify and correct performance issues before deployment.

Yphise

YPHISE SHORTLIST
Positioning of the market segment The evaluated market segment is a specific one, with a precise positioning and list of software solutions. To avoid confusion, we differentiate it from the following list of market segments. A solution belonging to one of these market segments is inappropriate for attaining the expected benefits. Test process optimization (TPO) TPO solutions drive the testing activities on applications. They manage the test definition, planning and execution. They manage both manual and automated tests. TPO solutions help project managers plan test cycles by taking into account the resource availability, the environments, and the risks or the sequence constraints. They provide executives and project managers with accurate monitoring of testing activities. Performance testing is part of the whole service validation and testing process. Performance, load and stress tests must be included and planned in the test cycle. PTFs are integrated into TPO solutions in order to guarantee effective planning of performance tests. E.g. CARS (Compuware), Quality Center (HP) and Silkcentral Quality Optimization (Borland Software). Functional testing automation (FTA) FTA solutions validate that new or changed applications cover all the business requirements. They must test all the use cases defined by the business. FTA solutions rely on automated tools that run the applications step by step in order to check functional coverage. PTFs test the performance of applications, i.e. response time, availability and number of concurrent users supported. Both types of solutions are complementary. They take place in the overall process of application quality assurance. E.g. Functional Testing (HP), Rational Robot (IBM), SilkTest (Borland) and TestPartner (Compuware). Application security assurance (ASA) ASA solutions test applications in development in order to detect and remove vulnerabilities. They rely on a static and dynamic analysis of code. Security testing and PTFs are complementary. Performance issues can lead to vulnerabilities in applications, and security issues can represent a threat to the 9 Yphise

availability of applications. Both test types are required to enforce the security, stability and robustness of the applications. E.g. HP Security Center (HP), Fortify 360 (Fortify) and IBM Watchfire (IBM). End-user experience monitoring (EUEM) EUEM solutions measure the service level delivered to business users by applications in operations. They measure the application availability (i.e. success rates), the response time perceived by end-users and the integrity of data displayed to end-users. EUEM solutions monitor performance and provide an alert in case of incidents. They help identify the business impact of performance degradation. They diagnose performance incidents, i.e. identify the root cause. PTFs detect performance issues during development cycles, while EUEM solutions are dedicated to the monitoring of application performance in operations. Both solutions use the same type of features to capture performance metrics and diagnose performance issues. Tight integration of both types of solutions allows continuous performance improvement. Performance monitoring scripts defined during tests can be reused to check application performance in operations. Real business transactions captured in operations environment can also be used to build and improve performance tests. Finally, integration into EUEM tools extends the monitoring options available in PTFs and helps make an accurate diagnosis. E.g. TM-ART (BMC), Vantage Agentless Monitoring (Compuware), Business Availability Center (HP), ITCAM (IBM) and Foglight (Quest). Transaction performance analysis and diagnostic (TPAD) TPAD solutions help identify the root cause of incidents or performance degradation. They help identify failed components. They make it possible to track transactions throughout the various protocols and environments and to drill down in details. PTFs submit the applications to various load scenarios in order to check their behavior and to highlight performance issues. Identifying the components that lead to performance issues may be difficult in highly distributed applications. TPAD solutions help identify and locate these components. They are complementary to PTFs. Some solutions integrate PTF and TPAD in order to enforce performance analysis of complex transactions. E.g. ApplicationVantage (Compuware), Business Availability Center (HP), Foglight (Quest), ITCAM (IBM), Transaction Management Root Cause Analysis (BMC) and Wily Introscope (CA). Code performance optimization (CPO) CPO solutions are designed for development teams. They rely on dynamic code analysis. They provide performance profiles with metrics on memory use, CPU use and similar indicators. This information helps developers optimize the performance of 10 Yphise

programs. It also helps identify problems such as performance bottlenecks or memory leaks. Many performance issues are caused by bad programming practices. Such performance issues can be identified early in the development cycle, through a detailed code analysis performed by CPO solutions. PTFs interface with CPO tools in order to facilitate root-cause analysis and performance diagnosis. This also helps reduce the number of performance issues detected at the later stage of application development. E.g. Appsight (BMC), ITCAM for J2EE (IBM), Jprobe (Quest), PerformaSure (Quest) and Vantage Analyzer (Compuware). Application development performance measurement (ADPM) ADPM solutions allow assessment and optimization of application portfolios. They analyze the code of components and services in order to measure the quality of the code and the architecture. ADPM solutions assess over time the quality of changes to application portfolios, whether these changes are made by internal teams or by external partners. ADPM solutions help evaluate the cost and duration required for each change request. ADPM solutions improve the quality of application code through static analysis. They can highlight some types of defects in the code that lead to performance issues, but many performance problems appear only when the code is executed. PTFs improve application performance through dynamic analysis. Both solutions can be used in a complementary way to help build robust and easy-to-maintain applications. E. g. Application Intelligence Platform (Cast), Enterprise View (Micro Focus) and System Code (Metrixware). Application change and release management (ACRM) ACRM solutions manage consistent sets of application changes within releases, from development to deployment in operations. They ensure consistency of the IT environments, from development sandboxes to operations environments. They control the various test stages, from unit to user-acceptance tests. They guarantee the consistency of versions, releases and environments. They manage the application change workflow. They control, execute and track the building and deployment of releases in various platforms. Performance tests are a key step in projects. Each change and each release must be tested in order to ensure service level non-regression. Both types of solutions must be interfaced in order to facilitate the scheduling of performance tests and to track performance test advancement. E. g. Change and Release Management (IBM Rational), Dimensions (Serena Software), Integrity (MKS) and Synergy (Telelogic/IBM Rational).

11

Yphise

Change orchestration and management (COM) Change orchestration and management (COM) solutions control the various IT changes. They drive the change process, from requests for change (RFCs) to compliance checking. They schedule the changes and manage the priority of these changes. They manage the change workflow and automate the various technical tasks required to ensure efficient changes. COM solutions contribute to compliance with ITIL requirements on change and release management. Each change must be tested. Performance tests help check that the changes do not degrade the service level of existing applications. They can be integrated into and scheduled in the change process through integration into COM solutions. E.g. Altiris Total Management Suite (Symantec), Operation Manager and Orchestration Manager (Bladelogic/BMC) and Remedy Change Management (BMC). Helpdesk Service support management (SSM) Service support management (SSM) solutions provide an integrated set of helpdesk functions. They help manage and resolve incidents. They manage incident tickets and help their resolution by the IT department. PTFs help identify and localize performance issues in applications before deployment. They reduce the number of problems detected in the operations environment. However, some problems are only detected after deployment. Such issues are managed by SSM solutions. Test teams need to know when performance issues appear in operations, in order to adapt the tests. Integration of PTFs and SSM solutions reduces the time required to solve performance problems detected in operations. E. g. Remedy Service Desk (BMC), Service Desk (HP) and Service Request Manager (IBM). Business service monitoring (BSM) Business service monitoring (BSM) solutions help model the business services and their relationships with all the IS components (e.g. applications, infrastructure). They collect the IT events that impact on business services. They control and monitor business services over time according to service level agreements (SLAs). Business services rely on various applications and technical services. PTFs assess the performance of these applications and services before deployment. They check whether services and applications are able to deliver the service level required by the business. BSM solutions also need performance metrics to monitor business services. Both types of solutions share the same monitoring technology. They can also share test scenarios in order to control the performance of critical business transactions or services. E.g. BEM/SIM (BMC), Business Availability Center (HP), Managed Objects BSM (Managed Objects) and Tivoli BSM (IBM). 12 Yphise

Network and system monitoring (NSM) NSM solutions measure server and network performance. They check whether or not IT elements are running. Network monitoring tools measure the network traffic and bandwidth use. Server monitoring tools measure the performance and availability of servers. They monitor the IT items from a technical point of view. Some types of performance issues can be tied to the network or to the systems on which applications run. PTFs need metrics on how the infrastructure is used, in order to detect such problems and identify the root cause. They can retrieve metrics from NSM solutions in order to build an end-to-end view of application performance. Integration of both types of solutions facilitates performance diagnosis. E.g. ApplicationVantage & NetworkVantage (Compuware), Operations Center & Network Management Center (HP), Patrol (BMC), Tivoli Monitoring (IBM) and Unicenter NSM (Computer Associates). The shortlist for the evaluated business case We selected this business case in the Yphise portfolio of project opportunities for the first time. Testing the performance of applications during development is a key step in order to ensure that these applications are fit for use and fit for purpose, as described in the ITIL V3 Service Validation and Testing process. Performance tests used to be neglected because of the complexity of performance test design and performance analysis. This could lead to various performance issues in operations. Outsourcing the performance testing used to be the most common solution. However, this makes the industrializing of performance improvement over time impossible. Outsourcing is appropriate for point performance testing; it is inappropriate for performance testing factories. Current business challenges listed in the chapter entitled Benefits require frequent performance testing and effective regression testing. Investing in PTFs is now the recommended solution in order to meet the current business challenges. Our final shortlist focuses on solutions that can manage performance from requirements design to deployment in operations, including test scenario design, test result analysis, regression testing and root cause analysis. The ISO 9001:2000 certification of our process is evidence of independence and robust methodology for assessing the solutions according to the issues and priorities of large companies. All shortlisted vendors demonstrate their commitment to responding to the requirements of this process. This process challenges them, as in a customer situation. It is evidence of the vendors' commitment to responding to a demanding independent assessment.

13

Yphise

Our final shortlist is as follows: QACenter Performance Edition (Compuware); Performance Center (HP); and Rational Performance Tester (IBM). QACenter Performance Edition (Compuware), Version 5.7 The Compuware portfolio covers IT Portfolio Management, Application Delivery Management and IT Service Management. Application Performance Assurance is addressed as a specific market segment within the Application Delivery Management offering, distinct from Quality Assurance. The Compuware performance factory includes QALoad, Server Vantage (performance monitoring) and File-AID/CS (management of test data pools). The solution is also interfaced with DevPartner (code analysis), Application Vantage (transaction profiling) and Vantage Analyzer (infrastructure analysis and capacity planning). Performance Center (HP), Version 9.1 HP provides an enterprise platform for Business Technology Optimization (BTO). Tests are divided in three categories: Application Security, Quality Assurance and Performance Validation. Most of this offering comes from the acquisition of Mercury Interactive in 2006. HP Performance Center includes HP Project and Portfolio Management (demand, projects and resources management for load testing), HP LoadRunner (test execution) and HP Diagnostics (analysis of test results). Rational Performance Tester (IBM), Version 7.1 The IBM Rational product line covers the whole application delivery process, including testing. IBM addresses functional testing, performance testing and security testing in order to deliver reliable, enduring and functional applications. Performance testing is managed by Rational Performance Tester (RPT). The solution is interfaced with the other Rational software solutions, e.g. ClearCase (software configuration management) and ClearQuest (change and release management). RPT is also interfaced with IBM Tivoli monitoring solutions (e.g. ITCAM and ITSM). The acquisition of Princeton Softech in 2007 also enriches IBM solution with a data management solution, Optim, which can be used for the management of data pools.

14

Yphise

YPHISE RANKING

HP Performance Center

1. BUILDING A PERFORMANCE TESTING FACTORY ALIGNED WITH THE BUSINESS NEEDS

4. INDUSTRIALIZING THE PERFORMANCE IMPROVEMENT PROCESS

2. GOVERNING THE PERFORMANCE TESTING FACTORY

3. INCREASING THE PRODUCTIVITY OF PERFORMANCE TESTING

HP Performance Center

Worst evaluation

Average evaluation

Best evaluation

Compuware QACenter Performance Edition

1. BUILDING A PERFORMANCE TESTING FACTORY ALIGNED WITH THE BUSINESS NEEDS

4. INDUSTRIALIZING THE PERFORMANCE IMPROVEMENT PROCESS

2. GOVERNING THE PERFORMANCE TESTING FACTORY

3. INCREASING THE PRODUCTIVITY OF PERFORMANCE TESTING

Compuware QACenter Performance Edition

Worst evaluation

Average evaluation

Best evaluation

15

Yphise

IBM Rational Performance Tester

1. BUILDING A PERFORMANCE TESTING FACTORY ALIGNED WITH THE BUSINESS NEEDS

4. INDUSTRIALIZING THE PERFORMANCE IMPROVEMENT PROCESS

2. GOVERNING THE PERFORMANCE TESTING FACTORY

3. INCREASING THE PRODUCTIVITY OF PERFORMANCE TESTING

IBM Rational Performance Tester

Worst evaluation

Average evaluation

Best evaluation

The above charts are presented in order of final top-down ranking of the shortlisted solutions after their in-depth ISO 9001:2000-certified assessment by Yphise. The four assessment dimensions show the expected benefits by large companies. They aggregate more than 180 functional or technical control items. This list of requirements is practical, in order to evaluate the capacity of the solutions to provide an adequate return on investment. It is also proactive in order to appreciate the upcoming perspectives. Assessment dimensions 1. Building a performance testing factory aligned with the business needs A business requires available and usable services. Business performance requirements are expressed through service level agreements (SLAs). The ability to identify and capture these SLAs is essential in order to define and prioritize the performance tests. Business services rely on complex and heterogeneous information systems. The growth of service-oriented applications increases the need to build and run crosstechnologies tests. Test factories must be able to address these complexities in order to build adequate test scenarios. 2. Governing the performance testing factory Performance tests require time, people and computing resources. Accurate governance and planning is a key factor in ensuring that performance tests can be run efficiently and on time. Test campaigns must be planned accurately. Performance tests must also be included in the project planning, especially when the load testing environment is shared between many projects. 16 Yphise

Performance tests are a key step in the application lifecycle. Performance testing factories need to integrate into the other solutions involved in this lifecycle in order to implement industrialized test processes. 3. Increasing the productivity of performance testing The more performance tests are run, the easier the identification of performance issues and their correction on time become. The ability to build performance test scripts and to manage data variation easily is essential for the productivity of performance testing factories. An important part of performance test design also involves load and monitoring customization. Flexible management of load variation facilitates test execution. Exception handling during the test is also important for building robust and reliable performance tests. Last but not least, the reuse of test assets increases the productivity of performance testing. Reusing these assets efficiently requires fine-grained control on the test repository. 4. Industrializing the performance improvement process Building efficient and accurate performance test processes requires supporting various levels of tests during the application lifecycle. Running performance analysis early in development makes it possible to detect and correct performance issues more easily and at a lower cost. Analyzing the component and transaction performance before running traditional load testing helps reduce the number of issues identified at the end of the application development. Supporting various levels of performance analysis also facilitates diagnosis in case of performance problems. Industrializing the performance testing also means sharing easily the test results with all the involved stakeholders. This includes development teams, project managers but also operations teams. The REQUIREMENTS volume provides the detailed requirements. Comments The detailed assessment of the shortlisted solutions demonstrates that the best ones have the strengths that would be useful for large companies and would represent a valuable investment. The assessment results demonstrate that the shortlisted solutions are valuable. However, there are significant differences among the solutions in the coverage of the different assessment dimensions. We remind you that building a Proof of Concept in the targeted technical environment and for the expected value-added benefits remains a requirement.

17

Yphise

HP Performance Center (HP) has the best coverage of our list of requirements. It is designed for large-scale enterprise deployments, e.g. implementing an expertise center dedicated to application performance. We appreciate the focus on performance test industrialization and how the solution accelerates and facilitates the test design. This helps the performance experts focus on actual performance analysis and improvement. A first key strength of the solution is the Click and Script technology, which accelerates the creation of test scripts for rich-client applications. A second key strength is the fine-grained role-based access control and the shared Web portal, which make it a robust platform for enterprise-wide testing projects. Finally, the solution is tightly interfaced with other HP software products, especially with Quality Center, which provides a governance portal for all the testing projects. QACenter Performance Edition (Compuware) is an industrialized platform for application performance assurance. It provides strong coverage of our list of requirements. We appreciate the focus on integrating the performance management into the whole project, from requirement analysis to deployment into operations. The solution benefits from integration into other Compuware solutions, such as the ChangePoint governance portal, the Vantage performance monitoring suite and the OptimalTrace requirements management solution. The solution provides advanced diagnosis functionalities that facilitate the root cause analysis of performance issues in multitiered applications. A key strength of the solution is the strong options for data pool management, available with File-AID. Rational Performance Tester (IBM) is easy to use and implement. This reduces the time required for building performance test scripts. The design and the variabilization of test scripts are entirely done through wizards and require no coding. The IBM Optim solution provides test teams with an advanced solution to manage data pools. The solution also benefits from its interface with the Rational ClearQuest Change and Release Management solution and the Rational ClearCase Software Configuration Management product. Rational Performance Testing includes native performance profiling functionalities for Java EE applications. Integration into IBM Tivoli ITCAM and ITSM solutions extends the performance monitoring capabilities of the solution.

18

Yphise

19

Yphise

1. BUILDING A PERFORMANCE TESTING FACTORY ALIGNED WITH THE BUSINESS NEEDS

HP Performance Center

Compuware QACenter Performance Edition

IBM Rational Performance Tester

Im p het leme n er o gen ting eou perf o se Ma nv rman n ag iro nm ce tes ing ent ts i the s nc p er om for ple ma xa nce nd req uir em ent so ver tim e

Sorted Aggregated Table

Yphise

1. BUILDING A PERFORMANCE TESTING

FACTORY ALIGNED WITH THE BUSINESS NEEDS
All the solutions can manage the performance requirements over time. However, advanced requirement management functionalities may rely on integration into dedicated solutions for some of the shortlisted solutions. The solutions also differ on ability to manage various types of test cycles, e.g. capacity planning. The most advanced solutions include out-of-the-box capacity planning wizards. The ability to simplify test design is a key differentiating factor in Implementing performance tests in complex and heterogeneous environments. The best solutions include tools to facilitate the design of test scripts for rich-client Web applications and Web services. The solutions also differ in the variety of supported protocols and platforms. Performance Center (HP) ranks high in this dimension. Service level agreements can be defined on key business transactions and tracked directly in the solution. Performance Center provides advanced tools to build test scripts for rich-client Web technologies, e.g. Ajax or Adobe AMF. It includes a module dedicated to Web services testing, which helps define tests on Web services not associated with user interfaces. HP solution also offers the widest coverage in terms of supported protocols and technical environments. QACenter Performance Edition (Compuware) has the best coverage of Managing the performance requirements over time. The solution is tightly interfaced with OptimalTrace, i.e. the Compuware requirements management solution. It can easily manage various types of test cycles. The solution also includes out-of-the box predictive tools for capacity planning. QACenter also has good coverage of Implementing the performance tests in complex and heterogeneous environments. Batch processes can be included in the tests through integration into the ServerVantage and Strobe (MVS platforms) monitoring solutions. Rational Performance Tester (IBM) covers the standard requirements in this dimension. The solution is integrated into the other Rational solutions. Integration into Rational ClearQuest facilitates the traceability of performance requirements during the entire test cycle. The solution provides good support for Web-based applications, especially on the Java EE platform.

21

Yphise

2. GOVERNING THE PERFORMANCE TESTING FACTORY

atin gt

egr

HP Performance Center

Compuware QACenter Performance Edition

IBM Rational Performance Tester

Pla

Int

nni ng a

e te nd stin gov gi er n nto ing the the app tes lica t ca tio mp nl aig ife ns cyc

he per f

orm anc

le

Sorted Aggregated Table

Yphise

2. GOVERNING THE PERFORMANCE TESTING

FACTORY
The best solutions in Planning and governing the test campaigns integrate into mature project governance solutions. This is a key criterion for ensuring that performance tests are not neglected in the overall planning of development projects. All the shortlisted solutions provide good coverage of Integrating the performance testing into the application lifecycle. They offer tight integration into other solutions involved in development management. Differences appear in integration into change orchestration management solutions, which focus on operations environments. Performance Center (HP) has full coverage of Planning and governing the test campaigns. The solution is tightly integrated into HP Quality Center, an overall governance platform for test process optimization. This facilitates the planning of performance tests and the allocation of testing resources and test teams. In Integrating the performance testing into the application lifecycle, Performance Center benefits from integration into various HP solutions, such as Service Manager for change management, Operations Orchestration for change automation, Quality center for the governance of all quality assurance processes and Business Availability Center for operational services monitoring. QACenter Performance Edition (Compuware) has full coverage of Planning and governing the test campaigns. QACenter is tightly integrated into the Compuware ChangePoint portal. Performance tests can be easily integrated into cross-project planning and scheduled accurately. In Integrating the performance testing into the application lifecycle, integration into ChangePoint and Vantage solutions helps deliver full traceability of performance issues during the entire application lifecycle. An open API is included in order to integrate into other solutions. Rational Performance Tester (IBM) currently provides basic coverage of Planning and governing the test campaigns. The solution ranks well in Integrating the performance testing into the application lifecycle, thanks to integration into other IBM solutions, especially the Rational ClearQuest and Tivoli monitoring solutions.

23

Yphise

3. INCREASING THE PRODUCTIVITY OF PERFORMANCE TESTING

HP Performance Center

Compuware QACenter Performance Edition

IBM Rational Performance Tester

Cu st o mi zi n gt he tes Fa t ex cili ecu tati tio ng n the reu Bu s e ild of ing t es t as eas s et ily s the per for ma nce te

st s

cen ar i os

Sorted Aggregated Table

Yphise

3. INCREASING THE PRODUCTIVITY OF

PERFORMANCE TESTING
Differences appear in Building easily the performance test scenarios. A key issue for the design of test scripts is productivity. The selected solutions provide tools and wizards to accelerate the design of performance test scripts. However, options for defining the data pools vary considerably from one solution to another. All the shortlisted solutions have good coverage of Customizing the test execution. Some differences do exist in how exceptions are handled during test execution. Solutions differ in Facilitating the reuse of test assets. The key difference is access management to test assets. The best solution includes fine-grained access control in order to manage user rights. Other solutions manage user rights only at the overall project level, and implementing access control requires integration into governance platforms. Performance Center (HP) has the best coverage of this dimension. It ranks high in Building easily the performance test scenarios. The solution provides various tools in order to customize test script parameters. It includes a tool to check whether the data pools provide enough data before running the tests. It also implements the Click and script technology to simplify data variabilization on rich-client applications. The solution also ranks high in Customizing the test execution. How exceptions are to be handled during tests can be customized accurately through SLAs. Performance Center has the best coverage in Facilitating the reuse of test assets. It includes fine-grained access control to the assets repository. Test assets are documented, and their use is tracked directly in the solution. QACenter Performance Edition (Compuware) has strong coverage of this dimension. It has the best coverage in Building easily the performance test scenarios. The solution benefits from integration into File-AID, i.e. the Compuware data management solution. This helps generate data pools based on real data, manage data anonymization and compare data sets after various tests. The solution also ranks high in Customizing the test execution. It provides various options in order to customize load simulation and performance monitoring during the test. In Facilitating the reuse of test assets, the solution relies on the ChangePoint access control system to define user rights on the test assets. Assets are also documented in ChangePoint. Rational Performance Tester (IBM) offers good coverage of this dimension. The solution uses a fully graphic mode to capture and customize test scripts. No coding is required to manage data variabilization, but custom Java code may be integrated for 25 Yphise

advanced scripting. IBM RPT can use the advanced data management capabilities of the IBM Optim solution in order to manage data pool generation, data anonymization and data set comparisons. The solution also provides various options aimed at customizing the execution of test scenarios. In Facilitating the reuse of test assets, the solution benefits from integration into other IBM Rational solutions. Access to the testing repository is managed through Rational ClearQuest, while versioning is managed by Rational ClearCase.

26

Yphise

4. INDUSTRIALIZING THE PERFORMANCE IMPROVEMENT PROCESS

Compuware QACenter Performance Edition

HP Performance Center

IBM Rational Performance Tester

agn osi ng qui ckl Im yt he ple per me nti for ng ma ac nce oll iss abo ues rat ive im pro vem ent pro c

Di

es s

Sorted Aggregated Table

Yphise

4. INDUSTRIALIZING THE PERFORMANCE

IMPROVEMENT PROCESS
All the shortlisted solutions can diagnose quickly the performance issues. Minor differences appear in code performance analysis. Other differences exist in the tools included in the solutions to facilitate diagnosis, such as errors tracking. Solutions differs in Implementing a collaborative improvement process, due to the options available to exchange performance information between the development and the operations teams. Performance Center (HP) has strong coverage of this dimension. The Diagnostics module includes advanced diagnosis functionalities that allow a deep understanding of both code performance issues and transactions performance issues. The solution can incorporate metrics from the HP SiteScope monitoring platform in order to extend its performance monitoring capabilities. Performance Center has the best coverage in Implementing a collaborative improvement process. The solution includes a portal for sharing information on performance tests. This facilitates the exchange of test reports and recommendations with operations. Performance Center also has bidirectional integration into HP Business Availability Center, which monitors business services in operations. This helps align performance tests on operational needs. QACenter Performance Edition (Compuware) has the best coverage in this dimension. It provides powerful diagnosis functionalities at the component level, the transaction level and the application level. DevPartner development studio can locate the lines of code involved in performances issues. ApplicationVantage includes a wizard to analyze the performance metrics collected and to understand complex performance issues. DevPartner can simulate various types of system errors in order to run stress tests. Bidirectional integration into Compuware Vantage monitoring solutions facilitates exchanges between test teams and operations. Rational Performance Tester (IBM) meets the requirements of this dimension quite adequately. The solution provides native performance profiling for Java EE components. The performance of components based on other languages and platforms can be analyzed by means of the Rational PurifyPlus solution. The solution can share information on performance issues with the IBM Tivoli ITCAM and ITSM monitoring solutions. This integration also extends the monitoring metrics that can be collected during the tests. The ASSESSMENT volume provides the detailed assessment of each shortlisted product.

28

Yphise