Application Performance Management Survey

Application Performance Management (APM) is a hot topic for the IT industry and media, but to what extent are real organisations getting to grips with the issues? Are IT managers taking steps to deal with performance problems before they happen, or are the old reactive habits still with us?

Introduction
Application Performance Management (APM) is a hot topic for the IT industry and media, but to what extent are real organisations getting to grips with the issues? Are IT managers taking steps to deal with performance problems before they happen, or are the old reactive habits still with us? Compuware decided to find out. In July 2006 we conducted a survey of APM practice, inviting European IT executives to complete an online questionnaire. We were pleased to receive 150 responses from senior IT managers responsible for testing and performance working at large organisations across Europe. Successful APM depends on tackling performance issues right from the start of the development cycle, when they are still easy and cheap to correct. However, 39% of our respondents organisations had yet to give developers significant responsibility for performance issues. Its important to provide the right tools to enable developers to tackle performance early on, such as profiling solutions, which can predict how an application under development will perform within the production environment. Only 56% of organisations saw profiling as critical, but of these, the vast majority reported performance improvements from profiling. Once an application is live, organisations should proactively tackle performance problems before users notice them. Yet in our survey 71% of respondents said problems were typically identified by endusers via the help-desk rather than through monitoring. While all this might suggest some nonchalance about performance issues, respondents were well aware of their impact on the business. 80% said that customer satisfaction could be affected by performance issues, and other damaging effects, including additional costs from unplanned hardware upgrades, were also frequently cited. It appears therefore that while some organisations are making good progress on the APM front, many of our respondents are in the uncomfortable position of knowing that they have a problem, but have been unsuccessful in improving their application performance. For them, Compuwares Application Performance Assurance approach offers a lifeline.

Executive Summary
Historically, many problems with new applications have arisen from failure to address performance issues until the later stages of development, or worse, until after the application is live. Unfortunately our survey suggests that this failure is still occurring in many organisations, and with the same consequences. On the whole, IT managers are far from confident about the performance of their new applications: only a minority of our respondents said they would give a new application better than 50:50 odds of going live without any performance problems. While there may be an element of not wanting to tempt fate, their pessimism suggests that true APM benefits have not yet been realised. Indeed, over half of organisations said they experience unexpected performance issues in 20% or more of their newlydeployed applications. As we would have expected, formal APM methodologies, consistently applied, were associated with the lowest levels of performance problems in production systems. 44% of our respondents did have some kind of formal methodology in place, though in only 30% of cases was it consistently applied to all projects.

Compuware

Survey Results
Question 1: Which statement best describes your companys typical approach to performance management? Question 3: On average, what percentage of application deployments have unexpected performance issues?
100%

11% 45% 14%

We consistently apply a formal performance management methodology on all projects We have a formal methodology, but dont use it for every project We have an informal methodology

90% 80% 70% 60% 50% 40%

30%

We dont have a methodology

30% 20% 10%

Key findings: Only 14% of organisations consistently use a formal Application Performance Management (APM) methodology, while a further 30% do have a formal methodology but apply it only selectively. Surprisingly perhaps, over 10% of organisations lack any sort of methodology.

0%
0 - 20% 20 - 40% 40 - 60% 60 - 80% 80 - 100%

Key findings: Over half of all organisations experience performance issues in 20% or more of their applications surely an unacceptable level. Once again, the APM methodology emerges as the key to preventing unpleasant surprises when applications go live. Question 4: Have any of the following issues occurred when you've deployed an application?
80%

Question 2: In general, how effective has your organisation been in minimising performance defects before deployment?
4%

8%

Very effective

70% 60%

29% 59%

Somewhat effective

50% 40%

Somewhat ineffective

30%
Very ineffective

20% 10% 0%

Key findings: Just 8% of respondents rate their organisation very effective in eliminating performance problems. This finding is likely to reflect the fact that few organisations consistently apply a formal APM methodology (question 1). Our experience shows that adherence to a systematic approach is vital when it comes to ensuring that applications perform satisfactorily by the time they go live.

Services have not met users expectations

Existing Performance systems has been and services below what have been was originally adversely planned eg. affected agreed Service Level Agreements

The application was delivered late

The project budget was exceeded

Other

None of the above

Key findings: The fact that organisations have experienced these issues is no real surprise, but the scale on which they do so might be: as many as two-thirds of respondents have delivered applications late. These are not isolated incidents: each organisation has typically suffered from two or three of the five main issues. When an application is subject to performance problems, there are inevitably knock-on consequences.

Compuware APM

Question 5: On the day that an application is deployed how confident are you that there will be no performance problems?
100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%
Very confident Confident 50 : 50 Not very confident Not at all confident

Question 7a: Is profiling the application with respect to the production environment considered a critical stage in the application development cycle for your organisation?

Yes

44% 56%

No

Key findings: On the day that an application is deployed, a startling 52% of managers rate the chances of the application experiencing performance problems at 50% or higher. If internal customers were offered these odds at the start of a project, they would almost certainly not accept it. Only 5% of managers pronounced themselves very confident. Evidently APM measures organisations have in place are not sufficient to give managers confidence when an application is about to go live.

Question 7b: If Yes (to question 7a), How much does this profiling improve the performance of your application once in production?
100% 90% 80% 70% 60% 50% 40% 30%

Question 6: Which statement best describes the team that is responsible for performance of applications?

20% 10% 0%
Significantly improves Moderately improves Does not improve Dont know

Operations has primary responsibility for performance

32%

29%

A testing team is primarily responsible for performance Development has primarily responsible for performance

14%

25%

Our entire IT organisation is engaged and involved in performance management

Key findings: Of the organisations that do profile new applications with respect to the production environment, over 80% benefit from the exercise. It could be time for a rethink on the part of the 44% of organisations who do not yet see profiling as critical to application development.

Key findings: Successful APM depends on making performance assurance a theme of the entire development cycle. Our survey shows that 39% of organisations assign responsibility for performance to testing or operations teams, suggesting that they have not yet embraced the concept of engineering good performance into the software from day one. Despite the potentially catastrophic impact of performance problems on the business (see question 9), many organisations are still fixing problems after they occur rather than preventing them from happening in the first place.

Compuware APM

Question 8: When during the application life-cycle are metrics captured to track application performance?

Question 10: The majority of performance issues that occur in production are typically found by:
100% 90%

18%

In the requirements phase

80%
During development

30%
8%

70% 60%

10%

Once errors are revealed in testing Once errors occur in production Metrics around performance are not captured

50% 40% 30% 20% 10%

34%

Key findings: Around half of organisations wait until performance problems occur in testing or production before collecting the data necessary to prevent recurrences; 8% do not collect any performance metrics at all. These omissions carry high costs and risks, particularly if performance problems are allowed to occur in production systems, as question 9 shows.

0%

End-users reporting issues via the helpdesk

IT operations teams using infrastructure monitoring tools

Question 9: When a performance issue occurs in production, what is the impact on the business?
90% 80% 70% 60% 50% 40% 30% 20% 10% 0%
Service levels of existing applications are affected Additional costs are incurred (unplanned hardware upgrades) Application Business revenue downtime is reduced (resulting in a loss of productivity) Customer satisfaction is affected Other

Key findings: Yet again this finding highlights the key issue surrounding APM that most organisations are extremely reactive, with the vast majority of performance issues typically only being identified after they impact the end-user, when the damage is already done. A good APM strategy involves not only preventing problems from getting into new applications, but also monitoring live applications continuously to detect the early signs of performance degradation before it impacts users.

Question 11: Which statement best describes your use of historical metrics and trends within your organisation?

13%

22%

They are reviewed on a fixed schedule and regular action taken They are reviewed on an ad hoc basis and actions are taken as identified They are captured but not reviewed They are not captured

16%

49%

Key findings: : If evidence were needed that it is vital to address performance problems before going into production, then this is it. The consequences of performance problems in a live application include almost inevitable customer dissatisfaction, a high probability of lost productivity and, in around half of organisations, extra costs or a deterioration of other applications performance. Worse, you may lose business. On average, each organisation should expect to experience two or three of these effects whenever performance issues arise in production.

Key findings: : While it is reassuring that 65% of organisations are taking some steps to learn the lessons of history by reviewing historical metrics, the remaining 35% are wasting an opportunity to improve overall software quality by avoiding past mistakes. A mere 16% of organisations make systematic use of their historical performance metrics.

Compuware APM

Conclusion
Some organisations are evidently making headway with APM, but others have cause for concern. The fact that 39% of organisations leave responsibility for performance with testing or operations teams suggests that messages about embedding quality into the development cycle have so far fallen on deaf ears in some quarters: no wonder many IT managers find it so hard to be confident about the performance of their soon-to-be-live applications. Given that 71% of organisations rely on users to report performance problems in production systems, it is clear that many organisations including some of those with an APM methodology in place are not yet proactively managing their live applications from a performance perspective. The solution to both sets of problems pre-production and in production - is Compuwares Application Performance Assurance, a best practices approach that makes performance a priority throughout the application life-cycle and that gets development and operations teams working together towards the same performance goals, and with the same tools. Because performance assurance starts from the requirements phase, there are no unforeseen performance problems on go-live day. And because the same proactive approach carries through to production, any issues that do arise in live applications are spotted by the operations team before they inconvenience the end-user.

To find out more, read our Best Practices for Building Performance in From the Start White Paper and a report written by Forrester analyst

Carey Schwaber entitled PerformanceDriven Software Development. To obtain FREE copies of the White Paper or Analyst Report, or for more

information about Compuwares Application Performance Assurance solution please visit www.compuware-apa.com.

Austria Compuware Austria GmbH Mariahilfer Strasse 37-39 1060 Vienna Tel: +43 1 585 40 85 0

France Compuware Sarl 1 Avenue de la Cristallerie 92316 Svres Cedex

Norway Compuware Nordic AS Lysaker Torg 25 P.O. Box 198 N-1325 Lysaker Tel: +47 67 12 8800

Tel: +33 1 4114 2000

Sweden Compuware AB Box 1117 164 22 Kista Kista Entr Knarrarnsgatan 7 Tel: +46 8 522 912 00

Belgium Compuware NV/SA Ikaros Business Park Ikaroslaan 21 B-1930 Zaventem Tel: +32 2 7159900

Germany Compuware GmbH An der Trift 65 63303 Dreieich

Tel: +49 6103 9488 0

Portugal Compuware S.A. Sucursal em Portugal Campo Pequeno, 48-7 Edificio Taurus 1000-081 Lisboa Tel: +351 21 792 81 60

Switzerland Compuware AG Tfernstrasse 3 5405 Baden-Dttwil

Tel: +41 56 483 26 26

Denmark Compuware AS Telegrafvej 5 2750 Ballerup Denmark

Italy Compuware SPA Via Gozzano nr. 14 20092 Cinisello Balsamo MI

Tel: +45 4468 0122 Finland Compuware Finland Oy PL 38 (Lintulahdenkatu 10) 00501 HELSINKI

Tel: +39 02 66123 1 Netherlands Compuware BV Hoogoorddreef 5 P.O. Box 23199 1101 DR Amsterdam Tel: +31 20 311 8800

South Africa Compuware Southern Africa Northdowns Office Park 17 Georgian Crescent Bryanston East Gauteng 2152 Tel: +27 11 516 2900 Spain Compuware S.A. PO de la Castellana, 77 3a Plta. 28046 Madrid Tel: +34 91 418 4750

United Kingdom Compuware Ltd. 163 Bath Road Slough SL1 4AA

Tel: +44 1753 444 000

Tel: +358 9 7746 810

All Compuware products and services listed within are trademarks or registered trademarks of Compuware Corporation. Java and all Java-based marks are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. All other company or product names are trademarks of their respective owners. 2006 Compuware Corporation