EuroSTAR

Software Testing

Community

The Benefits and Practice of Early Lifecycle Performance Testing

An Opportunity for Optimizing Software Delivery Processes

EuroSTAR
Paul Herzlich Research Analyst, Creative Intellect Consulting

A Creative Intellect Consulting Thought Leadership Paper

Creative Intellect Consulting UK is constantly monitoring adoption of software testing tools and best practices. In this paper we examine the benefits and challenges of early lifecycle testing for a cluster of technical test types, commonly known as performance testing. The aim is to bring managers up to date on the latest thinking on the subject and practice in the field, with a view to cutting away the myths and removing obstacles blocking adoption.

The Benefits and Practice of Early Lifecycle Performance Testing

Summary Messages
Early Testing Reserved for Functional Testing
Early testing delivers business benefits by enabling IT organizations to deliver more at lower cost. However, organizations have concentrated on introducing early functional testing and have largely ignored the potential of early performance testing.

However, to most testers and QA managers, there are two aspects of performance testing that render it unsuitable for early testing. The beliefs are: You can only test performance on a fully built system which doesnt exist early on. You dont have suitable component level performance requirements. The solutions to these objections require a finer look at what performance testing consists of and a pragmatic approach to achieving your goals without necessarily achieving textbook perfection.

The dynamics of early testing apply equally or more to performance testing

Early testing delivers benefits in broadly two ways: It saves money on rework It mitigates the risks of surprises late in a project. These dynamics are extremely relevant to performance issues, which perhaps even more than functional errors hold the potential of sending a project back to the drawing board if components or infrastructure dont perform as required.

Introducing early performance testing is a change management project

Starting down the road towards early performance testing will involve developing motivation among those who will be responsible for doing it. You will need to make controlled changes to all your existing development processes, enlist the help of specialists including your existing late cycle performance testers and adopt tools that give you the kind of automated support that works best alongside existing development tools. You should only undertake this within the context of a properly conceived change management project.

1 PAGE

Practitioners believe there are two unsolvable challenges to early performance testing
There are a whole host of potential difficulties to introducing early performance testing. Most of them are the same as those that impede the adoption of early functional testing. They are largely soft, people issues, centered around the roles and psychology of software coders. There are fairly well understood change management techniques for dealing with them.

The Business Imperative

Most software delivery organisations accept the need for performance testing, but it is still commonly done very late in the application lifecycle. Some even wait until after software has been released in production.

The Benefits and Practice of Early Lifecycle Performance Testing

We know from studies over the last 30 years that testing early in the lifecycle yields big benefits. Although some resistance in the field still remains, the argument has been largely won for early functional testing. However, despite this general acceptance, the practice lags far behind for performance testing. Although QA managers at the leading edge of process maturity are beginning to adopt it, there is a widespread perception that early performance testing, even if desirable, is impractical or unfeasible. The challenges to adopting early functional testing were soft issues mainly people and process. Once the benefits of early functional testing were understood, it became a matter of building the practice into development processes and winning the hearts and minds of developers to adopt it. Developers traditionally resisted the notion that formalised testing was part of their job. Textbook unit testing was largely a myth. However, Agile methods, powerful IDEs and unit test frameworks have been significant in bridging the gap between testing and development activities. Early functional testing increasingly has traction throughout the industry. Early performance testing is a practice that presents not only the same set of soft behavioural challenges as early functional testing, but also a set of hard, technical constraints. We know how to solve the soft challenges, but the technical constraints seem intractable to all but a few very experienced specialists in test processes. An example of a widely held view, stated in the extreme, is that you can only test performance meaningfully when you have a fully assembled system in a nearly production-like environment, things you dont have early in the lifecycle. If that truly were the case, early performance testing would indeed be impossible. In fact, these are not technical constraints at all but simply intellectual obstacles that can be addressed. Creative, innovative and forward-thinking test practitioners have found solutions. If early performance testing is so unintuitive and counter to the received wisdom, then why is it worth pursuing? The answer is that we have an obligation to make software delivery more responsive to the perennial business needs to deliver more and cost less. Test, QA and development managers aligned with these business goals understand that early performance testing is yet one more tool to achieving this.

How Early Testing Delivers Benefits

There are two ways in which early testing delivers benefits: cost and timescale reduction and increased project predictability.

2 PAGE

Both of these are extremely important in a commercial environment where business success depends on IT to minimize costs and meet changing market demands quickly. Businesses cannot afford cost overruns and delayed product releases. The days when an estimated 60% of all projects get canned usually for not delivering the right systems at the right time are over. Project budgets must be controlled, timescales managed and uncertainty kept to a minimum.

Rework reducing the costs of going backwards

Many studies have been performed over the years that conclude that testing early has financial benefits. The National Institute

The Benefits and Practice of Early Lifecycle Performance Testing

of Standards & Technology (NIST) study from 2002 on The Economic Impacts of Inadequate Infrastructure for Software Testing is comprehensive and accessible for understanding the whole picture. It demonstrates that bugs are significantly more expensive to correct after code leaves the developer for system and performance testing. It states that, it is costlier to repair a bug that is created in the unit stage in the component or system development stage than cost of fixing a coding bug Relative it is to remedy the same bug in the unitat stage when it was introduced. The report estimates that a bug introduced in the coding stage is 10 times more costly to locate and correct in system testing phase, 20 times more costly in pre-production or beta stages and 30 times more costly post-release than it would have been at the unit stage when the bug was created. Programmers create the code and their code contains bugs. Getting them to find and fix them is not an unreasonable demand.

art 1:

later lifecycle stages

30 20

3 PAGE

10 1 Coding System Test Pre-producon Post-live

Chart 1: Relative cost of fixing a coding bug at later lifecycle stages For bugs in the initial requirements the picture is equally as dramatic. This is worthy of attention since most bugs emanate from badly captured or ill-defined requirements. If the cost of finding a requirements bug during the requirements capture stage is x, it costs 5x in coding and unit test stages, 10x in later test stages, 15x in pre-production and 30x in the released system. If youre skeptical, think about the processes required to locate and fix a bug. When a bug is found in a stage later than coding and unit test, it has to be documented and returned to a developer. A developer has to establish an environment to reproduce the bug. This may entail re-creating test data and restoring an earlier version of source code. Locating the bug the error that is the source of a fault is complicated by the fact youre now dealing with a complex assembly of components rather than simple units. Attending to a bug report comes with an immediate disruption to the developers other work, which in turn has a cost to activities wholly unrelated to this project. When the bug is fixed, code must be promoted and retested at later stages, with all the headaches that entails for source code and build management.

The Benefits and Practice of Early Lifecycle Performance Testing

It is clear that if there is any possibility of producing and running a test while a unit is in the original coding environment, any bug that is found is significantly cheaper to locate and correct. Another way of understanding the nonfinancial benefit of early testing is to consider it as a risk mitigation technique. In fact, all testing is about risk mitigation. The earlier you can close down a risk determine that the risk is mitigated the more confidently you can proceed to later stages and the better you can focus your late cycle activities.

Predictability mitigating risks and avoiding surprises

Although no one asserts that early lifecycle testing would ever find all bugs, when the practice is deployed effectively, companies benefit from shorter project timescales and lower project costs. Early lifecycle testing has another beneficial effect. It enhances the predictability of software delivery, a benefit that managers cant quantify easily, but miss sorely when they get unwelcome surprises in the late stages of a project.

Early testing principles applied to performance testing

The experience of those doing early functional testing is that it brings clear business benefits and better project management. But are there potential advantages from detecting and correcting performance problems at an early stage? You bet, and in spades. Performance problems need to be treated as bugs. Whether performance requirements are explicit or implicit, if someone in a project is capable of declaring that performance is inadequate, the discrepancy is a potential bug just like any other gap between requirement and implementation. It needs the same scrutiny that a functional failure undergoes. Ultimately there will be a determination whether the failure is indeed a bug. When you find a performance failure at a late stage in the lifecycle, all the costs of identifying the cause and reworking the solution are magnified. Functional code bugs (when they are not caused by requirements errors) usually relate back to individual, small sequences of code. However performance bugs found late in the lifecycle can call into question a whole architecture or implementation. The source of the bug could be a code sequence, or it could be the configuration or environment. It is hugely helpful if coding bugs that hit performance have been eliminated by the time you do a full-scale performance test on

4 PAGE

Early testing contributes to greater predictability by providing better information about the true state of a project in the early stages. As work is passed from coding to integration or system testing (or iteration to iteration), there is a great deal more certainty about the codes readiness when it has been unit tested by developers (real unit testing, not the mythical or ad hoc testing developers have historically performed.) A manager has a great deal more confidence that coding is truly complete for 10 programs handed over with a full suite of passed unit tests than 10 programs without documented tests. This confidence works not only at handovers within a project, but also outwards from the project. Without completion evidenced by passed tests, all but the most experienced managers fall into the 95% complete syndrome, and pass that misinformation to their managers, the sponsor and end users. Such self-delusion means that everyone is in for an unwelcome surprise not just a surprise about the software quality but also about the project progress.

The Benefits and Practice of Early Lifecycle Performance Testing

a complete application. By doing this, you enter your late cycle performance testing with more certainty about code, giving you a chance to concentrate on infrastructure and configuration. measure meaningfully requires tests under conditions that represent, approximate or can be mapped onto the production environment. So software developers and testers most commonly test for response time under production-like loading conditions with production-size data volumes load testing and volume testing, respectively. Typical performance, load and volume tests are measuring against targets based on normal and peak workloads executed over timescales of less than one working day. When a test is run to ramp a system to breaking point, it becomes a stress test and when the test execution spans days, weeks or even months, it becomes a soak test. Its important to note that most performance testing is impossible without tools. Tools are needed for driving execution, measuring activity and reporting results. Even crowdsource approaches for driving tests require tooling. Load and stress testing are particularly tool intensive. But there are two subtly different approaches to using a load test tool. Most commonly in load testing, testers attempt to create realistic transaction scenarios through a load test tool and drive them through large numbers of virtual users operating against production-like volumes of data. During the course of the test, measurements are gathered across the software components and infrastructure under test. This is a major logistical exercise that characterizes what most software engineers understand as performance testing. However, load can be used in a much simpler way, which well call background load testing. Without attempting to mimic realistic user activity, virtual users are used

What We Mean By Performance Testing

Up to now, we havent been precise about what we mean when we talk about performance testing. However in order to uncover the possible approaches and opportunities for early testing of code that can impact on performance we need to be more specific.

5 PAGE

Performance testing is commonly used as a term for several related test types whose goals are to establish that an end user will get a functionally correct response from an application and that the timing of the response will be acceptable. Some of the constituent test types are load, volume, stress, concurrency and soak. We need to look more closely at the individual test types to understand which are most suitable for early testing. Straddling the functional and performance domains is concurrency testing. Running more than one instance of a process concurrently can raise functional problems like inconsistencies from simultaneous database updates and performance issues like badly formed locks on resources, which slow things down. In the narrow sense, performance testing is a measurement of response time or latency and the resources consumed to achieve it. To

The Benefits and Practice of Early Lifecycle Performance Testing

6 PAGE

Chart 2: Relationships among various performance-family test types to create a background of infrastructure activity disk access, CPU, network traffic that represents a particular normal loading of the production environment. Then user transactions are driven manually and performance is measured of the manually entered workload. When we look at how to do early performance testing, we need to keep in mind all the various test types available and not get locked into the image of the set-piece, largescale performance test. test and QA managers plus specialist test service vendors? History and tradition in software testing practice, and more widely in development, have a lot to answer for. The Waterfall model for development and, to a lesser extent, the V model widely known among testers, both convey a picture in which technical tests such as performance are relegated to late in the lifecycle. That is, at least, the most common interpretation, even if it was not strictly the intention. Testing tool vendors have also contributed. They have bought into the late cycle testing vision for commercial reasons and, in turn, the available tools have shaped (or misshaped) current practice. We alluded earlier to the soft issues surrounding people and process that are used to undermine early functional testing. Those issues are still present for performance testing: developers dont like testing and are generally not accountable for it. Testing is

The Case Against Early Performance Testing

With so much to be gained, why is the practice of early performance testing still relatively rare the domain of a few visionary

The Benefits and Practice of Early Lifecycle Performance Testing

considered a lesser technical skill (although this is patently untrue for performance or security testing), confers lower status, is inherently less interesting, less sexy and probably a bad career move. No one has made a film called The Social Testing Network. Agile processes have fewer such biases. Cross-disciplinary teams are the norm, and it is common for developers to perform testing. With Agile processes and good understanding of the benefits of early testing for functional testing, and with better IDEs and unit test frameworks, it has been possible to win the hearts and minds of experienced developers (although newbies may still need to fail once or twice before they get the idea) to take some responsibility for functional testing.

Pragmatic Approach To Early Performance Testing

You need to have a pragmatic mindset to make beneficial use of early performance testing. If you demand complete knowledge for example of requirements or total realism for example, of testing environment you will never get started and never get the benefit of all the positive things you can do early in the lifecycle to verify that your software will perform as required. Your objective is to close down as many doubts about performance as you can with the resources you have. Were going to take a closer look at what you can do to make early performance testing a reality. There is no one-sizefits-all prescription. Our aim is to open up the possibilities and move away from the intimidating complexities of the big event late cycle load test.

7 PAGE

However, adoption of Agile does not totally solve the problem. Early performance testing poses two key intellectual challenges that experienced developers on both Agile and Waterfall projects have trouble seeing their way through. Challenge 1: You can only test performance on a fully built system in a production-like environment. Challenge 2: It is impossible to set and you almost never have meaningful performance requirements at the level of individual components. It takes a highly motivated and creative test or QA practitioner to see how these challenges can be met.

Choosing what to test tackling the risks

You could attempt to do performance testing on every unit within the development environment, but when that is not practical (forget about possible) concentrate on the riskiest components. Risky components would include elements that are: Resource intensive in bandwidth, CPU, disk access, etc. Time-critical latency within the component is critical. Database routines a perennial source of performance bottlenecks.

The Benefits and Practice of Early Lifecycle Performance Testing

The developer is the person most ideally placed to evaluate the technical risks of the components they are building. depends on the architecture of the system under test and where the risks lie. However, in developing a general early performance testing process for a typical commercial software delivery organisation, the following types are arranged from the most to least well suited.

Selecting test types

No one type of performance-family testing is totally unsuitable for early testing. It

8 PAGE

Chart 3: Test types by suitability for early performance testing Stress tests are unlikely candidates for early performance testing. Load (as opposed to background load) and soak are possible primarily as part of continuous integration. The others are good candidates for developers alongside functional unit testing. components is impossible when they dont have performance requirements at the individual component, method or unit level. The obvious solution is to analyse the highlevel performance requirements and create requirements for components. If no one in a team not even an architect or designer is capable of doing that, you have to conclude there is no certainty that the built application will perform adequately when the components are assembled into a whole application. In which case, the need for more granular performance requirements and early testing is even greater.

Strategies for absent performance requirements

Most performance requirements are cast in business terms as end-to-end transaction responses, database capacities, and operational SLAs. Developers can easily argue that performance testing of individual

The Benefits and Practice of Early Lifecycle Performance Testing

In fact, we find that adopters of early performance testing can routinely create reasonable working performance requirements for components. They sometimes do this in two distinct ways, depending on whether they have a reference point or not. If there is no existing system or similar components, they develop low-level performance requirements in a simplistic, uncomplicated way taking high-level requirements and using crude averages for, for example, numbers of methods per business transaction and an assumption about client-end and network latency. If there are existing systems, they use past performance as a baseline. Agile developments have the advantage that they quickly produce baselines from prior iterations. Crude as they are, these two approaches produce reasonable results, especially coupled with a developers experience and insight into the code. The information you get from such tests may not demonstrate definitively that your codes performance is adequate. But when a test like this fails, it is a useful signal to a developer to investigate more closely. None of this is intended to remove the need for good performance requirements for risky components, but these methods serve a useful purpose when you need to be pragmatic. minority of software delivery projects. When enhancing existing systems, there is often a complete application to test, one which is built with one or more new components. So the objection that you dont have a full application at the early development stages turns out to be less true than imagined. The objection that the development infrastructure is not sufficiently close to production is also surmountable. The analysis to compare a development systems capabilities to a production system is feasible. Capacity planners do this sort of calculation all the time. Putting aside for the moment the process management issues for maintaining a performance testing environment for developers, some organizations do indeed have available test environments run as labs with known relationships to production. While these are generally utilized for late cycle performance tests, they could be made available for early cycle testing too. Testing outside the coding environment may entail changes to source management and build procedures, but these are not insurmountable obstacles either. In other cases, the capacity and scale of the development environment compared to the production environment are well understood and can be used to calibrate results. Once again, Agile iterations make such comparisons easier. In our pragmatic approach to early testing, we can work with ballpark figures. Development environments are less powerful than production environments but they are also less loaded. A component that performs badly in an early cycle performance test is an early warning, and a signal to investigate further. You still need the big event near the end Just as you must still do system testing after early functional testing, you must still expect a full late cycle performance test. However, you should enter the late test phase with

9 PAGE

Strategies for working with nonproduction environments

As a rule, developers have neither a complete application nor a production-like environment on which to perform early performance testing, but, actually, sometimes they do. We tend to look at changes in software delivery processes in the context of brand new systems, but they represent the

The Benefits and Practice of Early Lifecycle Performance Testing

fewer uncertainties and more confidence in code performance. You will find fewer coderelated performance bugs and save time and costs from less rework. instils confidence. In return, coders should be clear that they become as accountable for early performance tests whatever scope you have chosen as they are for code and functional unit tests. Integration with existing processes A software delivery organisations process maturity matters. You need to be already operating at a mature level of testing process before you undertake early performance testing. We dont see any intrinsic unsuitability for early performance testing within Waterfall or Agile projects, but Agile does have some characteristics that make introduction of a new test type a little easier, for example: The cross-disciplinary nature of Agile teams means they are used to coders and testers working together Agile iterations quickly begin to provide the information needed to define component level requirements and map development environment performance onto production performance The Agile culture is generally more dynamic and accepting of change and uncertainty. Agile developers are more pragmatic about incomplete requirements.

Introducing Early Performance Testing Practices

Introducing early performance testing should be treated like any change management project. Good communications and sensitivity to people are always essential. However, here are some specific points to keep in mind. Developing motivation

10 PAGE

You need to convey two distinct messages: that the benefits of early cycle performance testing are worth it and that it is feasible to do. The starting point has to be a clear explanation of how finding defects early has both a financial benefit to the business and improves project manageability and predictability. It must be made clear that project delivery schedules will be adjusted to take into account the additional up front work, and that any learning curve is built-in. Whats in it for those who will adopt the practice? It depends on your organisations culture. The certain benefit is that for an additional effort in development, there will be far fewer awkward disruptions later for debugging and rework. Beyond the benefits, developers need to be reassured of the feasibility. For this you will need to ensure that the change management project includes a skilled champion or evangelist someone who

We recommend that your process include risk-based analysis for deciding whether to do early performance testing on a component, how much and what type (based on the types we outlined earlier.) There are several existing processes that are likely to be touched by early performance testing:

The Benefits and Practice of Early Lifecycle Performance Testing

Coding stage testing Build process and/or continuous integration Source code management. We would expect to see processes for the test types we identified earlier assigned to the coding stage. Mature integration and build processes, with integrated functional unit test suites, can be extended to included performance tests as well. Those performance tests could include any component performance tests, but also further performance tests on the built assembly of components. It is at this stage that some early load testing becomes feasible in addition to the early background load testing you can do on components. Changes to source code management and build procedures may be needed if performance tests are executed outside the coding environment. The pragmatic approach should be encouraged across the board to ensure flexibility. It is essential that imperfect knowledge of requirements and environmental behaviour not be used as an excuse not to test, where experience and common sense could with the right attitude plug the gaps. the industry has developed with its bias for late-cycle testing you may find that performance testing specialists may feel threatened that their specialism is being undermined.

Tools
Performance testing of all types requires tools. For early performance testing, you need tools that fit in with coding and integration environments and that dont require all the scaffolding and preparation of the set piece late cycle performance test. They need to support middleware and backend components as easily as they support client-end scripts. One commonly held idea is that functional unit tests can form the basis of performance tests. More often than not, this proves not to be true, so dont rely on it. It is worth pointing out that for background load testing where you are just trying to create a background of machine loading against which you will run manual transactions you dont need as fine a control of the timing and ramp up of automated scripts as you need for full load testing. Virtual user scripts and orchestration by a virtual user controller can be relatively simple. You are not concerned about the performance of the individual virtual user transactions; you are interested in the load they place overall on resources. A key factor in the cost-benefit equation of early performance testing is the cost of tools. When they are available in developers IDEs the marginal cost is low. However, the cost of virtual users for driving load tests can become a significant expense. It is important to make a reasonable stab at estimating how many are really required. On one hand, there is a tendency to over-

11 PAGE

Specialists
You will need the buy-in of those who are usually involved in late cycle performance testing, including any specialist performance testers, architects, infrastructure experts (network, DB, service orchestration). They will need to change their own processes to include support for the early performance testing. This is sometimes viewed as an opportunity because they have been at the sharp end of the problems inherent in leaving all the testing towards the end. However, as

The Benefits and Practice of Early Lifecycle Performance Testing

specify the number, perhaps in the interests of expediency. On the other hand, it can be a mistake to scrimp and attempt to make up for the shortage by distorting the content of the scripts. The essential point is that you want the right number of virtual users to achieve your goals, whether that is 25, 1000 or more. And one final soft issue: you will need to develop the skills of tool specialists who can mentor others.

Conclusion
Early performance testing may not be the first test process improvement a development organization chooses to make, but there is no good reason why it should be as overlooked as it is. Agile processes and powerful development environments with integrated testing capabilities are conducive to a pragmatic approach. Determination to obtain the benefits of lower rework costs and more manageable projects provides the motivation.

12 PAGE

Microsofts ALM Assessment has been designed by expert ALM practitioners to help you understand how well your organisation currently approaches ALM and where and how you can make improvements. To find out more, click here.

EuroSTAR

Software Testing

C o n fe re n c e

Dont forget to join us for EuroSTAR 2012 in

Amsterdam from the 5 8 November. Its our 20th anniversary and we are going to

EuroSTAR
celebrate in style!

Software Testing

Community

EuroSTAR

Join the EuroSTAR Community

Access the latest testing news! Our Community strives to provide test professionals with resources that prove beneficial to their day-to-day roles. These resources include catalogues of free on-demand webinars, ebooks, videos, presentations from past conferences and much more...

Follow us on Twitter @esconfs

Remember to use our hash tag #esconfs when tweeting about EuroSTAR 2012!

Become a fan of EuroSTAR on Facebook Join our LinkedIn Group Contribute to the EuroSTAR Blog Check out our free Webinar Archive Download our latest ebooks

The EuroSTAR Blog

Written by Testers for Testers

w w w. e u r o s t a r c o n f e r e n c e s . c o m