WHITE PAPER

The Test Engineering Maturity Model

Abstract
Recent software testing research indicates that under pressure to meet deadlines, the testing teams often lose focus on the business goals expected from the activity. This white paper talks about the Test Engineering Maturity Model (TEMM) that Impetus Technologies has developed. TEMM helps clients meet software testing goals while keeping continuous focus on business needs. The paper talks about how this framework helps companies save on time and costs, while improving the overall quality of the product. The five levels of the TEMM have been described and explained. The white paper also talks about the benefits of each level, and how they impact the testing function.

Impetus Technologies, Inc. www.impetus.com May - 2012

The Test Engineering Maturity Model

Introduction
For most companies, software testing is an arduous and expensive activity. Organizations are often faced with a lack of time and resources, which limits their ability to test each and every line of code during the test execution cycle. It therefore becomes imperative for them to optimally utilize the limited resources available within time-to-market constraints, to deliver better quality products. The size of regression test suites grows as software evolves. Their size grows to such an extent that regression testing alone can account for as much as 50 percent of the cost of software maintenance. Due to various constraints, it is not possible to test each and everything evolving in a software product. Typically, such testing increases the cost and time associated with measuring quality and stretches the release cycle without providing an adequate Return on Investment (ROI). To meet business deadlines and measure product quality, stakeholders use cost and time as the main driving factors. It is often seen that companies try to measure quality based on cost and time, which leads to unsuccessful testing endeavors. While a huge amount is spent on testing, the product outcomes are not up to the mark and make companies feel that testing has lost track, somewhere. There is a clear relationship between the three elementary forces i.e. Time, Cost, and Quality in any project. Time denotes the available time to deliver the project, cost represents the amount of money or resources available and quality represents the fit-to-purpose that the project must achieve to be a success. A product is always created to meet business needs. Business is the driving factor that decides on parameters such as time, cost, and quality. However, as the delivery pressures increase in a release cycle, and companies focus on cost and time to achieve a predefined quality, the concept of business loses shine. It is therefore important to ensure that business is not ignored in a testing environment. Industry is demanding a comprehensive testing package, a package that can help in balancing between business, cost, quality, and time. What is required then is a framework that takes into account the four pillars associated with a product, and how these work in relation to each other. Impetus Technologies testing practice offers comprehensive end-to-end test engineering solutions in order to reduce the product development costs, improve quality, and reduce time-to-market. It has developed the Test Engineering Maturity Model, which helps in showing a definite direction for complete product testing.

The Test Engineering Maturity Model

The key objectives of TEMM

The TEMM focuses on the following: Resolving the ambiguities arising during software product testing Providing a definite direction to attain higher ROI in product testing by keeping business goals and risks within sight Defining the right set of Key Testing Activities (KTAs) Providing a matrix with business, cost, quality and time as the primary components Offering five levels, which serve as benchmarks for any time of testing Enabling organizations to opt for a desired level, based on the their product testing requirements

Overall, the Test Engineering Maturity Model, provides companies with a set of guidelines for improving their software testing.TEMM defines Key Testing Activities, popularly known as KTAs that helps in providing a definite direction to companies for their testing endeavors and helps in attaining a higher Return on Investment (ROI). The model removes the need for organizations to reinvent the wheel. TEMM provides incremental and non-linear benefits to the teams that want to mature their testing activities. All they need to do is select the level they want to achieve, based on their business requirements and follow the activities that are defined.

The Test Engineering Maturity Model

TEMM structure

TEMM reference table

TEMM reference table can be used to get quick information on levels scope, outcome and limitations. Depending on the requirement of the project and the maturity level that any team wants to achieve, an appropriate level of TEMM needs to be selected. Each level defines the key testing activities that will help the team to achieve the desired level. Identifying the layer and implementing the activities will provide incremental benefits to the product testing teams, level over level.

The Test Engineering Maturity Model

Level

Type

Scope

Outcome

Limitations

Illusive

Crude product testing

Ad-hoc

Test coverage based on time available before release

Product quality mainly decided based on gut feeling of test team Domain/Application knowledge resides in individuals Testing activities are person driven High probability of critical bugs getting leaked into production

Primitive

Elementary product testing

Manual-UI, Functional, Integration, System, UAT Ad-hoc - NonFunctional

Basic manual testing is planned and performed on the product Well defined test cases for the covered areas

Testing on different environments is a cumbersome task Longer testing cycles Non-Functional aspects of the application are not completely tested High probability of critical bugs getting leaked into production

Reflexive

Essential product testing

Manual- UAT and NonFunctional Classic AutomationFunctional and White box

Complete, planned manual testing is performed on the product that includes functional and nonfunctional testing UAT is planned and executed beforehand, minimizing surprises at later stages Prioritization helps in minimizing defect Leakage into production

Automation remains secondary so all the benefits of automation are not reaped Lack of automation strategy results in high maintenance of test scripts Requires longer implementation time

The Test Engineering Maturity Model

Systematic

Streamlined product testing

Manual- Functional and Strategic NonFunctional Planned AutomationFunctional and NonFunctional

Completely planned Testing activities, both manual and automation Repeatable tests on different versions of an application, even if the application changes is enhanced Reduced regression test cycles Increased test coverage Unattended execution of test suites Optimal utilization of resources

The testing metrics do not give complete view of the product status

Comprehensive

Progressive product testing

End-to-End (E2E) Automation Engineering to Testing (E2T) Automation Continuous Improvement

Automation of major activities involved in product testing Automation of End-to-end business scenarios Continuous Integration helps in detecting defects early in the life cycle Visibility into complete product status from both the engineering and testing perspective Continuously look for ways to improve the current practices

The Test Engineering Maturity Model

Summary
To minimize testing nightmares and enable the software product testing process to achieve maturity, the thumb rule is to revisit business needs at every stage. The participation of business at every step in the testing cycle ensures that the critical factors are balanced and producing positive results that meet business objectives. Cost, quality, time, and business are the inseparable axes that will certainly lead to the path of productive testing. By implementing TEMM, companies can quickly meet their desired quality objectives, while focusing on critical business needs. Using the designed path, that touches each and every aspect of the testing cycle, TEMM enables enterprises to mature their testing activities, keeping process implementation worries behind. As organizations move from Level 1, through Level 5, the testing activities will yield higher benefits in terms of quality and business improvementwhile keeping costs and time at a minimum. Impetus Technologies TEMM assessment methodology helps analyze and review the testing progress, and provides updates on the exact level achieved and the requirements for reaching the next stage.

About Impetus Impetus Technologies offers Product Engineering and Technology R&D services for software product development. With ongoing investments in research and application of emerging technology areas, innovative business models, and an agile approach, we partner with our client base comprising large scale ISVs and technology innovators to deliver cutting-edge software products. Our expertise spans the domains of Big Data, SaaS, Cloud Computing, Mobility Solutions, Test Engineering, Performance Engineering, and Social Media among others. Impetus Technologies, Inc. 5300 Stevens Creek Boulevard, Suite 450, San Jose, CA 95129, USA Tel: 408.213.3310 | Email: inquiry@impetus.com Regional Development Centers - INDIA: New Delhi Bangalore Indore Hyderabad To know more visit: www.impetus.com

Disclaimers
The information contained in this document is the proprietary and exclusive property of Impetus Technologies Inc. except as otherwise indicated. No part of this document, in whole or in part, may be reproduced, stored, transmitted, or used for design purposes without the prior written permission of Impetus Technologies Inc.