Sie sind auf Seite 1von 14

Tackling Software

Testing Challenges
in the Agile Era
About the Authors Chapters:

Vu Lam is CEO of QASymphony, a leading 1. Reimagining the Tester

developer of Quality Management solutions for the Agile Age
for software developers and QA testers. He (Vu Lam)
was previously with First Consulting Group
and was an early pioneer in Vietnams
offshore IT services industry since 1995. 2. Finding the Right Mix With
An entrepreneur at heart, he started multiple technology
ventures, including Paragon Solutions, a technology
Your Testing Methods
consulting company with international development centers (Sellers Smith)
in Bangalore, India, and Ho Chi Minh City, Vietnam. Following
Paragons acquisition by NASDAQ traded FCG, he led the
growth of the organization from 400 to 1300 resources. In 2011, 3. Surprising Truth About
he co-founded QASymphony, a vendor of software testing Exploratory Testing
tools, with Josh Lieberman. He holds a MS degrexe in electrical (Vu Lam)
engineering from Purdue University. You may reach him at
4. Building Good Tests:
Why, What, How?
Sellers Smith is Dir. Quality Assurance & (Sellers Smith)
Agile Evangelist for Silverpop of Atlanta,
GA, a digital marketing technology provider
that unifies marketing automation, email,
mobile, and social for more than 5,000
global brands. Prior to Silverpop, he led
technical implementation of new Internet business initiatives
for WebMD,, and the e-commerce
site for and He has held
software engineering positions at Turner Broadcasting, A.G.
Edwards & Sons, and MITRE. Currently, Sellers is leading
the implementation of Agile Testing practices including
Acceptance Test Driven Development, Specification by
Example, Test Automation using FitNesse and Exploratory
Testing. He has a MS degree in Software Systems Engineering
from George Mason University and a MBA in Information
Systems from Rensselaer Polytechnic Institute. You may
reach him at
The testing industry is undergoing some rapid and The final chapter focuses on the practical side of
far-reaching changes driven by major shifts in the building good tests by breaking the process down into
software development landscape. Software is moving a deceptively simple trio of questions: why, what, and
from concept to end user faster than ever before and this how? By applying these queries testers can identify the
throws up a whole new set of challenges for testers. The value of individual tests to ensure that maximum value
traditional procedures and techniques that have served is extracted from each new test cycle. This ability to
QA departments in the past are no longer up to the job. continually reassess and refocus effort in the right places
The techniques and tools required for effectively testing is the key to delivering for the business.
modern software has evolved way beyond the traditional
plan test write test run test model of yesteryear. QA departments are often left to their own devices.
Excluded from exciting new developments and last
In this eBook we bring together the thoughts of a vendor to the table for budgetary consideration, testers are
-- intent on building something that truly speaks to the being squeezed to deliver more with less. The testing
needs of the modern tester -- with a practitioner from community has to get more active and vocal about the
the testing trenches, to bring you a multidimensional challenges facing them. A love of the craft and real
perspective on the issues testers are facing and how they dedication is the first step towards challenging that
might be mastered. underappreciated status.

We begin with a chapter on reimagining the tester for We offer this eBook in the spirit of opening a dialogue.
Agile. The Agile mindset has transformed how software The ultimate hope, which well delve into in more
is developed, but QA departments have been slow to depth in our conclusion, is that the testing craft can be
adapt their own methodology. Testers are uniquely professionalized further. Good testers have key skills
placed to serve as advocates for the end user. Testing and expert knowledge thats in serious demand. By
practices must be modernized; testers have to be in the adapting to seismic shifts in the wider software industry,
development loop to gain the insight they need to test developing new approaches, and finding new tools,
effectively, and realistic planning is paramount. testers can stay relevant and effective.
Our next chapter explores how testers need to identify
the right balance of testing methods if they are to
establish a comprehensive testing strategy. We discuss
automated, exploratory, and user-acceptance testing,
with advice on when to employ them and how to get the
best from each approach.

In the third chapter we drill down into exploratory

testing and analyze its origins to uncover its essence.
The time pressures of modern development make it
an indispensable technique. Continuous deployment
and the fast feedback loop which drives new features,
necessitates a flexibility and focus that can be found
through exploratory testing.

Reimagining the Tester

for the Agile Age
Vu Lam

The ideal tester is an advocate for the end user.

Its been 12 years since the Agile Manifesto was created, and the majority of
developers have jumped on the bandwagon since then. Agile methodologies are
now dominating the software development landscape. It has overtaken Waterfall
as the dominant approach. But in the rush to implement the Agile mindset,
something has been overlooked the tester.

The focus on delivering a good product without endless documentation is

undoubtedly a positive move for developers. By keeping the needs of the end user
in sight at all times, and creating feedback loops, projects generally deliver better
software, and faster than before. Given that the Agile method was conceived by
developers for developers, it left the testers in the dark. As a report from Forrester
points out, Agile teams must invite testers to the table.

Dangerous Assumptions At first, testing each sprint is not

It should be obvious that Agile a problem, because testers have a
development completely disrupts limited number of features to test,
traditional testing practices, but as the software grows they
yet many companies have done have to test new features, verify
nothing to update processes or bug fixes, and perform regression
arm testers for the new challenges testing. Everything has to be
they are facing. tested within the sprint timeline;
the shorter it is, the more rounds
Imagine a typical scenario on an of regression testing is called
Agile project with a team of five for. Soon, the test team is bogged
developers, and a pair of testers. down and overloaded.
From the development point of
view, five developers can work Automated Unit Tests developed
at a steady, measurable velocity, alongside with the code helps to
adding new features to each new relieve some of the pressure, but
build. But that simply isnt true the workload for the testers still
for testers. remains high. How can the same
QA Symphony | Reimagining the Tester for the Agile Age | Page 5

two testers cover all the new There has to be a change in They need tools to support the way
features alongside fix verification routine: leverage Exploratory they explore and test software in a
and regression testing? The testing to put new functionality fluid, unscripted manner. It should
answer is simple: they cant. through its paces, write a limited be fast and easy to identify and
set of core test cases for future document test cases for regression
A New Approach regression testing, and automate and possible automation. They
In order to ensure that the quality them as soon as the software in also need a test management tool
remains high, management that area is stable to reduce a that fully supports all the Agile
could scale up the test team manual testing burden. testers tasks, not a tagged-on
significantly as the development module to the development teams
progresses, but there is a much In the absence of detailed project management software.
more cost-effective solution. requirements and documentation, Something dedicated that can
Regression testing has to be Agile testers must understand link user stories and test cases,
documented and then automated the needs of the end user. It is import and export bugs, and keep
as you progress. Theres no way vital they can put themselves a complete record of the testing
to write up these test cases or that has been completed and what
scripts before the code has the results were. Something thats
been delivered. truly built for the Agile tester.

New feature testing and bug Time For Agile Testers

validation is where you So much effort has been put
want your testers to focus. into creating Agile practices and
You can avoid duplicating tools for developers, isnt it about
work by using the test flow time that testers were given due
they are capturing as they attention? For all the good that
test to generate test cases Agile development brings, it doesnt
that can be used to automate dispense with the need for skilled
the regression testing for testing if you fully expect to deliver
the next build. Its an all-at-once an excellent product for users.
approach that requires testers to in the end users shoes and have
be on their toes. a good understanding of exactly Integrate testers into the Agile
what they need from the software process from the start, equip them
A New Tester and how they are likely to use it. with the right tools to help them
For Agile testing to work, testers This means full participation in develop exploratory skills and
need to adopt the same Agile Scrum meetings, absorbing user use automation to cope with the
mindset as developers, but stories, and asking questions repetition of regression testing,
ditching the traditional waterfall whenever something isnt clear. and you have yourself a tester for
process for sprints does require The ideal tester is an advocate the Agile age.
some upfront planning. for the end user. ###

Previously, the testers For Every Nail Theres A Hammer

routine involved reading and Looking beyond the mindset, Agile
understanding requirements, testers need the right tools for
writing test cases before the the job. The vast majority of Agile
build arrived, and testing the project management software
software when the build was ignores the vital role of the tester.
made available. For the Agile Test management solutions were
project, the tester cannot afford created to support more traditional
to write test cases before the build testing. In short, project and
arrives since theres not enough testing software have not caught
detail available. up with the unique demands of the
modern Agile tester.

Finding the Right Mix with

Your Testing Methods
Sellers Smith

With the right balance of automated, exploratory, and user-acceptance

testing, QA teams can help build great products.

Any comprehensive testing strategy is going to rely on the right mix of tests for
the product. There are various methodologies that can be employed, but rather
than viewing them as separate, discrete approaches, it is worth remembering that
they all lie on a common continuum. At one end we have mechanistic automated
testing, and we pass through scripting, beyond exploratory testing, and on to
private and public betas at the opposite end.

The testing journey starts from a tightly controlled base and winds its way towards
real world conditions. Were going to focus on three important stops along the way
in this article and explain a little about how they bring value to the process, what
their weaknesses are, and how you decide when to use them.

Automated Testing Can you successfully complete

The testing of every product the checkout process?
will start with some scripted,
automated tests that are designed These are all useful things
to exercise the system and to know, but there are some
validate that it works the way it is obvious limitations. Automated
supposed to. Automated tests are tests cannot go beyond their
repeatable, fast, and they can be original scope. The lack of
run as many times as needed. human involvement means that
an automated test may run and
To give an example, if you were to successfully add items and then
test an e-commerce shopping cart, check out, but it wouldnt pick up
you might have an automated test on things like misspelled item
that tries to add each individual names or stretched photos. If it
item in your store and check out wasnt included as a parameter
successfully. Does the cart add when the automated test was
up the combined cost correctly? written, then it wont be picked
Is it possible to add each item? up when it runs.
QA Symphony | Finding the Right Mix with your Testing Methods | Page 7

Exploratory Testing this feedback before your product adopters. Feedback from beta
With any piece of software you can truly be finished. users is of limited value if primary
can use documentation, user usage scenarios are not working.
stories, and conversations with End users will always come up This approach can be used
key stakeholders to determine with scenarios that development iteratively around release and or
how a system should work. There and QA didnt consider. With features (for folks doing featurebased
are certain concrete rules to how the shopping cart, they might development).
the software should behave and fill it up and then come back a
automated or scripted testing can week later with the expectation In short, start with a well-defined
cater for those, but what happens that their shopping items will set of automated tests to validate
when someone goes off script? still be waiting for checkout. core business functionality, shift
Exploratory testing is about Unencumbered by knowledge of to exploratory testing as the core
combining your knowledge and the documentation, and not biased functions are working, and then
experience as a tester with an by proximity to the project, end begin introducing real users as
understanding of what the product users will try unusual things that the exploratory testing shows an
is supposed to do to uncover seem intuitive to them. That can acceptable level of usability.
potential problems that automated be great for discovering defects,
tests will not catch. but it also teaches you a lot ###
about expectations. This is how
It goes beyond ad hoc testing, your product will be used in the
because there has to be a clear real world.
area of focus and a predetermined
scope, but the tester is not bound Bringing It All Together
by a check-list approach. They The exact mix of tests will vary
can delve into how the experience depending on the type of system
feels, unusual scenarios that and users. Generally, systems with
may have been missed by the heavy business logic are going to
developers, and leverage specialist favor more test automation (e.g.,
knowledge related to the domain verification of business rules),
or industry that the software is while consumer-facing systems
aimed at. Using our shopping will favor more beta testing.
cart analogy again they might
add and then remove products As an example, Silverpop
from the cart, test how it deals provides marketing automation
with multiples, and see if they and email marketing in a SaaS
can exploit the system to produce model. The company focuses on
negative values and checkout with a larger automation test suite,
a refund. and does exploratory testing in
order to test complex usage of its
User-Acceptance Testing application. Silverpop has several
There is ultimately no substitute mechanisms, things like feature
for real world conditions. You switches and pilot environments
can test a product professionally -- to support beta users and
for months on end, but as soon early adopters.
as you release it into the wild
your end users will do a variety A good approach is to focus on
of unexpected things with it. automated tests (or scripted tests),
Whether you opt for a private beta, until you have a generally usable
which involves inviting some of platform. Exploratory testing is of
your prospective customers to limited use if the basic features
use the software, or you open it are not working. Once exploratory
up publicly and work fast to meet testing is underway, you can focus
expectations, you need to collect on introducing beta user or early

The Surprising Truth About

Exploratory Testing
Vu Lam

Exploratory testing is an important tool in any experienced testers kit.

In the modern age of Agile development, with software being updated
and released faster than ever, exploratory testing is nothing short of
essential. The trouble is that there seems to be some confusion over
what the term actually means.

What Is Exploratory Testing? tasks, objectives, and deliverables

Real exploratory testing is not the that make it a systematic
same as ad hoc testing; its much process and went on to say,
more sophisticated and organized In operational terms, exploratory
than that. The term was originally testing is an interactive process
coined by Kem Caner in his book, of concurrent product exploration,
Testing Computer Software, to test design, and test execution.
describe an alternative approach The outcome of an exploratory
to traditional formal planning. He testing session is a set of notes
implored testers to Trust your about the product, failures found,
instincts. But he also warned that and a concise record of how
you should always write down the product was tested. When
what you do and what happens practiced by trained testers, it
when you run exploratory tests. yields consistently valuable and
auditable results.
Another author, James Bach,
described exploratory testing More than a decade has passed
as scientific thinking in realtime. since Bach wrote those lines,
In his 1999 paper, General yet his wisdom is still lost on
Functionality and Stability Test many people.
Procedure, Bach expounded on
that, explaining that unlike
traditional informal testing, this
procedure consists of specific
QA Symphony | The Surprising Truth about Exploratory Testing | Page 9

When To Employ It Good testers will naturally The Right Approach

Its very common for testers to be develop the skills required for The surprising truth is that
under time pressure, and the shift exploratory testing; the ability exploratory testing encourages
to Agile methodology in software to think creatively and generate the right approach to testing any
development has had a tangible useful tests in real-time. A dogged, software. You need to build a real
impact on documentation. To put it rigid approach to testing software, insight into the product and think
simply: testers have very little time especially software that is rapidly imaginatively about how it will be
to test a new build and must often evolving, does not make the best used. Combine that creativity with
do so without access to detailed use of a testers intellect, and metrics and detailed reporting, and
documentation. wont result in the best final you can see the big picture. Unlike
product possible. ad hoc testing, exploratory testing
Exploratory testing is ideal for is an organized, well-structured
this scenario, but should not Extracting Value approach that lets testers excel
be an exclusive approach. It Youve established your in the ever changing landscape
complements scripted testing boundaries and the scope of your of agile development.
activities. Exploratory testing testing session upfront. In order
is simply part of an overall plan to get the maximum benefit from ###
that must include new feature your discoveries you have to
exploration, bug validation and generate reports of your findings.
regression testing.
You should be recording your
If youre not sure what to test next, expectations and the actual results.
or you want to dig deeper into a Were you able to achieve your goal?
complex piece of software, then its Did it take longer than expected?
time for some exploratory testing. Did you encounter weaknesses
in the software? Remember that
How To Do It Right another tester, or a member of
What youre really looking to do the development team, should be
is design and execute tests at the able to follow your thoughts and
same time. Although you wont reasoning. Clarity is vital.
be using requirements to prepare
detailed test cases before you begin, Because you have a frame of
that doesnt mean you should test context the data you generate
randomly. There should be a focus can really benefit the development
on specific parts of the product, or team. Instead of the
an agreement to employ specific isolated defects, suggestions, and
strategies during the testing. concerns that an ad hoc testing
Combine your intuition with the approach might generate, you get
knowledge you have amassed on in-depth information on how well
the software from previous tests, the software meets intended goals
and your perceptions about the and how it will be perceived by
needs of the end user. You must the end user.
understand what the software
is supposed to do. Then you can In addition to verifying software
appreciate how developers tried and discovering issues,
to solve specific problems and exploratory testing can help
decide whether they accomplished identify useful test cases for future
their objectives. By leveraging regression testing. An exploratory
your insight and targeting your testing session can be the bridge
exploration, new issues will be between unscripted and scripted
discovered. This is not a random testing for your project.
bug hunt.

Building Good Tests:

Why, What, How?
Sellers Smith

Good testers and good tests always retain and use an awareness
of what the intended audience wants and expects.

Tests are an investment in the quality of any given system. Theres always a cost
to build, run, and maintain each test in terms of time and resources. Theres also
a great deal of value to be extracted from running the right test at the right time.
Its important to remember that for everything you do test, theres something
else youre not testing as a result.

Understanding that some tests are more important than others is vital to
creating a useful and fluid test plan capable of catering for modern software
development techniques. Traditional waterfall development -- where everything
is delivered for a specific release date in a finished package -- has been
succeeded by continuous feature roll outs into live systems. This necessitates
a different approach from QA departments.

How Do You Build Good Tests? Why is a higher level overview must be able to add and remove
You cant design or execute the that really ties into the business items from their shopping cart, or
right tests without understanding side. Its the big picture thinking that they shouldnt be able to add
the intended purpose of the system. that reveals why youre building something thats out of stock.
Testers need to have an insight the software in the first place.
into the end users expectations. What audience need is your How relates to the practical
Communication between the product fulfilling? For example, application of your testing. How
product people at the business end, we need to build an e-commerce exactly is the software going to
the engineers working on the code, website to sell our product to the be tested? How is the success and
and the test department enables general public. failure measured?
you to score tests in terms of their
importance and work out where What is really focused on Good tests are always going to
each test cycle should be focusing. individual features or functions cover our trio, but it can be a useful
of a system. Using a shopping exercise to break things down.
We can break it down into three cart analogy for an ecommerce
simple queries: why, what, and how. website, you might say that users
QA Symphony | Building Good Tests: Why, What, How? | Page 11

The Why
If you get too caught up in the what and the The value is determined by multiplying two factors:
how its possible to miss the why completely Impact to the user what are they trying to
and its the most important element because it accomplish and what would the impact be if they
dictates that some tests are more important than couldnt? How critical is this action?
others. The business case for developing your Probability of failure how likely is it that this code
software in the first place has to remain front will fail? This is heavily influenced by how new it is
and center throughout the project. If you begin to and how much testing it has already undergone.
lose sight of what the end user needs, then you
could be focusing your efforts in the wrong places. If we return to our ecommerce website analogy then
Delivering value to your customers is critical. Good we could take the action of being able to buy goods,
testers and good tests always retain and use an clearly thats essential, so it would be assigned a 3.
awareness of what the intended audience wants However, the functionality for adding goods to the
and expects. basket and checking out has been there since day
one, so it has already been tested quite extensively,
One technique we can employ is risk-based but some new features have been added which could
analysis of tasks. With risk-based analysis, we impact on that code, so that might result in a score
can arrive at a numerical value for each test of 2. Multiply the two together and youve got a
which gives you a sense of its importance. 6. This figure will change over time, because
We can assign a score of between 1 and probability of failure will go up if this part of
9 to each test. At the top end, a score of the system is significantly altered, and it
9 would be a critical test, and at the will go down over time if it isnt. Theres
other end of the spectrum, a score also a discretionary factor that might
of 1 might indicate a test that only lead you to bump that 6 up to a 7 if you
needs to be used sparingly. feel its merited.

The What
Testers come up with specific scenarios of how an Adopting a technique like Specification by Example
end user might interact with the software and what or Behavior Driven Design, youre going to lay each
their expected outcome would be. A typical test may test out in this format:
consist of many steps detailed in a script, but this Given certain preconditions
approach can cause problems. What if a new tester When one or more actions happen
comes in to run the test? Is the intent of the test clear Then you should get this outcome
to them? What if the implementation of the feature
changes? Perhaps the steps no longer result in the Regardless of the specifics of the user interface, or
expected outcome and the test fails, but that doesnt the stops along the way between A and Z, the Given,
necessarily mean that the software is not working When, Then format covers the essential core of the
as intended. scenario and ensures that the feature does what
its intended to do, without necessarily spelling out
The steps and scripts are delving into the how, but if exactly how it should do it. It can be used to generate
the what is distinct from the how then theres less tables of scenarios which describe the actions,
chance of erroneous failure. Allow the tester some variables, and outcomes to test.
room for an exploratory approach and youre likely to
get better results. If something can be tightly scripted, (Continued on next page)
and you expect it to be a part of your regression
testing, then theres an argument for looking at
automating it.
Building Good Tests: Why, What, How?
(Continued from pg. 11)

The How
Getting down to the nuts and bolts of how testers will create, document,
and run tests, we come to the how. Since projects are more fluid now and
requirements or priorities can change on a daily basis, there needs to be
some flexibility in the how and a willingness to continually reassess the
worth of individual tests. Finding the right balance between automated
tests, manually scripted tests, and exploratory testing is an ongoing
challenge thats driven by the what and the why.

Traditionally, manual tests have been fully documented as a sequence

of steps with an expected result for each step, but this is time consuming
and difficult to maintain. Its possible to borrow from automation by
plugging in higher level actions, or keywords, which refer to a detailed
script or a common business action thats understood by the tester.
Theres also been a move towards exploratory testing, where the intent
of the test is defined, but the steps are dynamic. Finally, theres a place
for disposable testing, where you might use a recording tool to quickly
document each step in a manual test as you work through it. These tests
will need to be redone if anything changes, but as its a relatively quick
process, and youre actually testing while you create the test, thats not
necessarily a problem.

Continuous assessment
Each individual test should be viewed as an investment. You have to
decide whether to run it, whether it needs to be maintained, or if its time
to drop it. You need to continually assess the value and the cost of each
test, so that you get maximum value from each test cycle. Never lose
sight of the business requirements. When a test fails, ask yourself if its a
problem with the system or a problem with the test, and make sure that
you always listen to the business people, the software engineers, and most
importantly the customer.

Every decade or so, we see a radical new idea or In order to spark the same level of activity and
technology emerging that completely changes the even controversy in testing that we see in software
software development scene, and moves the goalposts for development, we need to work together to drive the craft
testers. A trio of trends, in the shape of cloud computing, to become more professionalized. Generally speaking,
Agile development methodology, and the explosion in testing roles may not be valued in the same way that
mobile devices, have converged to give us software that developer roles are, but we can push to change that.
is put together quickly and made available cheaply. Seasoned testing professionals can act as advocates for
Drastically cutting the time from concept to market has that change, open lines of communication, and agitate
a big impact on the way software is tested and a big for the whole community to pull together.
impact on testers.
The testing profession should move to an expertise
The waterfall method necessitated meticulous planning value base, focus on improving skills and knowledge
from detailed requirements, and a linear development to create better career paths and ultimately take greater
schedule that dictated testing should follow the design satisfaction and pride in the important work it does.
and implementation phases. Its hopelessly outdated. Finding the right blend of techniques and tools,
the perfect marriage of process and function cannot
We are in the midst of this transformative period be achieved if we cant get beyond testing as
and if you find that youre struggling, then its time to an afterthought, or an addendum to development.
realize why. You need to catch up on what is changing,
understand how you can be a better tester, and adopt new Its time to reevaluate how we approach testing. Consider
techniques and tools that will help you to be as valuable tools that have been designed specifically to meet the
as possible to your company. needs of real testers in the field, and blend techniques
that are capable of providing the speed, transparency and
Its not a level playing field. There are lots of conferences flexibility that modern software development demands.
for developers, plenty of networking opportunities, Our comprehensive test management solution, qTest,
magazines and online articles sharing new ideas. Theres was designed to empower modern QA departments with
a whole community that targets developers with new the features and functionality they demand. We also
tools and products, a community thats geared towards offer qTest eXplorer, a tool that enables testers to capture
getting developers up to speed on new languages or screenshots, edit, annotate, and share them directly. We welcome
methodologies. Unfortunately, the support offered to your input and feedback as we strive to produce tools that
developers does not match or exist on the same scale put testers first.
for testers.
We hope you found this eBook useful and welcome your
Too many testers have their heads down, working away comments and input. We hope to foster a greater spirit of
on their own projects. Theres a lack of appetite for community across the testing profession.
forum discussions, conferences, or new techniques. This
lack of a community evangelizing new procedures and ###
spreading new ideas, makes it that much harder for best
practices and exciting techniques to spread. The field is
relatively slow to modernize. New trends are in motion,
and there are QA professionals pressing for change, but
many testers unfortunately are not aware it.
Join the conversation!

For more information on

QASymphony please contact us at:

Phone: 1-844-798-4386