Life Imitates Code. Essays by Ka Wai Cheung from around the web.

TESTING APRIL 2015

Where automated testing

should (and shouldn't) fit in
your strategy
Having worked with automated testing in various forms for a few
years now, Ive come to a few conclusions about where it fits in the
mindset of a software developerand about how it can become a
crutch in programmer behavior if we fit it improperly.

Testing vs. checking

For a while now, somethings felt strange to me about the way that
automated tests are often marketed to the development community.
For one, I think the term test itself is quite misleading. It conjures up
this idea that weve found a viable replacement to manual testing; that
if all tests passwe are guaranteed bug-free software. But,
automated tests arent really testing a system at-large, so much as
they are checking specific behavior within a system.

For example, suppose I want to test out the security around our
billing page. I could write a couple of integration tests, by using a tool
like Selenium, to confirm the following assumptions:

If I login as an account owner, I should be able to access the

billing page.
If I login as a normal user, I should not have access to the billing
page.

These tests confirm what we expect to happen actually happens, but

they dont confirm that what we dont expect to happen (because we
arent testing for it) actually wont happen. For instance, neither test
would prove that going to the billing page doesnt, say, inadvertently
kick off a process that deletes your user account, or the infinite
number of other possibilities that could go wrong. We arent testing
that a system actually works; We are merely checking that specific
assumptions we made arent broken.

The same is true for unit tests. While they test very narrow and well-
defined functionality, they arent going to test for, as Uncle Bob states,
the stuff out at the boundaries of the system. Also, with unit tests,
there is often the talk of 100% code coverage. James O Coplien
argues that its pragmatically impossible to achieve if we define this
as:

having examined all possible combinations of all possible paths

through all methods of a class, having reproduced every possible
configuration of data bits accessible to those methods, at every
machine language instruction along the paths of execution.

If we truly achieved this, he argues, wed have on the order of trillions

of scenarios to test for even modest software.

So, where does this get us? I often visualize automated tests like
building a frame (the tests we write) around a constantly morphing
structure (the true behavior we want). We can nail lots of posts
together to build a frame that begins to resemble the behavior we
want, but well never quite get there.
 Writing automated tests provide basic boundaries for bug-free code, but are by no means an
end-all solution.

So, whats a better way to describe these things? The term

automated checks feels much more appropriate to me, and others
agree. As James Bach writes, there is a big difference between what
humans can do and what automated tools can do.

In the Rapid Software Testing methodology, we distinguish

between aspects of the testing process that machines can do
versus those that only skilled humans can do. We have done this
linguistically by adapting the ordinary English word checking to
refer to what tools can do.

With that said, Bach comes up with a differentiation between

software testing and software checking.

Testing is the process of evaluating a product by learning about it

through exploration and experimentation, which includes to
some degree: questioning, study, modeling, observation,
inference, etc.Checking is the process of making evaluations by
applying algorithmic decision rules to specific observations of a
product.

Automated tests are really automated checks. And, checking is just

one small part of a complete testing strategy. Its one kind of
insurance policy amongst a whole suite of policies designed to make
bugs more unlikely (but, certainly, not impossible) to introduce into a
system.
Maintaining your own human vigilance against bugs

So, whats the big deal about calling a check a test? When we start
becoming overly confident in our ability to write bug-free code simply
based on a successful series of automated tests, the returns begin to
diminishrapidly. Our minds start to let go of some of the natural
deliberateness we may have once put into code prior to the feeble
safety net of automated tests. At its worst, it means we become
narrowly-focused developers, using one relatively brittle measure of
success as false justification that our testing is now complete.

If we were to truly test every possible scenario in even a modest

system, there would quickly be orders of magnitude more tests to
write than would be practical. We best rely on a series of other
strategiesin addition to automated checksto cover all the other
permutations.

The TDD debate and the real question we should be

answering

Is test-driven development (er, check-driven development) a good

approach to writing code? If you read the tech pundits today, most
everyone has a very strong opinion on the matter. But, I think were
asking the wrong question. Asking if you should be employing TDD is
like asking a basketball coach whether they should play a 2-1-2 zone,
2-3 zone, or man-on-man defense. They can all work and they can all
not work. Ive seen quality software written both with and without a
TDD approach.

The real question we should ask is, how do we pragmatically make the
introduction of bugs as improbable as possible, (while weighing all the
other important matters of software, like new features, deadlines,
etc.)? To me, it starts with the right programmer mentality.

We should be talking just as much about these other things, because

they are just as important, if not more important, than automated
testing:

Writing expressive code. Many bugs and points of confusion can

 be mitigated by focusing on code clarity. As Phil Karlton once
said, naming things is one of the two hardest skills in
programming. Devoting more time to precise and meaningful
naming can be a big benefit.
Peer code review. Have your colleague review code youre going
to check in to play devils advocate. (The Git pull request concept
is a great facilitator for this). They will almost certainly see your
code from a different angle, and theyll spot things you might not
have been looking for.
Writing in small iterations. Get in the habit of committing
incremental changes to your source control repository. This
helps someone else visualize how you got from point A to point B,
and makes it easier to see where things might have gone wrong.
Use the software you write. This sounds so simple, but is actually
a thing most of us dont really do fully. So many obvious bugs
would be caught if we took off our development cap periodically,
closed the debugger, and used the software as a consumer prior
to committing any code. Even with the luxury of a dedicated QA
team, this is something every developer should put into their
own arsenal.
Think through all the possibilities. Automated checks are only
as good as what they test for. So, we should spend more time
discussing what to test. Ive written about knowing what to test
in a couple prior posts:

Here are five cases to test with every kind of object you manage
in your system.
Heres a more general approach to thinking through your
checking scenarios.
Finally, a general heuristic for tackling mysterious bugs.

To be clear, automated checks have a very important place in building

sound software. But we need to be careful with our expectations.
Placing too much value into them might give us a false sense of
security making us even more prone to introducing bugs. We need to
make sure that our long-tested human processes still play a majority
role in our overall testing strategy.

Originally published Apr 28, 2015 at DoneDone. Go to the next essay in Testing, Cut the
problem in half.

Tweet Share

All 4 essays in Testing

Where automated testing should (and shouldn't) fit in your strategy
Cut the problem in half
The hard thing about testing
Want fewer bugs? Test these five cases

Other categories
Business & Work
Coding
Design
Teaching
Testing

Miscellaneous

All essays (by year)

About the author
Random essay
The Developers Code
Follow @developerscode