www.martproservice.

com

Why Automation Projects Fail

(and what you can do to prevent failure)

Art Beall, Chief Technical Officer

11/22/08
 Contents Introduction
Automation is the new mandate in today’s
Introduction 2 fast paced world of software development.
Problem Statement 2 Companies are hard-pressed to get their
products out as quickly as possible while
MART Pro-Service Solution 3
hampered under Capex and Opex
Summary 3 constraints. Traditionally testing has long
been seen as the bottleneck to getting
software releases out the door and software
automation as the solution to removing that
bottleneck. However, software automation
more often than not fails to live up to
expectations and in fact seldom returns the
investment put forth.

Problem Statement
A majority of automation projects fail. In
fact, most if not all automation projects fail
to live up to peoples’ expectations and many
automation projects are flat out abandoned.
The reason for failure can be explained by
referring to the familiar expression “tip of
the iceberg”. Automation (like the tip of an
iceberg) seems rather like a small problem
superficially however later (often too late) it
is realized that the problem domain is much
bigger (like the true size of an iceberg) than
first realized. Often by this time test groups
either accept severe limitations in the
automation platform or they give up on
automation all together (like a ship colliding
with an iceberg). The impression among
many is that automation is simply a matter
of purchasing the “right” box and assigning
a resource or two to the project. However,
companies soon realize (to their surprise)
that their testing needs are not meet by the
“box” in fact they need significant
(unforeseen) investment to save the project
from failure.

11/22/08
MART Pro-Service Solution
This white paper will cover five key
areas to consider while planning,
implementing, and maintaining an
automation platform. Use these key
areas as a guide to help plan, deploy,
and maintain your automation
project.
Automation is akin to any
Complex Software System
Building Automation solutions is
akin to building any (complex)
Software system and requires the
same effort (usually high effort) to
implement effectively. You will be
integrating a system with many parts
as well as software (i.e. scripts and
glue software). Therefore the same
lessons learned in building systems
and software should be applied to
your automation project. Some items
to consider are:
1. What are the high level goals
of the automation system:
performance testing, unit
testing, feature testing?
2. Create a list of requirements.
What are the features you
need in the automation
system? Some examples are
listed below.
a. Logging capabilities
b. Graphical automation
scenario creator
c. API for a remote
procedure calls
d. Reporting
e. Resource
Management of
test/lab resources
f. Concurrent scripts
g. Capability to
preparing or “clean
up” the test
environment
h. Versioning of test scripts
i. Documentation for
administration/test script
creation.
3. Design and document an automation
architecture. An automation project
needs a framework and the
framework needs to be implemented
and documented. The framework
should lay the foundation and rules
that everyone lives under. For
example a shared software library for
automation scripts should be
implemented and used by all
automation script writers. Also, the
high-level format of a test case
should be specified and documented
with examples. Some other items to
consider are listed below:
a. Automation Architecture
must be extensible.
Automation requirements
will grow in step with the
requirements of the software
under test and the automation
system must be able to cope
with that growth.
b. Must have instrumentation
for debugging purposes.
Running scripts is the easy
part. Debugging scripts is
what takes the most time. Be
sure you have a system in
place that quickly
summarizes all the problems
from a test execution run.
c. Your scripts (and the
underlying libraries used by
your scripts) should be
modularized.
d. Keeping statistics on your
executions. Identify test cases
with a high false positive rate
and either fix the root cause
or remove from your
executions. Conversely,
 identify tests that find product? If so how likely is it
real software defects. these developers or
e. Can the automation development group will be
scripts be re-used available three to five years
later for different from now?
purposes. For
example you might
develop scripts for Consider the Long Term Implications of
feature testing and your Choices (i.e. don’t bite off more
later re-use them for than you can chew)
integration testing or Every choice in building an automation
performance testing. platform has consequences. Some
Will other groups re- consequences seem fine in the short term but
purpose your scripts? can be project killers in the long term. Be
4. Tool Selection (Open source aware that you have scarce resources: time,
vs. Commercial vs. In- computing, money, and people. Some issues
House). Regardless of which to be aware of are as follows:
direction you go (i.e. Open a. Focus on the low hanging fruit
Source, Commercial, Home early on. Focusing on the low
Grown) be sure you realize hanging fruit allows the
the risks/rewards you will automation architect to
have whichever path you “prototype” the new system and
choose. Ask yourself if the easily make changes to his/her
tool fits your specific approach. This also allows the
requirements right now and automation team to firmly
in the future. Some other establish and enforce the correct
points for consideration: processes for running an
a. Is the tool well effective automation program.
supported? b. Some scenarios are too much
b. What is the average work to automate. For example if
time for a feature a scenario takes 1 hour to run
request or bug fix to manually but 40 hours to
be implemented? automate you don’t want to
c. If using Open Source, automate the scenario if it is run
is the said community only once a year. That means it
active in developing would take 40 years for a return
features and fixing on your labor investment. 1The
defects? Perhaps this average life-span of software is a
is not an issue if you mere 5 years.
have people on staff c. Some scenarios introduce a much
that can maintain the greater level of complexity to the
software. automation system. This
d. If the software is complexity can affect the
home grown do you
have developers in- 1
Atkinson, A. A., Kaplan, R. S. and S. M. Young.
house to support the Management Accounting. International 4th Ed. New
Jersey: Pearson Education International, 2004.
 stability and the
scalability of the
automation system as a
whole. There should be a
very strong business case
to justify the risk in
automating in these
situations.
d. Don’t over-engineer your
automation test cases.
Just because it seems
reasonable to perform
certain “extra” steps
when running test cases
manually does not mean
implementing the same
checks in automation
scripts is a good thing.
Adding extra
functionality in scripts
unless absolutely needed
adds a greater chance of
test cases failing due to
false-positives and
obfuscates the real intent
of the test cases. If you
really need to check
something not
specifically listed in a test
case then create a new
test case for that specific
scenario.
Full-Time Personnel must be
qualified and fully committed to
build and maintain an effective
automation program
As mentioned earlier an effective
automation platform demands a
disciplined focus. Some issues to
consider when staffing up for your
automation endeavor should include
the following:
a. Carefully vetting your
team from a list of job
requirements including but not
limited to past work experience,
industry certifications, and
personal recommendations
related to the job position. When
staffing up for an automation
project create a list of job
requirements that will be
required or desirable for the new
team members. Unfortunately
many automation projects are
spear-headed by co-ops, interns,
or unqualified/unmotivated staff
with little or no experience with
software automation. Rate the
complexity of your automation
on a scale from 1 to 10. The
closer you are to 10 requires very
technically skilled automation
engineers. Be sure your team
reflects the appropriate level of
talent.
b. If automation staff focus is
divided between multiple
complicated projects this will
reduce the chance of success for
your automation program.
2
Recent research on Human
Information Processing is
starting to prove what common
sense tells us. The more tasks
that divide your time the more a
penalty you pay in lost time
switching between tasks. In fact
the more complicated the tasks
you are juggling between the
higher the penalty the people pay
in lost time getting back on task.
c. Staff for the project is transient.
Staff for the automation project
is cycled through without regard
2
“Executive Control of Cognitive Processes in Task
Switching” American Psychological Association.
Journal of Experimental Psychology: Human
Perception and Performance. Published December
2001.
 to the deleterious effects
of the automation system.
Automation’s ROI
increases with careful
planning, disciplined
execution and meticulous
maintenance procedures.
This is very challenging
to say the least and
becomes next to
impossible without a
dedicated full time staff.
d. Don’t bother automating
more test scenarios if you
don’t have the staff to
maintain the increased
quantity. Executing tests
includes running the test
scenarios, debugging
failed scenarios,
maintaining the
automation
platform/scripts, and
devices under test. It is
not feasible for a part-
time automation engineer
to run 10,000 test
scenarios and debug 1000
failures in a reasonable
amount of time. If your
staff is tightly constrained
keep the automation
system small but effective
by running the “key”
scenarios only.
The Test Tool Product you Buy is
a “Product” not “Your Automation
Solution”
Don’t expect to purchase an
automation system in a box. You
will be required to do much of the
work yourself when creating an
automation platform.
Purchasing/Obtaining a tool is just one of
many steps along the way. Obviously the
quality of the solution depends on the other
issues and items discussed in this White
Paper.
Defining a Successful Automation
System
Create a list of criteria for determining
success/failure and revisit these criteria at
specific intervals over the life-span of the
automation project. Some possible criteria is
listed below:
a. Low rate of False Positives
b. High rate of defects found
c. Reducing or Maintaining staff
growth costs
d. Reducing Manual testing time
Summary
Successful automation involves careful
planning, disciplined execution, meticulous
maintenance procedures, and a dedicated
staff as well as a management with a
willingness and patience to fully support
(with money, time, and labor) an automation
project. The first and most important step in
building an automation system is realizing
the complexities involved and the positive
consequences of careful planning and
execution of your automation system.