You are on page 1of 6

www.martproservice.

com

Why Automation Projects Fail


(and what you can do to prevent failure)

Art Beall, Chief Technical Officer

11/22/08
Contents Introduction
Automation is the new mandate in today’s
Introduction 2 fast paced world of software development.
Problem Statement 2 Companies are hard-pressed to get their
products out as quickly as possible while
MART Pro-Service Solution 3
hampered under Capex and Opex
Summary 3 constraints. Traditionally testing has long
been seen as the bottleneck to getting
software releases out the door and software
automation as the solution to removing that
bottleneck. However, software automation
more often than not fails to live up to
expectations and in fact seldom returns the
investment put forth.

Problem Statement
A majority of automation projects fail. In
fact, most if not all automation projects fail
to live up to peoples’ expectations and many
automation projects are flat out abandoned.
The reason for failure can be explained by
referring to the familiar expression “tip of
the iceberg”. Automation (like the tip of an
iceberg) seems rather like a small problem
superficially however later (often too late) it
is realized that the problem domain is much
bigger (like the true size of an iceberg) than
first realized. Often by this time test groups
either accept severe limitations in the
automation platform or they give up on
automation all together (like a ship colliding
with an iceberg). The impression among
many is that automation is simply a matter
of purchasing the “right” box and assigning
a resource or two to the project. However,
companies soon realize (to their surprise)
that their testing needs are not meet by the
“box” in fact they need significant
(unforeseen) investment to save the project
from failure.

11/22/08
MART Pro-Service Solution h. Versioning of test scripts
This white paper will cover five key i. Documentation for
areas to consider while planning, administration/test script
implementing, and maintaining an creation.
automation platform. Use these key 3. Design and document an automation
areas as a guide to help plan, deploy, architecture. An automation project
and maintain your automation needs a framework and the
project. framework needs to be implemented
and documented. The framework
Automation is akin to any should lay the foundation and rules
Complex Software System that everyone lives under. For
Building Automation solutions is example a shared software library for
akin to building any (complex) automation scripts should be
Software system and requires the implemented and used by all
same effort (usually high effort) to automation script writers. Also, the
implement effectively. You will be high-level format of a test case
integrating a system with many parts should be specified and documented
as well as software (i.e. scripts and with examples. Some other items to
glue software). Therefore the same consider are listed below:
lessons learned in building systems a. Automation Architecture
and software should be applied to must be extensible.
your automation project. Some items Automation requirements
to consider are: will grow in step with the
1. What are the high level goals requirements of the software
of the automation system: under test and the automation
performance testing, unit system must be able to cope
testing, feature testing? with that growth.
2. Create a list of requirements. b. Must have instrumentation
What are the features you for debugging purposes.
need in the automation Running scripts is the easy
system? Some examples are part. Debugging scripts is
listed below. what takes the most time. Be
a. Logging capabilities sure you have a system in
b. Graphical automation place that quickly
scenario creator summarizes all the problems
c. API for a remote from a test execution run.
procedure calls c. Your scripts (and the
d. Reporting underlying libraries used by
e. Resource your scripts) should be
Management of modularized.
test/lab resources d. Keeping statistics on your
f. Concurrent scripts executions. Identify test cases
g. Capability to with a high false positive rate
preparing or “clean and either fix the root cause
up” the test or remove from your
environment executions. Conversely,
identify tests that find product? If so how likely is it
real software defects. these developers or
e. Can the automation development group will be
scripts be re-used available three to five years
later for different from now?
purposes. For
example you might
develop scripts for Consider the Long Term Implications of
feature testing and your Choices (i.e. don’t bite off more
later re-use them for than you can chew)
integration testing or Every choice in building an automation
performance testing. platform has consequences. Some
Will other groups re- consequences seem fine in the short term but
purpose your scripts? can be project killers in the long term. Be
4. Tool Selection (Open source aware that you have scarce resources: time,
vs. Commercial vs. In- computing, money, and people. Some issues
House). Regardless of which to be aware of are as follows:
direction you go (i.e. Open a. Focus on the low hanging fruit
Source, Commercial, Home early on. Focusing on the low
Grown) be sure you realize hanging fruit allows the
the risks/rewards you will automation architect to
have whichever path you “prototype” the new system and
choose. Ask yourself if the easily make changes to his/her
tool fits your specific approach. This also allows the
requirements right now and automation team to firmly
in the future. Some other establish and enforce the correct
points for consideration: processes for running an
a. Is the tool well effective automation program.
supported? b. Some scenarios are too much
b. What is the average work to automate. For example if
time for a feature a scenario takes 1 hour to run
request or bug fix to manually but 40 hours to
be implemented? automate you don’t want to
c. If using Open Source, automate the scenario if it is run
is the said community only once a year. That means it
active in developing would take 40 years for a return
features and fixing on your labor investment. 1The
defects? Perhaps this average life-span of software is a
is not an issue if you mere 5 years.
have people on staff c. Some scenarios introduce a much
that can maintain the greater level of complexity to the
software. automation system. This
d. If the software is complexity can affect the
home grown do you
have developers in- 1
Atkinson, A. A., Kaplan, R. S. and S. M. Young.
house to support the Management Accounting. International 4th Ed. New
Jersey: Pearson Education International, 2004.
stability and the requirements including but not
scalability of the limited to past work experience,
automation system as a industry certifications, and
whole. There should be a personal recommendations
very strong business case related to the job position. When
to justify the risk in staffing up for an automation
automating in these project create a list of job
situations. requirements that will be
d. Don’t over-engineer your required or desirable for the new
automation test cases. team members. Unfortunately
Just because it seems many automation projects are
reasonable to perform spear-headed by co-ops, interns,
certain “extra” steps or unqualified/unmotivated staff
when running test cases with little or no experience with
manually does not mean software automation. Rate the
implementing the same complexity of your automation
checks in automation on a scale from 1 to 10. The
scripts is a good thing. closer you are to 10 requires very
Adding extra technically skilled automation
functionality in scripts engineers. Be sure your team
unless absolutely needed reflects the appropriate level of
adds a greater chance of talent.
test cases failing due to b. If automation staff focus is
false-positives and divided between multiple
obfuscates the real intent complicated projects this will
of the test cases. If you reduce the chance of success for
really need to check your automation program.
2
something not Recent research on Human
specifically listed in a test Information Processing is
case then create a new starting to prove what common
test case for that specific sense tells us. The more tasks
scenario. that divide your time the more a
penalty you pay in lost time
switching between tasks. In fact
Full-Time Personnel must be the more complicated the tasks
qualified and fully committed to you are juggling between the
build and maintain an effective higher the penalty the people pay
automation program in lost time getting back on task.
As mentioned earlier an effective c. Staff for the project is transient.
automation platform demands a Staff for the automation project
disciplined focus. Some issues to is cycled through without regard
consider when staffing up for your
automation endeavor should include 2
“Executive Control of Cognitive Processes in Task
the following: Switching” American Psychological Association.
a. Carefully vetting your Journal of Experimental Psychology: Human
team from a list of job Perception and Performance. Published December
2001.
to the deleterious effects Purchasing/Obtaining a tool is just one of
of the automation system. many steps along the way. Obviously the
Automation’s ROI quality of the solution depends on the other
increases with careful issues and items discussed in this White
planning, disciplined Paper.
execution and meticulous
maintenance procedures.
This is very challenging Defining a Successful Automation
to say the least and System
becomes next to Create a list of criteria for determining
impossible without a success/failure and revisit these criteria at
dedicated full time staff. specific intervals over the life-span of the
d. Don’t bother automating automation project. Some possible criteria is
more test scenarios if you listed below:
don’t have the staff to a. Low rate of False Positives
maintain the increased b. High rate of defects found
quantity. Executing tests c. Reducing or Maintaining staff
includes running the test growth costs
scenarios, debugging d. Reducing Manual testing time
failed scenarios,
maintaining the
automation
platform/scripts, and
devices under test. It is Summary
not feasible for a part- Successful automation involves careful
time automation engineer planning, disciplined execution, meticulous
to run 10,000 test maintenance procedures, and a dedicated
scenarios and debug 1000 staff as well as a management with a
failures in a reasonable willingness and patience to fully support
amount of time. If your (with money, time, and labor) an automation
staff is tightly constrained project. The first and most important step in
keep the automation building an automation system is realizing
system small but effective the complexities involved and the positive
by running the “key” consequences of careful planning and
scenarios only. execution of your automation system.

The Test Tool Product you Buy is


a “Product” not “Your Automation
Solution”
Don’t expect to purchase an
automation system in a box. You
will be required to do much of the
work yourself when creating an
automation platform.