You are on page 1of 6



Why Automation Projects Fail
(and what you can do to prevent failure)

Art Beall, Chief Technical Officer


companies soon realize (to their surprise) that their testing needs are not meet by the “box” in fact they need significant (unforeseen) investment to save the project from failure. However.Contents Introduction Problem Statement MART Pro-Service Solution Summary 2 2 3 3 Introduction Automation is the new mandate in today’s fast paced world of software development. In fact. Companies are hard-pressed to get their products out as quickly as possible while hampered under Capex and Opex constraints. Automation (like the tip of an iceberg) seems rather like a small problem superficially however later (often too late) it is realized that the problem domain is much bigger (like the true size of an iceberg) than first realized. Problem Statement A majority of automation projects fail. The impression among many is that automation is simply a matter of purchasing the “right” box and assigning a resource or two to the project. most if not all automation projects fail to live up to peoples’ expectations and many automation projects are flat out abandoned. software automation more often than not fails to live up to expectations and in fact seldom returns the investment put forth. 11/22/08 . The reason for failure can be explained by referring to the familiar expression “tip of the iceberg”. However. Often by this time test groups either accept severe limitations in the automation platform or they give up on automation all together (like a ship colliding with an iceberg). Traditionally testing has long been seen as the bottleneck to getting software releases out the door and software automation as the solution to removing that bottleneck.

Running scripts is the easy part. Capability to preparing or “clean up” the test environment h. Graphical automation scenario creator c.e. An automation project needs a framework and the framework needs to be implemented and documented. For example a shared software library for automation scripts should be implemented and used by all automation script writers. Some other items to consider are listed below: a. API for a remote procedure calls d. Automation requirements will grow in step with the requirements of the software under test and the automation system must be able to cope with that growth. Automation Architecture must be extensible. a. You will be integrating a system with many parts as well as software (i. unit testing. What are the high level goals of the automation system: performance testing. Some items to consider are: 1. b. Use these key areas as a guide to help plan. . Identify test cases with a high false positive rate and either fix the root cause or remove from your executions. Debugging scripts is what takes the most time. Keeping statistics on your executions. feature testing? 2. deploy. Logging capabilities b. Your scripts (and the underlying libraries used by your scripts) should be modularized. d. and maintaining an automation platform. the high-level format of a test case should be specified and documented with examples. Resource Management of test/lab resources f.MART Pro-Service Solution This white paper will cover five key areas to consider while planning. Also. c. Documentation for administration/test script creation. Create a list of requirements. Concurrent scripts g. Reporting e. Must have instrumentation for debugging purposes. scripts and glue software). implementing. The framework should lay the foundation and rules that everyone lives under. Conversely. and maintain your automation project. Automation is akin to any Complex Software System Building Automation solutions is akin to building any (complex) Software system and requires the same effort (usually high effort) to implement effectively. Therefore the same lessons learned in building systems and software should be applied to your automation project. Design and document an automation architecture. Be sure you have a system in place that quickly summarizes all the problems from a test execution run. Versioning of test scripts i. What are the features you need in the automation system? Some examples are listed below. 3.

This also allows the automation team to firmly establish and enforce the correct processes for running an effective automation program. Focusing on the low hanging fruit allows the automation architect to “prototype” the new system and easily make changes to his/her approach. Some other points for consideration: a. Some scenarios are too much work to automate. InHouse). Kaplan. For example you might develop scripts for feature testing and later re-use them for integration testing or performance testing.identify tests that find real software defects. A. Management Accounting. 2004. b. Will other groups repurpose your scripts? 4. Some scenarios introduce a much greater level of complexity to the automation system. 1The average life-span of software is a mere 5 years. Commercial vs. Be aware that you have scarce resources: time. New Jersey: Pearson Education International. money.e. A. Can the automation scripts be re-used later for different purposes. Focus on the low hanging fruit early on. Some issues to be aware of are as follows: a. S. Young. Is the tool well supported? b. Open Source. For example if a scenario takes 1 hour to run manually but 40 hours to automate you don’t want to automate the scenario if it is run only once a year. e. Commercial. Home Grown) be sure you realize the risks/rewards you will have whichever path you choose. That means it would take 40 years for a return on your labor investment. don’t bite off more than you can chew) Every choice in building an automation platform has consequences.. What is the average time for a feature request or bug fix to be implemented? c. and people. This complexity can affect the 1 Atkinson. Ask yourself if the tool fits your specific requirements right now and in the future. . d. Some consequences seem fine in the short term but can be project killers in the long term.e. R. International 4th Ed. M. Regardless of which direction you go (i. is the said community active in developing features and fixing defects? Perhaps this is not an issue if you have people on staff that can maintain the software. If using Open Source. and S. If the software is home grown do you have developers inhouse to support the product? If so how likely is it these developers or development group will be available three to five years from now? Consider the Long Term Implications of your Choices (i. c. Tool Selection (Open source vs. computing.

. Full-Time Personnel must be qualified and fully committed to build and maintain an effective automation program As mentioned earlier an effective automation platform demands a disciplined focus. 2 Recent research on Human Information Processing is starting to prove what common sense tells us. Be sure your team reflects the appropriate level of talent.stability and the scalability of the automation system as a whole. Rate the complexity of your automation on a scale from 1 to 10. The more tasks that divide your time the more a penalty you pay in lost time switching between tasks. If you really need to check something not specifically listed in a test case then create a new test case for that specific scenario. or unqualified/unmotivated staff with little or no experience with software automation. c. Just because it seems reasonable to perform certain “extra” steps when running test cases manually does not mean implementing the same checks in automation scripts is a good thing. interns. Published December 2001. and personal recommendations related to the job position. Adding extra functionality in scripts unless absolutely needed adds a greater chance of test cases failing due to false-positives and obfuscates the real intent of the test cases. Unfortunately many automation projects are spear-headed by co-ops. When staffing up for an automation project create a list of job requirements that will be required or desirable for the new team members. Staff for the automation project is cycled through without regard 2 “Executive Control of Cognitive Processes in Task Switching” American Psychological Association. Staff for the project is transient. If automation staff focus is divided between multiple complicated projects this will reduce the chance of success for your automation program. Some issues to consider when staffing up for your automation endeavor should include the following: a. The closer you are to 10 requires very technically skilled automation engineers. d. Carefully vetting your team from a list of job requirements including but not limited to past work experience. Don’t over-engineer your automation test cases. There should be a very strong business case to justify the risk in automating in these situations. b. In fact the more complicated the tasks you are juggling between the higher the penalty the people pay in lost time getting back on task. industry certifications. Journal of Experimental Psychology: Human Perception and Performance.

Executing tests includes running the test scenarios. High rate of defects found c.000 test scenarios and debug 1000 failures in a reasonable amount of time. maintaining the automation platform/scripts. The Test Tool Product you Buy is a “Product” not “Your Automation Solution” Don’t expect to purchase an automation system in a box. Automation’s ROI increases with careful planning. and labor) an automation project. and devices under test. time. If your staff is tightly constrained keep the automation system small but effective by running the “key” scenarios only. Low rate of False Positives the deleterious effects of the automation system. Obviously the quality of the solution depends on the other issues and items discussed in this White Paper. disciplined execution. Reducing Manual testing time Summary Successful automation involves careful planning. Don’t bother automating more test scenarios if you don’t have the staff to maintain the increased quantity. You will be required to do much of the work yourself when creating an automation platform. This is very challenging to say the least and becomes next to impossible without a dedicated full time staff. d. . Purchasing/Obtaining a tool is just one of many steps along the way. disciplined execution and meticulous maintenance procedures. The first and most important step in building an automation system is realizing the complexities involved and the positive consequences of careful planning and execution of your automation system. debugging failed scenarios. and a dedicated staff as well as a management with a willingness and patience to fully support (with money. Defining a Successful Automation System Create a list of criteria for determining success/failure and revisit these criteria at specific intervals over the life-span of the automation project. Some possible criteria is listed below: a. Reducing or Maintaining staff growth costs d. It is not feasible for a parttime automation engineer to run 10. meticulous maintenance procedures.