Infosys Technologies Limited

IVS L1-T101

IVS L1 CERTIFICATION AUTOMATION TESTING CONCEPTS (Version 1.2)

Infosys Technologies Limited

IVS L1-T101

Revision History:
Date 01-Sep-05 28-Oct-05 18-Nov-05 Author(s) Pinaki Sen Sharma Aruna Shankar Gomathi Ramasubramanian / Amber Kumar Reviewer(s) Shishank Gupta Ramesh Pusala Version 1.0 1.1 1.2 Description Initial Version Modified the TOC Embedded the objects to point to the documents referenced by some reference hyperlinks

Infosys Technologies Limited Index:

IVS L1-T101

1 Introduction.......................................................................................................................4 2 Why Test Automation.......................................................................................................4 3 Automation Life Cycle.....................................................................................................5

3.1 3.2 3.3 3.4 3.5 3.6

Automation Feasibility Analysis.........................5 Planning and Design..........................................6 Test Environment Setup....................................7 Automation Script Generation...........................7 Test Execution...................................................8 Defect Analysis & Fixing....................................8

4 Cost Involved in automation.............................................................................................9 5 Myths & Realities...........................................................................................................10

6 Feasible Strategies.............................................10 6.1 Functional Decomposition Method..................10 6.2 Test Plan Driven Method.................................11

7 Resource Concerns..........................................................................................................13 8 Things to remember........................................................................................................14 9 References.......................................................................................................................15

Infosys Technologies Limited

IVS L1-T101

1 Introduction
After a certain point of time, the repetitive manual testing of test scripts becomes very monotonous and mundane. Also, for certain types of testing like regression testing, it always helps if somebody or something could go on testing the same thing again and again. With the current trend of software release cycles becoming shorter and the list of tests in the regression test suite becoming larger, it becomes improbable to do complete manual regression testing with every release. Incomplete and inefficient testing may cause new bugs (defects introduced while fixing a bug) going undetected before the release. By automating the process (regression testing in this case), the time needed to test run the full suite reduces, thereby providing the tester more time to focus on manual testing of new code and on areas of more risk. The concept of automation finds its use in all such cases. In short, its mainly about Using software tools to execute tests without manual intervention Use of other software tools to control the execution of tests, the comparison of actual with the expected outcome and other test control & test reporting functions.

Automation Testing as such refers to automating the manual testing process. Automation can be applied to - Web applications - Client -server environments. The process basically consists of test cases that contain expected results based on business requirements and also an exclusive test environment having a test database in which the test cases can be repeated whenever the application undergoes a change.

2 Why Test Automation

The following points elucidate the reasons behind the need for automation (selective and not exhaustive): Running the existing tests on a new version(s) of a program/application More number of tests can be run in a lesser period of time Bugs that can be uncovered only with long runs Better use of resources 4

Infosys Technologies Limited Testers can do better jobs, machines can be run all the time Consistency, Reliability and Repeatability of tests

IVS L1-T101

The same tests can be repeated exactly in the same manner, every time when the automated scripts are executed thus eliminating manual testing errors

3 Automation Life Cycle

Stages involved in automation process are (in order) 1. 2. 3. 4. 5. 6. Automation Feasibility Analysis Planning and Design Test Environment Setup Automation Script Generation Test Execution Defect Analysis & Fixing

Each stage has specific Entry Criteria, Activities, Measurements and Exit Criteria defined for them. The details are as follows: 3.1 Automation Feasibility Analysis Overview Requirements & expectations have to be defined clearly & explicitly before starting off any project and so is the case involving automating a project. Things like How and what is to be automated Which all modules can be automated should be decided in the beginning itself to prevent any confusion after the automation process has been flagged off. This stage hence requires a great deal of attention. Entry-criteria Clearly defined Requirements Sound understanding of client expectations

Activities Requirement analysis for a thorough understanding Identify specific requirement tests that can be automated Identify test cases that cannot be automated due to tool limitations and / or complexity of the functionality 5

Infosys Technologies Limited IVS L1-T101 Understanding the System and the environment in which the application has to be installed Analyse applicability of a suitable tool to carry out the automation Rough estimate of the size, effort and cost involved in this automation Analyze if automation is actually possible and benefit in the long run Prepare a feasibility analysis report Validation Review of Feasibility Analysis Report Exit-criteria Approved Feasibility Analysis report 3.2 Planning and Design Overview Once the requirements & expectations have been highlighted clearly and the feasibility studies yield a positive result, methodical planning has to be done. This would include defining the approach, decide on which framework is to be used and subsequently, estimating the time and finance needed to complete the same. Entry-criteria Automation Feasibility analysis report Activities Identify applicable approaches Analyse the suitability of each approach and select the best among them Identify the frameworks that can be used Analyze every framework for their pros and cons Finalizing the framework to be used Form a test plan based on the above. Analyse manual test scripts from an automation standpoint. Define automation scripting standards (Common language functionality) Identify common functionality across test cases and design compiled modules Categorize test scripts based on functionality Identify functionality which need to be parameterised Prepare the detailed cost, size and effort estimate

Validation Review of Test Plan by the client 6

Infosys Technologies Limited Exit-criteria Approved automation Framework Approved automation test plan 3.3 Test Environment Setup Overview

IVS L1-T101

Environment set-up is one of the critical aspects of testing process. Test team may not be involved in this activity if the customer/development team provides the test environment in which case the test team is required to do a readiness check of the given environment. Test environment decides the software and hardware conditions under which a work product is tested. Entry-criteria System Design and architecture documents are available Environment set-up plan is available Activities Understand the required architecture, environment set-up Prepare environment setup checklist Prepare hardware and software requirements Finalize connectivity requirements Setup Test Environment

Validation Effort for setup Defects in setup Exit-criteria Environment setup is working as per the plan and checklist

3.4 Automation Script Generation

Overview The project team will go through the test plan & obtains clarifications, if needed. It will then decide upon the type of automation to be used, it could be capture/playback, scripting etc. The project team will share the knowledge and experience through presentations and/or discussions at appropriate intervals during the progress of the project. Entry-criteria Test plan is available 7

Infosys Technologies Limited Test tool available Manual test case if any Activities Create test scenarios Identify reusable test scenarios Group the common test cases together Finalize Test Scripts

IVS L1-T101

Validation Review of scripts Exit-criteria Scripts are tested and reviewed Defects are fixed 3.5 Test Execution Overview The project team member(s) will carry out testing based on the test requirements. Entry-criteria Baselined Test plan is available Test environment is ready Test data set up is done Test Scripts are available Activities Perform the testing as per the test plan Perform the batch test wherever necessary Update the status in the status matrix Log the defects

Validation Testing Exit-criteria All test cases are executed. Defects logged in the defect tracking tool 3.6 Defect Analysis & Fixing 8

Infosys Technologies Limited IVS L1-T101 Overview The project team members will analyze the defects logged and will fix the defects. The project team members will also prepare a document on the defects found, their priority and also the solution for the same. Entry-criteria Testing has been done Test records are available Defect log is available Activities Analyze the defects for severity and priority Fix the defects Document the defect along with the solution Validation Testing Exit-criteria Defects are analysed and fixed Application is ready to be retested for bug fixes

4 Cost Involved in automation

Fixed Cost Automation feasibility analysis cost Tool selection and Acquisition cost Hiring skilled resources OR training existing team members Time in learning the application/business processes Pilot project identification effort and Proof of Concept First time automation of the identified parts of application/product Test suite Documentation effort

Variable cost Test script and documentation maintenance cost Automated Test infrastructure maintenance cost Execution cost

Infosys Technologies Limited

IVS L1-T101

5 Myths & Realities

Automated testing is just an add-on to the testing process to make the testers life easier. Unlike prevalent myths, it doesnt necessitate ramp-down of resources in the testing department. Also, automation is not as inexpensive as we think. As a matter of fact, it may take much longer to develop, verify, and document an automated test case than to create and execute a manual test case! This is even truer if the record/playback feature is selected as the primary testing technique. In fact, record/playback is the least costeffective method of automating test cases. Having said that, it doesnt mean that automation testing cant be made cost effective at all. Depending on the type of testing requirements of the company, the tool could be decided upon. If there be a case wherein the same functionality is achieved by two different tools wherein one is significantly costlier than the other, the less expensive one could be easily preferred over the other. Again, the tool chosen should be serving the interests of the company well. Otherwise, even a small amount of money spent on a less expensive tool wont do any good. Simply put, its the requirements that come first while choosing a tool and then all other monetary and other considerations. Apart from the monetary aspect, there are other issues relating to time management as well. Trying to automate very complex test cases wouldnt serve much purpose as they would consume a lot of time and at the end of the day, no one can be sure if the automation would actually be successful time and again. So rather than breaking heads over some very complex test cases, the tester would do well to automate a significant number of relatively easier test cases in the mean time. This method is the most costly (in terms of time consumption) among all methods. The record/playback feature of a test tool is useful for determining how the tool interacts with the application under test, and can give ideas regarding developing test scripts, but its usefulness ends there. It cant handle any unexpected pop-up that might suddenly appear or any such unexpected behavior for that matter that was not a part of the recording.

6 Feasible Strategies
Other than the record/playback strategy (which has more disadvantages than advantages), there are a few more methodologies which are very effective for automating functional or system testing. A few of such strategies are: 6.1 Functional Decomposition Method Essentially, it involves reducing all test cases to their most fundamental tasks, and write user-defined functions, business function scripts, and sub-routine or utility scripts which perform the tasks independently of one another. In general, these fundamental areas include navigation, specific business function, and data verification & return navigation. For achieving this, it is mandatory to separate data & function. This allows an automated test script to be written for a business function, using data-files to provide both the input 10

Infosys Technologies Limited IVS L1-T101 and the expected-results verification. The hierarchical architecture employed, has a structured or modular design. The engine of the test is the driver script which is at the highest level. This driver script has a number of calls to one or more test scripts. These test scripts contain the actual logic, calling the business function scripts necessary to perform the application testing. Functions & utility scripts are called as needed by Drivers, Main, and Business Function scripts. While the driver scripts do the initialization and call the scripts in the correct order, the test case scripts perform the application test case logic using business function scripts which in turn perform specific business functions within the application. The subroutine scripts perform specific tasks required by two or more business scripts while user-defined functions are general, application-specific and screen-access functions. The business function and subroutine scripts invoke user defined functions to perform navigation. The Test Case script would call these two scripts, and the Driver Script would call this Test Case script some number of times required to perform all the required Test Cases of this kind. In each case, the only thing that changes are the data contained in the files that are read and processed by the business function & subroutine scripts. This method, however, requires only that we add the data-files required for each test, and these can easily be updated/maintained using Notepad or some such text-editor. Updating these files does not require any knowledge of the automated tool, scripting, programming, etc. meaning that the non-technical testers can perform this function, while one technical tester can create and maintain the automated scripts. Advantages: The best part is that scripts can be created while application development is still in progress. If the functionality were to change, only the specific business function script would have to be modified.. Being text records, the data involved (both input and output) & the expected results are easily maintainable. Decreases duplicity in creating automated test scripts as it uses a modular design & uses files to input and verify data. Better error handling as the functions return values to the calling script, rather than aborting, thereby increasing the robustness of the test scripts. Disadvantages: Prior knowledge needed in the scripting language used by the tool Multiple data-files are needed for every test case. Maintenance of the Detail Test Plan with specific data becomes increasingly important. 6.2 Test Plan Driven Method 11

Infosys Technologies Limited IVS L1-T101 It is also called the key-word driven approach. This method uses the actual test case document developed by the tester using a spreadsheet containing special "Key-Words". This method preserves most of the advantages of the "Functional Decomposition" method, while eliminating most of the disadvantages. In this method, the entire process is data-driven, including functionality. The Key Words control the processing here. Architecture The architecture of the Test Plan Driven method appears similar to that of the Functional Decomposition method, but in fact, they are substantially different. The driver script performs the initialization if needed (just as in functional decomposition method) and calls the application specific controller script, which in turn processes the file name received from driver and matches on key words contained in the input file, builds a parameter-list from the records & calls utility scripts associated with the key words. The utility scripts in turn processes input parameter-list received from the controller script, perform specific tasks, raise errors that might have occurred and then return to the controller script. User defined functions, both general and application-specific ones can be called by any of the scripts from above. Advantages Apart from the advantages that the functional decomposition method has, it has a few more plus points too and they are: Ease of working as the detail test plan need be written only once in the spreadsheet format containing all input and verification data. The best part is that even if the tester doesnt have prior knowledge on the tool, the work can be started! I.e. before the plan is written, if the utility scripts are created by someone proficient in the tools scripting language, then the tester can use the test tool immediately via the spreadsheet-input method, without having to learn the scripting language. All he needs to know are the specific key-words and the format to use within the test plan. This saves time as there is no dire need of spending time on training the tester about the tool first and then starting off with coding. Re-usability is another desirable thing in this approach. After a number of generic Utility scripts have already been created for testing an application, most of these can be re-used to test another application. Disadvantages

Prior knowledge of the tool specific scripting language is needed for creating customized functions and utilities. So if someone technically sound is present to do it, then fine otherwise that would become a bottleneck as in the 12

Infosys Technologies Limited IVS L1-T101 advantage of the tester not having to know anything about the tool would no more hold good. In the event of application requiring more than a few customized utilities, the tester is required to learn a number of key words and special formats which can be pretty time-consuming. Once the testers get used to this, however, the time required to produce a test case is greatly improved. Automation applied to different types of testing Test Stage Regression Testing Type of testing Black Box Amenity to Automation High Factors that influence Model Office Testing Black Box High System Testing Black Box Medium Focus is to test the impact on the stability of the original application Tests are repetitive (Same functionality need to be tested for every release) Application will be highly stable Final level of acceptance tests and therefore less number of outstanding defects Application will be fairly stable Tests are repetitive in nature (Same tests need to repeated for various parameters) Application will be fairly stable Depends on the # of outstanding defects from System Testing Number of test cycles planned (At least 2 cycles should be planned) No repeatability / No stability in terms of functionality High cost for automation System level integration testing White box and not a functional testing No repeatability and no stability in terms of both technical and functional aspects Component level white box testing No repeatability and No Stability as system is still in construction phase

Performance Testing

Black Box

High

User Acceptance Black Box Testing

Medium to High

Integration Testing

Grey Box

Low

Unit Testing

White Box

Low

7 Resource Concerns
As discussed earlier, the myth that automating a test process would make a company to cut down on its staff size is not even distantly true. All that these tools do is accelerate the testing process. These test tools use scripts which automatically execute test cases. These 13

Infosys Technologies Limited IVS L1-T101 test scripts are nothing but programs. They are written in the tool specific scripting language. Since these are programs, their management has to be done in as much the same way as any other code. To make this possible, an expert in that particular tool must be hired or somewhat similar role must be created and armed with at least one seniorlevel programmer. This expert or whoever that person is must be able to design, develop, test and debug the code apart from documenting the same. Also, apart from all this work related to creating scripts and functions, this person must be responsible for developing standards and procedures for script development, creating change-management procedures for script/function implementation as also testing, implementing, and managing the test scripts as well.

8 Things to remember
Sufficient preparations must be made before starting the testing automation process. More often than not, this is not the case! Following are the things that must be done to prepare for the effort. An exact replica of the actual production environment must exist in which the same type of hardware and programs are installed. These automation scripts must have all exclusive computers (dedicated ones) to be run on. This is because as such they would be consuming a large amount of memory and would thus be slow. So if any extra process were to run on the same computer that would slow down the rate of script execution even more resulting in more time delay. It must be possible to restore the test environment's database to a known baseline; otherwise tests performed against the same cant be repeated, as the data would have changed Initially the company would need some professional trainers to train the testers on the training tool to be employed, but going forward, it is up to the testers themselves as to how much value they could add to the work by honing their skills in the shortest possible time. As they say, the sooner the better!

That was with regards to the preparation required before actually starting off. But there are other things as well that matter. Decisions like which tool to use and how to implement it are made by the management, often without consulting the people who are actually going to test. This sometimes meets with a certain amount of resistance from the testers. The thought that the tool can replace the testers is devoid of any bit of truth. All it does is helps the tester in doing the job better and faster. Initially the testers might face a few difficulties in learning a new thing from the scratch altogether (in this case, how the tool works etc.), but then practice makes a man perfect! To start off, professional trainers can be hired to overcome the initial hiccups but with time, the people who are actually going to work on the tool must pick up the finer details from the trainers and move on. If the test plan driven method is used, the testers will not have to learn how to use the tool at all if not needed. All that they have to learn is a different method of documenting the 14

Infosys Technologies Limited IVS L1-T101 detailed test cases, using the key-word/spreadsheet format. And frankly, techies that they are, it wont take them long to learn such things. All said and done, there still are many things that need to be taken care of apart from the ones discussed above. For e.g. things like having defined and reasonable goals as to what can and what cannot be achieved And of course, before starting off, educating oneself about the actual automation process wont be a bad idea at all! Getting a clear understanding of the requirements would always help. It always helps to identify the tests that are good candidates for automation. Too complex tests are better left for manual testing. Cost effectiveness is also of as much concern as the functional effectiveness of the tool. It is thus imperative to select a tool that implements the process in a way that ensures smooth long-term testing. Knowing all these things now, automation should no longer remain an unheard of concept any more.

9 References
http://www.sqa-test.com/w_paper1.html http://www.testing.com/writings/automate.pdf

Autom ation Testing

IVS-MAR Automation Repository in the IVS Portal PRIDE @ Infosys

15