Beruflich Dokumente
Kultur Dokumente
Version: ST/Handout/0907/1.0
Date: 28-09-07
Cognizant
500 Glen Pointe Center West
Teaneck, NJ 07666
Ph: 201-801-0233
www.cognizant.com
Automation
TABLE OF CONTENTS
Introduction ................................................................................................................................3
About this Module ......................................................................................................................3
Target Audience.........................................................................................................................3
Module Objectives .....................................................................................................................3
Pre-requisite ..............................................................................................................................3
References .................................................................................................................................24
Websites ..................................................................................................................................24
Page 2
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
Introduction
Target Audience
Entry Level Trainees.
Module Objectives
After completing this module, the student will be able to:
Explain automation.
Pre-requisite
Should know fundamentals of Testing
Page 3
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
Automation- An Introduction
Objective:
After completing this chapter, the student will be able to:
Explain automation
What is Automation?
Automated testing is automating the manual testing process currently in use.
Test Automation is a 'development' activity, not a 'test' activity.
Test automation involves automating a manual process already in place that uses a
formalized testing process.
Software testing assisted with software tools that require no operator input, analysis or
evaluation
What to automate?
Tests that will be run many times
Tests that will be run with different sets of data
When to Automate
By using automated techniques, the tester has a very high degree of control over
which types of tests are being performed, and how the tests will be executed.
We can Automate when there are multiple releases of the application.
We can automate only When the application under manual test is stable.
We can automate on tests that need to be run for every build of the application.
We can automate on tests that require detailed information from application internals
We can automate on tests that require execution of multiple data called
parameterization.
When the project have enough time for regression testing
Page 4
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
In the past, most software tests were performed using manual methods. This required a large staff
of test personnel to perform expensive, and time-consuming manual test procedures.
Owing to the size and complexity of today’s advanced software applications, manual testing is no
longer a viable option for most testing situations.
Every organization has unique reasons for automating software quality activities, but several
reasons are common across industries.
Using Testing Effectively
Reducing Testing Costs
Replicating Testing Across Different Platforms
Repeatability and Control
High speed cycle; code-test-analyze-fix-test…
Better testing if designed right
Good coverage for Data-oriented and/or repetitive verification tasks
Reduces manpower and time
More coverage within the same testing time.
Testing products which are very difficult to test manually. Eg Load Tests
Can look inside the software for memory leaks, redundant code etc.
Improves efficiency of testing
Reducing Testing Costs
Replicating Testing Across Different Platforms
Repeatability and Control
Greater Application Coverage
Page 5
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
Automated
2. Test tool
Acquisition
Testing Life-Cycle
Methodology (ATLM)
If the testing tool that you select, does not support your environment/application then you are in
trouble and in most cases you will need to revert to manually testing the application. So, Testing
tools depends on consistency of the test environment. The Tool Matrix is provided for quick and
easy reference to the capabilities of the test tools.
The following are examples of criteria that can be used to identify tests that are prime candidates
for automation.
Page 6
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
Examples include:
Creating customer records, invoicing and other high volume activities where software failures
would occur frequently.
Examples include: financial month-end closings, production planning, sales order entry and other
core activities. Any application with a high-degree of risk associated with a failure is a good
candidate for test automation.
Repetitive Testing -
If a testing procedure can be reused many times, it is also a prime candidate for automation. For
example, common outline files can be created to establish a testing session, close a testing
session and apply testing values. These automated modules can be used again and again without
having to rebuild the test scripts. This modular approach saves time and money when compared to
creating a new end-to-end script for each and every test.
There are also several key issues, which should be addressed when selecting an application
testing solution, which are given below:
Page 7
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
A robust tool will allow users to integrate existing test results into an automated test plan. Finally,
an automated test should be able to link business requirements to test results, allowing users to
evaluate application readiness based upon the application's ability to support the business
requirements.
Internet/Intranet Testing
A good tool will have the ability to support testing within the scope of a web browser. The tests
created for testing Internet or intranet-based applications should be portable across browsers, and
should automatically adjust for different load times and performance levels.
Ease of Use
Testing tools should be engineered to be usable by non-programmers and application end-users.
With much of the testing responsibility shifting from the development staff to the departmental
level, a testing tool that requires programming skills is unusable by most organizations. Even if
programmers are responsible for testing, the testing tool itself should have a short learning curve.
Page 8
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
PURPOSE
Identifying an automation tool accurately, effectively and efficiently is a difficult problem for any
large application, with diversified technologies and architecture. The primary purpose of this
document is to effectively conduct the evaluation of automation tool that suits the project needs.
The evaluation and selection is made upon comparing standard automation test tools based on
various criteria taking all major factors into consideration.
The intended audience for this document is business decision-makers who want to select the
automation tool for their application. This document is closely related to the accompanying
Evaluation Methods document (this document describes what we will do, what criteria we will use
and which tools we will use, and the methods document describes how we will do it).
With the advent or growth of technology and the need for Quality Work in IT field, Cognizant have
come out with a standard way of evaluating the tools that help us out in attaining the goal “Quality”.
After the application developed becomes stable and maintainable, need for automation is
proposed. Once the need for automation is recognized, automation tool evaluation is
accomplished. Due to availability of large number of automation tools in the market, it becomes
difficult to identify the appropriate automation tool for the project. Hence tool evaluation is
inevitable for selecting the automation tool that meets the project expectation. It requires prior
experience and process to reduce the complexity of the tool selection.
TOOLS USED
Based on the prior experience, popularity and support provided by vendors tools for comparison
are selected.
Vendors
Mercury
Mercury’s turnover is in excess of $190 million with staff in excess of 1250. In the UK Mercury
have at least 50% of the market in test tools. So as a tester you are more likely to find work as a
skilled Winrunner/QTP developer and also as an employer more likely to find people with
Winrunner/QTP skills.
Page 9
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
They have a number of complimentary tools TestDirector being the most integrated one. They
have a lot of third party support and test tools are usually compared first against Mercury than the
others. Mercury tends to use third party companies to supply professional services support for their
tools (e.g. if you require onsite development of test suites).
IBM’s Rational
Rational’s turnover is in excess of $I/2 billion with staff in excess of 3,300. For me they offer the
most complete lifecycle toolset (including testing) of these vendors for the windows platform.
When it comes to Object Oriented development they are the acknowledged leaders with most of
the leading OO experts working for them. Some of their products are worldwide leaders e.g.
Rational Rose, Clearcase, RequistePro, etc.Their Unified Process is a very good development
model with which it allows mapping of requirements to use cases, test cases and a whole set of
tools to support the process.
Segue
Segue has a turnover of in excess of $50 million and a staff of 400. Anyone who has used SilkTest
along side any of the other test tools will agree that this is the most function rich out the box.
However the learning curve (if you have no programming experience) is the steepest. It provides
the most robust facilities; an object map, test recovery facilities and object-based development
language. Segue’s performance test tool SilkPerformer also performs very well compared to it’s
rivals e.g. LoadRunner, LoadTest, etc.
SilkTest does not have a big user base in the UK compared to the other products which is a
shame because for straight automation testing it has excellent facilities for creating robust test
suites/frameworks.
Compuware
Of all the vendors mentioned in this document Compuware is by far the largest. It has a turnover in
excess of $2 billion and staff of more than 15,000. 9,500 of these are professional services staff
with skills covering all the development lifecycle. As such Compuware does not only supply the
tools but will provide staff to initially develop your test suite and handover to internal staff as
required. Their test tool set is second to Rational only on the windows platform (for coverage) but
for complete coverage across platforms including mainframe and Unix they are the best.
Page 10
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
TECHNIQUE USED
To ensure that the automation tool satisfies all the functional criteria, a specific technique needs to
be developed after comprehensive analysis by the automation team.
The activities and features expected from the tool are listed.
Technology/Environment supported by the tool is identified.
Based on prior experience, the support provided by each tool for a specific feature is
rated.
Level of importance of the individual feature for the project is evaluated.
Critical 6
Must Have 5
Very Important 4
Important 3
Nice to have 2
Not Important 1
No need 0
Based on the level of importance and the tool rating the weightage for each feature is
arrived.
Weightage = Level of importance * Tool rating
Final weightage is calculated by taking sum of weightage of each feature.
Final weightage = ∑ Weightage of individual feature
Based on final weightage the tool that best suits the project is selected.
Page 11
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
Tools Evaluation
Silk
Silk
QA Run - Compuware
Tools QA Run - Compuware
QTP QTP
Winrunner
Winrunner
Rational
Rational
FOCUS AREAS
Automation tool selection requires identification of the major factors and prioritization of the factors.
The automation tool should support most of the factors identified.
The tool evaluation process should address the following issues before selecting the suitable tool.
Compatibility issues: The tool should be compatible with the operating system(s) that
are supported by the application, the technology used for creating the application
(Add-ins should be available to support specific technology like .Net etc.), third party
tools integrated with the application for example Crystal Reports, Infragistics controls
used in application created using .Net technology etc.
Business requirements: Projects may have additional requirements that tool vendors
must meet. For example, the extent of support the tool provides for applications
related to insurance, Banking, Logistics etc.
Budget constraints: How much can be afforded and how much it is worth investing in
the tool should also be considered
Page 12
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
The major benefit of this document is to identify the best suited automation tool which provides the
support, reusability, speed, data accuracy etc. The document will help the business leaders in
decision-making process with regard to significant tool purchase and automating the application.
Functional Tools
Winrunner (TSL)
QTP (VB script)
Rational Robot (SQA)
Silk Test (4th GL)
WinRunner (TSL)
WinRunner records user actions and application responses to create scripts that can be used as
automated tests.
WinRunner offers your organization a powerful tool for enterprise-wide functional and regression
testing. The solution is a component of Mercury Interactive’s Optane™ Suite — a comprehensive
set of tools for testing, deployment, and performance management that provide the only integrated
solution for business technology optimization (BTO).
WinRunner captures, verifies, and replays user interactions automatically to identify defects and
ensure that business processes work flawlessly upon deployment and remain reliable. With
WinRunner, organizations gain several advantages, including:
Reduced testing time by automating repetitive tasks.
Optimized testing efforts by covering diverse environments with a single testing tool.
Maximized return on investment by modifying and reusing test scripts as the
application evolves.
Page 13
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
As WinRunner executes tests, it operates the application automatically, as though a real user were
performing each step in the business process. If test execution takes place after hours or in the
absence of a QA engineer, WinRunner’s Recovery Manger and Exception Handling mechanism
automatically troubleshoot unexpected events, errors, and application crashes to ensure smooth
test completion. Once tests are run, WinRunner’s interactive reporting tools help your team
interpret results by providing detailed, easy-to-read reports that list errors and their origination.
WinRunner enables your organization to build reusable tests to use throughout an application’s
lifecycle. Thus, if developers modify an application over time, testers do not need to modify
multiple tests. Instead, they can apply changes to the GUI Map, a central repository of test-related
information, and WinRunner will automatically propagate changes to all relevant scripts.
Supported Environments:
WinRunner supports many environments, including the following. For more information on
WinRunner and a complete list of supported environments, please visit
http://www.mercuryinteractive.com/products/winrunner.
Browsers
Internet Explorer
Netscape
AOL
ERP/CRM
Oracle
PeopleSoft
Siebel
Baan
Platforms
Windows 98
Windows NT
Windows Me
Windows 2000
Windows XP
Programming Languages
PowerBuilder
Visual Basic
ActiveX
Oracle Developer
C/C++
Java, JavaScript,
JDK
Delphi
Centura
Page 14
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
Mercury QuickTest Professional provides the industry’s best solution for functional and regression
test automation for every major software application and environment. QuickTest Professional is a
next-generation automated testing solution that deploys the concept of Keyword-Driven testing to
radically simplify test creation and maintenance. Unique to QuickTest’s Keyword Driven approach,
test automation experts have full access to the underlying test and object properties, via an
integrated scripting and debugging environment that is round-trip synchronized with the Keyword
View.
QuickTest Professional satisfies the needs of both technical and non-technical users, enabling
companies to deploy higher-quality applications faster, cheaper, and with less risk. It works hand-
in-hand with new test automation system, Mercury Business Process Testing, to bring non-
technical subject-matter experts into the quality process in a meaningful way that results in better
alignment of IT and the business — and ultimately superior applications. With this product, QA
organizations can achieve a number of advantages:
Empower the entire team to create sophisticated test suites with minimal training.
Ensure correct functionality across all environments, data sets, and business
processes.
Fully document and replicate defects for developers, enabling them to fix defects
faster and meet production deadlines.
Easily regression-test ever-changing applications and environments.
Become a key player in enabling the organization to deliver quality products and
services, and improve revenues and profitability.
QuickTest’s normal recording mode records the objects in your application and the operations
performed on them. This mode is the default and takes full advantage of QuickTest’s test object
model, recognizing the objects in your application regardless of their location on the screen.
Analog Recording - enables you to record the exact mouse and keyboard operations you perform
in relation to either the screen or the application window. In this recording mode, QuickTest
records and tracks every movement of the mouse as you drag the mouse around a screen or
window. This mode is useful for recording operations that cannot be recorded at the level of an
object, for example, recording a signature produced by dragging the mouse. Note: You cannot edit
analog recording steps from within QuickTest.
Low-Level Recording - enables you to record on any object in your application, whether or not
QuickTest recognizes the specific object or the specific operation. This mode records at the object
level and records all run-time objects as Window or WinObject test objects. Use low-level
recording for recording tests in an environment or on an object not recognized by QuickTest. You
can also use low-level recording if the exact coordinates of the object are important for your test.
Note: Steps recorded using low-level mode may not run correctly on all objects.
Run Options
Page 15
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
SilkTest
SilkTest® is the industry’s leading functional testing product for enterprise applications, whether
Web, Java or traditional client/server-based. SilkTest is an easy-to-use test and automation
environment that offers many features that enable users to be highly productive in their software
test automation efforts. These features include workflow elements for test creation and
customization, test planning and management, direct database access and validation, the flexible
and robust 4Test® scripting language, a built-in recovery system for unattended testing, and the
ability to test across multiple platforms, browsers and technologies with one set of scripts.
SilkTest can help mitigate the challenges of a successful automation effort, such as lack of time,
budget and resources. Successful automation consists of a manageable development effort, the
ability to run the automation whenever and wherever it’s needed, and quick and accurate results
analysis. And SilkTest enables all three of these elements.
Rapid reusable test and automation infrastructure development – With today’s shortened
product release cycles, creating enough tests to thoroughly verify your application’s reliability can
be a daunting task. SilkTest offers many capabilities to promote rapid test development such as
the Basic Workflow for recording tests, the Data Driven Workflow for customizing tests to use
application data for improved test reusability, and AutoComplete for improved productivity of script
creation and automation infrastructure development.
Test and automation execution – SilkTest offers unique features for running tests such as an
out-of-the-box, yet extensible Recovery System, a separate Agent process for accurate user
simulation and a cost effective means for distributing test execution across multiple platforms
running in parallel. The Recovery System allows tests to be run unattended; should unexpected
errors such as application crashes occur, the errors are logged and the application is restored to
its original base state so that subsequent tests can run and produce valid results. SilkTest’s Agent
technology provides user simulation capabilities, driving the application exactly how an end-user
would, resulting in complete and thorough testing of the user experience. Since the Agent is a
separate and light weight process, it can be deployed to multiple boxes and controlled by the main
SilkTest component, easily enabling the distribution and synchronization of tests across multiple
platforms to shorten automation execution cycles. The 4Test scripting language and other
Page 16
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
architectural capabilities, such as the GUI abstraction layer, enable highly portable tests to
facilitate test execution across mixed operating systems and browsers.
Ease of test and automation maintenance –To help you cope with changes to the application
being tested, SilkTest gives you a powerful, highly customizable and low maintenance GUI
abstraction layer that maps the graphical elements of the Application Under Test (AUT) to their
automatically constructed counterpart objects in SilkTest. When the AUT’s GUI has been
sufficiently modified to require updating the abstraction layer, Project Workspaces help you quickly
navigate through the layer to find the sole map location to update. Many times, all that is required
is modifying the abstraction layer; tests that reference objects defined in the layer can continue to
run unmodified. Project Workspaces also lessen the effort required to update tests or automation
infrastructure when application logic changes occur. SilkTest’s scripting language, 4Test, is an
object-based language, thereby providing built-in ease-of-maintenance features offering you the
advantage for a single point of change when test or automation infrastructure modifications need
to be made.
Performance Tools
Load Runner
Rational Performance
Silk Performer
LoadRunner Overview
LoadRunner is a tool, or more accurately a collection of tools, which facilitates in performing load
testing on a system. LoadRunner is used to create a load or stress on a system, and can be used
to measure the responsiveness of the system to that load. The key concept behind LoadRunner is
the use of virtual users, or vusers.
Virtual Users
To perform a load test, a system needs many users to access the system concurrently, often one
hundred or more. Performing such a test with people is a logistical nightmare. When considering
that the test cycle is generally iterative, this problem is compounded. LoadRunner overcomes this
problem through the use of vusers. A vuser performs the actions that a real user would perform,
except that a vuser is controlled by LoadRunner, and the actions are performed automatically.
Vusers are created to perform specific transactions in the system. Once created a vuser can be
replicated as many times as required, thus producing an effective load as large as is required. With
the exception of GUI vusers, virtual users don’t use the client to perform their actions; they take the
place of the client. This allows in many vusers to be run on one workstation, as the overhead
associated with the client is not required. This does mean, however, that non-GUI vusers do not
test the performance associated with the client side of a system, only the network and server
associated performance.
Page 17
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
GUI vusers are used not so much to generate load on a system, but to test end-to-end
performance associated with various business processes. Although LoadRunner can control GUI
vusers, it is unable to create them. For this purpose, WinRunner is required.
JUnit is a framework for implementing testing in Java. It provides a simple way to explicitly test
specific areas of a Java program; it is extensible and can be employed to test a hierarchy of
program code either singularly or as multiple units.
Why use a testing framework? Using a testing framework is beneficial because it forces you to
explicitly declare the expected results of specific program execution routes. When debugging it is
possible to write a test which expresses the result you are trying to achieve and then debug until
the test comes out positive. By having a set of tests that test all the core components of the project
it is possible to modify specific areas of the project and immediately see the effect the
modifications have on the other areas by the results of the test, hence, side-effects can be quickly
realized.
JUnit promotes the idea of first testing then coding, in that it is possible to setup test data for a unit,
which defines what the expected output is and then code until the tests pass. It is believed by
some that this practice of "test a little, code a little, test a little, code a little..." increases
programmer productivity and stability of program code whilst reducing programmer stress and the
time spent debugging. It also integrates with Ant http://jakarta.apache.org/ant/.
Nunit
What Is NUnit?
NUnit is a unit-testing framework for all .Net languages. Initially ported from JUnit, the current
release, version 2.2, is the fourth major release of this xUnit based unit testing tool for Microsoft
.NET. It is written entirely in C# and has been completely redesigned to take advantage of many
.NET language features, for example custom attributes and other reflection related capabilities.
NUnit brings xUnit to all .NET languages. NUnit has two different ways to run your tests. The
console runner, nunit-console.exe, is the fastest to launch, but is not interactive. The gui runner,
nunit-gui.exe, is a Windows Forms application that allows you to work selectively with your tests
and provides graphical feedback.
Features
Console Runner
The nunit-console.exe program is a text-based runner and can be used when you want to run all
your tests and don’t need a red/yellow/green indication of success or failure. It is useful for
automation of tests and integration into other systems. It automatically saves its results in XML
format, allowing you to produce reports or otherwise process the results.
Page 18
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
Gui Runner
The nunit-gui.exe program is a graphical runner. It shows the tests in an explorer-like browser
window and provides a visual indication of the success or failure of the tests. It allows you to
selectively run single tests or suites and reloads automatically as you modify and re-compile your
code.
Page 19
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
Page 20
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
Page 21
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
Page 22
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
Page 23
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
References
Websites
http://www.exforsys.com/forum/software-testing/97699-qtp-material.html
http://72.14.235.104/search?q=cache:z33jrUcFseYJ:www.autotester.com/content/pdfs
/BPG-pdf.PDF+automation+testing+guide&hl=en&ct=clnk&cd=1&gl=in
http://www.spc.ca/products/Segue%20Products/silktest.pdf
http://www.qualtech-int.com.au/filedb/uploads/LRUG.pdf
Page 24
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected
Automation
STUDENT NOTES:
Page 25
©Copyright 2007, Cognizant Technology Solutions, All Rights Reserved
C3: Protected