SQS Group Limited

Automation HealthCheck

Darryl Kennedy
February 2013
Executive Summary
• Vodafone have an existing test automation solution created using QC 10 and QTP 11
• The automated regression testing solution includes both ECC (GUI) and SRM (Portal)
• The existing solution has been in existence since it’s creation in December 2011
• The existing solution is maintained in Pune by VISPL Automation Team
• Test execution and reporting in carried out in Pune by VISPL Automation Team

Purpose of the Automation HealthCheck
• In order to maximise the benefits of automation, SQS have been asked to undertake an
independent review (HealthCheck) of the current test automation solution to ensure that
• Industry best practice has been employed
• The solution is extensible and provides the necessary portability and robustness
• The test assets created are re useable and easily maintainable
• Tests execute unattended, results and evidence are clear and easily interpretable
• SQS have been asked to provide any recommendations for improvement along with
supporting benefits of implementing any recommendations
Work Carried out
• SQS provided Vodafone with a test automation specialist who
• Reviewed a cross section of the QC and QTP assets available
• Assessed the approach to automation and the automation framework deployed
• Assessed the versatility, robustness and extensibility of the automation framework
• Assessed the approach to script development including test data
• Assessed error / exception triggers and recovery
• Assessed automation synchronisation and execution performance
• Assessed ability to execute unattended and the existing infrastructure

High Level HealthCheck Finding
• The QC and QTP solution adheres to certain design principles necessary for a
maintainable component driven framework but lacks necessary robustness
• The existing solution performs reasonably but essential attributes are missing or poorly
implemented which are necessary for enterprise strength automation
• Important strategic changes must be implemented if the true benefits of test automation are
to be realised before significantly extending the scale of the existing solution
Executive Summary
Automation Solution Summary
A good technical basis for an automated testing solution incorporating some features
provided by the HP Tool set. Consideration has been made to ensure the framework is
extensible and maintainable with minimal effort.
Some fundamental characteristics of test automation have been introduced such as shared
object repositories, data abstraction and the use of external function libraries to provide a
modular design.
Web and SAP automation present a number of challenges and although innovation has
been applied in the framework to provide solutions above what QC and QTP provide out of
the box some basic principles have not been adhered to:

 Modular design of the solution and use of external data and function library files
 Custom HTML reporting easily interpretable by non technical users
 XML Configuration files for multiple environment execution & Country specific test data
 Lacks an overall framework engine with the appropriate flow control
 Synchronisation and error handling needs a lot of work so able to run unattended
 No validation included into the functions which make up the Test Scenarios

The VISPL team were knowledgeable of their subject and extremely collaborative.
Executive Summary
Recommendation Priority & Implementation Complexity
The table below summarises all the recommendations made in this report into the priority
for implementation.

The table below summarises all the recommendations made in this report into the
complexity of the implementation.

Executive Summary
Implementation Priority Total Recommendations
High 36
Medium 13
Low 1
Implementation Complexity Total Recommendations
High 12
Medium 10
Low 28
Automation HealthCheck
The HealthCheck focussed upon
• Adherence to test automation best practice
• Ease and skill requirements of execution and maintenance
• Future proofing

The HealthCheck did not
• Test existing automation (any issues identified during the HealthCheck will be reported)
• Review the automated scripts for test coverage, accuracy to reflect business processes
or ability to reflect business testing needs
• Review the scope or prioritisation of test scenarios being automated
• Estimate the cost to run and maintain the scripts

Basis for the HealthCheck
• The findings are based on virtual meetings that took place on 4
and 5
• Thank you to the VISPL Automation Team for the help and support they provided
• Raghavendar Bandaru, Ganesh Kaushik and Nagesh Battula
Automation HealthCheck Scope
Criteria Rating Comments
Modularity Green
A good approach has been applied to ensure modular/component
design of the code for reusability.
Concurrent Use Amber
The modular design and abstraction layers allow for some concurrent
use. Large function library files and global objects create contention.
Reusability Amber
Some good solutions for reusability including different data tables for
country specific data. The configuration limitations make this difficult
during execution.
Suitability Amber
A good overall methodology adhering to modular design but there are
some limitations which must be addressed before further development
(see detailed recommendations).
Ease Of Use Amber
A clean solution using much of the standard QC and QTP
functionality. Easy to understand but lacks manual steps for end user.
Reliability Red
Standard QTP object timeouts may work well with SAP GUI but the
same approach can not be used for the Web Portal.
Maintainability Amber
Modular design and abstraction layers for code, test data, objects and
the use of external functions libraries and XML configurations.
Limitations with large function library files.
Reporting Red
HTML reporting introduced in addition to the standard QTP reports but
additional coding steps needed and no validation included.
Error Handling Red
Detection, reporting and handling of errors is poor. This will also
occur more often due to the lack of dynamic synchronisation.
Test Automation Score Card
Criteria Rating Comments
Audit Trail
Naming conventions exist for Test Scenarios and Functions which
are adhered to. Additional detail should be applied to make it easier
for end users to interpret these easier i.e. SAP T Code.
Test Scenarios and functions are distinctly named but do not contain
a unique identifier.
Mapping to
No requirements exist in QC. A matrix of requirements is
maintained by the functional test team using Excel. This makes
traceability time consuming.
Mapping to
Manual Tests
Manual and technical assets are duplicated in QC so traceability is
difficult and time consuming. Extra actions must be performed by
the end user to see the manual steps. Manual and technical assets
may get out of synchronisation with each other.
Requirements Red
No requirements exist in QC so there is no technical solution
available to ascertain coverage of the technical assets.
Test Automation Score Card
Day 1
• Initiation meeting with Tom Sewell to agree scope of the HeathCheck
• Review (SQS & VISPL) of the QC and QTP framework implementation
• Review (SQS & VISPL) of the Scenarios, Objects, Reporting and Error Handling
• First day wrap up meeting with Tom Sewell
Day 2
• Review (SQS & VISPL) of the Data, Config. File, Reporting, Execution & Development
• Execution (SQS) of Portal and SAP Test Scenario
• Initial Report Creation
• Walkthrough of the findings report structure with Tom Sewell
Day 3
• Report Finalised
• Assessment of the automation
• Recommendations and examples of findings
• Issues identified during the HealthCheck
• Walkthrough of findings (Vodafone & SQS)
Schedule of Activities
In order to conduct the HealthCheck, SQS requested access to the following
• Access to all test assets
• Access to the system under test and existing manual test scripts
• Access to working end-to-end automation test scripts
• Access to a sample of custom built drivers for the various automation tools
• Access to sample reports generated
• A demonstration of the test execution:
• Against the applications under review
• Using manual run
• Via build triggered run if applicable
• Availability of the VISPL Automation Team
• Environment support during demonstration and test execution of the solution
• Reasonable turn around times on queries raised whether against the VISPL
Automation Team or external (for example a manual test or solution design query).

All of the above was provided as requested by the VISPL Automation Team.
Pre-requisites and Support Criteria
Detailed Findings &
Recommendation Guidelines
• Each of the recommendations made has a unique identifier
• Each of the recommendations is clearly defined
• Each of the recommendations is supported by a benefit
• Each of the recommendations has an implementation priority assigned
• Each of the recommendations has an implementation complexity assigned

Ordering of Recommendations in Table format
• The unique identifier does not reflect an implementation order or importance
• Many of the recommendations are linked and must be considered as a whole
Recommendations for Quality Center
ID Recommendation Benefit Priority Complexity
Implement the usage of the Detail field
in QC for each of the assets stored
Provides the end user with a clear
overview of the purpose of the test
High Low
Implement the usage of the Status field
in QC for each of the assets stored
Creates a development workflow and
improves visibility of the overall
High Low
Implement the usage of the Design
Steps in QC for each of the assets
Ensures the manual tests assets and
the technical test assets do not verge
apart from each other. Provides the
technical team a clear and easy way of
High Low
Store the requirements in QC which
are currently tracked using Excel
Provides dynamic traceability of
progress and enables tracking of
issues found
High Low
Move the Test Resources outside of
QC to enable the implementation of an
enterprise strength version control
solution (unless the upgraded QC
version control provides the necessary
QC provides a basic level of version
control but not sufficient for
development of automated test suites
on an enterprise level with many
people working concurrently
Medium High
Recommendations for QTP Scripts
ID Recommendation Benefit Priority Complexity
Naming Convention with unique
identifiers applied to all assets
Simplifies the traceability of all assets
and improves the reporting in QC
High Low
Detailed headers and script comments
to be implemented to all assets
Improves the ease of use of the
technical assets for collaboration
Medium Low
Code repeated in each test which
should be a framework function i.e. Env
Simplifies and speeds up maintenance
activities required on all tests
Medium Low
All tests must be suitable for
environment portability. Load XML
environment variables from a central
file dynamically
Supersedes XML loaded manually to
every test which is a slow and
cumbersome process prone to error
High Low
Start up and teardown functions to be
applied to all assets.
Allows for quick changes to be made
globally to all test assets
High Low
Break away from functions and create
QC assets for each business
Will introduce overall framework and
flow control and ability to execute tests
from a failure point
Medium High
Use the QTP object to dynamically load
run-time setting at the beginning of
each test. Screen prints Never/Always
and global object timeouts
Applies consistency to every test case
running in the suite and removes the
need to execute VBS manually
High Low
Recommendations for QTP Objects
ID Recommendation Benefit Priority Complexity
Remove version numbers from current
OR in use
Removes the requirement to manually
associate this with every test using it
High Low
Changes the object description, don’t
just use the SAP generated name
Makes scripts readable, intuitive and
allows other users to understand them
High Low
Object Windows must be distinct,
merge _2 and _3 as appropriate
Keeps OR tidy and reduces the
number of windows to maintain
High Medium
Objects must be distinct, merge _2 and
_3 as appropriate
Keeps OR tidy and reduces the
number of objects to maintain
High Medium
Object property and value pairs must
be improved and regular expressions
applied where appropriate
Clearer object definition and reduces
fails during execution
Medium Medium
ORs for SAP and Web should be
broken up into smaller files
Reduces the change of corruption and
allows concurrent development
Medium Low
Portal OR cleaned up to remove many
instances of Browser object
Keeps OR tidy and reduces the
number of objects to maintain
High Medium
Remove all local objects once merged
into the shared OR
Prevents tests failing on shared and
local conflicts
High Low
Store shared OR in a central location
and not on a C:\
Easily accessible by all developers High Low
Recommendations for QTP Functions
ID Recommendation Benefit Priority Complexity
Large single files over 12,000 lines of
code must be broken down into smaller
libraries of technical functions and QC
assets for business components
Collaborative development is
impossible and contention exists.
Using QC assets will improve the
traceability and maintainability
High High
Remove the version numbers from the
names of function libraries
Reduces the manual overhead of
continually associating to every test
High Low
Library files and individual functions
must have detailed headers which
include modification history
Provides a level of self documenting
solution and improves collaboration
High Low
Naming conventions must be improved
and include SAP TCode where
Improved script readability and ease of
debugging when errors occur
High Medium
A Version Control solution must be
adopted, taking code out of QC if
Improves concurrent development,
traceability, multiple environment
execution and reversion where
High High
Create traceability with the manual
assets stored in QC by creating
technical assets in the same QC entity
Improves the overall knowledge of the
team. Keeps manual and technical
assets in sync and improves
High High
Recommendations for QTP Functions
ID Recommendation Benefit Priority Complexity
Dynamic synchronisation must be
applied to all business components and
all hard coded waits removed
Currently there is limited or none. The
standard QTP object synchronisation
points are not robust enough for
unattended execution and porting the
scripts across different test
High Medium
Login functions must handle multiple
OPCO logins for all authorisation levels
These should be stored in an external
Medium Low
An overall flow control must be
implemented into every business
component. Currently QTP simply
executes linear with no checks to
ensure events are happening as
expected to allow control to pass back
to the calling test for error reporting and
This will prevent tests from failing
unexpectedly and provide a much
greater visibility of what caused the
failure in the QTP execution report
High High
Evaluate all reporter message to
ensure failing steps are reported
Found an example of code which fails
but report message contained a pass.
An If / Else statement was completely
bypassed as SAP was on the incorrect
High Low
Recommendations for QTP Functions
ID Recommendation Benefit Priority Complexity
Remove the dependency on the QTP
Recovery Manager. This should be
used as a catch all not an every
instance solution. These should be
handled in the code itself
Much greater control of the error
handling and provides far more detail
in the QTP execution report as to the
cause of the failure. Can then also
incorporate screen captures
High High
Validation and checkpoints must be
applied to all business components
This provides detailed feedback as to
why tests failed and allows tests to run
in batches unattended without the
need for constant monitoring
High Medium
Error detection, error reporting, error
handling and recovery must be applied
to all business components.
QTP Run settings updated with a VBS
fail to suggest Test Stops on error, this
is only part of the necessary error
handling. Tests must also reset
conditions i.e. close browser and then
continue to the next test cleanly
High High
An overarching function is missing to
handle and recover from failure
Inbuilt QTP setting used to capture
screenshot on failure. This is typically
unreliable and should only be
incorporated in the code where failures
have been anticipated to occur
High Medium
Recommendations for QTP Functions
ID Recommendation Benefit Priority Complexity
Improve the robustness of the business
components by adding additional
validation to the objects before use.
Creating overload functions in QTP
could include this and many other
checks i.e. Enabled, Visible etc. As
they are overloaded no change to the
end user so scripts can still be
recorded in QTP.
High High
Function to create HTML report
requires lots of additional code in the
functions and should be included in the
overloaded functions.
This will simplify the script creation and
ensure consistency throughout over
business component
Medium High
HTML creation functions must also
upload to QC RUN table.
Reduce manual overheads High High
HTML functions for automated
reporting must be included in
overarching function to handle errors
Ensures these are created and
uploaded during test runs which
encounter errors
High Low
Recommendations for Test Data
ID Recommendation Benefit Priority Complexity
OPCO User Accounts are stored on a
central file governed by environment
and country
Easy for the end user to maintain and
provides versatility for multiple
environment execution
Medium Low
Different tabs for OPCOs should be
amalgamated to single sheet and QTP
function created to select appropriate
Reduces overall maintenance as there
are less Excel sheets to manage
Medium Medium
Needs to have a unique identifier for
each Excel sheet which map to the test
Improves visibility and traceability High Low
Additional iterations of data should be
incorporated into the Excel data sheets
to test more data permutations
Incorporating additional data sets
provides greater coverage easily with
applying a unique identifier to the data
row and mapping to test name.
Current constraint in the framework
does not allow this
Medium Low
Recommendations for Test Execution
ID Recommendation Benefit Priority Complexity
6 to 10 dedicated QTP servers hosted
virtually so accessible from anywhere
load balanced using QC host groups
instead of the 4 machines currently in
Utilises more of the 20 QTP licenses
available and provides greater
flexibility from the 4 machines used
currently which are also the day to day
machines of the team. Tests can then
be developed and executed

Removes the dependency that the
user must be present to monitor the

Removed the dependency of the
Developer running their own script
High Low
Hardcoded C:\ path in the HTML
reports must be removed
This must be addressed to allow many
machines to run unattended and see
results from anywhere
High Low
HTML report is uploaded dynamically
to QC in the same way manual tests
do, these should be uploaded to the
RUN table / Execution history
Ensures results can be accessed from
anywhere and not just the machine
they were executed on. Provides clear
and concise audit trail
High Medium
Recommendations for General Adoption
ID Recommendation Benefit Priority Complexity
QTP Patching level should be brought
up to the latest version
Issues already identified by HP will be
Low Low
Development work items must be
visible to everyone so it’s clear who is
working on what changes
Greater visibility of the changes
planned and progress tracked against
each. Allows work to be decided on a
priority basis
Medium Low
Solution documentation must be
created for the framework
Allows up skilling of anyone who
needs to develop, maintain or execute
Medium Low
Solid Version Control solution required
which may include moving outside of
Unable to develop and execute tests
concurrently so different versions of
QTP code base must be maintained.
QTP Environments in General Options
along with dynamic library loading in
QTP 11 allow the developer to quickly
change environments and versions
High High
Implementation Plan
Implementation Plan – To be followed as a pilot phase
Implementation Area Detailed Description
Build a Core Framework
1) Object Identification Mechanism to be streamlined
2) Object Repository to be re-structured
3) Custom functions to be built to replace QTP inbuilt methods
4) Incorporate Error handling, Object Validation, Screen Capture for evidence, Object sync
and Checkpoints in the scripts
5) Reduce the large Function Library by moving the "Application Specific" functions to
modular QTP scripts
Test Plan
1) Convert the Application Specific functions along with the new framework into individual,
modular, robust QTP scripts
2) Include the design step for each T-code - for better readability and review by Non-
Technical/ Business Users
Test Lab
1) Update the scenario details, description, Objective, Pre-requisite
2) Build the end-to-end scenario by calling the modular QTP scripts into Test Lab and
setting the execution flow
3) Incorporate custom reporting to generate a report at the end of each scenario execution
4) Datasheet consolidation
Scheduling/ Host Management
1) Get the physical assets in place (20 Concurrent user license for QTP available)
2) Prepare a scheduling strategy to run the scenarios overnight where possible to better
manage the execution timelines
SQS Group Limited
Darryl Kennedy
Principal Consultant
Email : Darryl.Kennedy@sqs.com
Tel: 07950 128 060

SQS Group Limited
82 King Street,
Manchester, M2 4WQ

Internet: www.sqs-uk.com
Internet: www.sqs-group.com

Sign up to vote on this title
UsefulNot useful