Case study for Large Insurance Systems Testing Using Onsite / Offshore Model

Client Institutional business (IB) Central Quality Group

Clients IB Testing Services mission is to provide End-to-End testing solutions to our customers and to exceed customers expectations by delivering greater Quality, on budget ,on Time

US Locations Des Moines, IA Denver, CO Boston, MA Los Angeles, CA Moosic, PA Jersey City, NJ Offshore Location Chennai, India

Cognizant Case Studies, 2004

QA Testing Engagement Types

Requirements Gathering Test Case Creation Product specific Functional Test Execution Regression Testing Test Automation Integration Testing (Scenario Testing) Documentation Testing Compatibility Testing System Testing in Production Environment

Cognizant Case Studies, 2004

QA Functional & Technical Spectrum

Functional areas
Agent Management, Licensing and Compensation New Business Application and Client Acquisition Policy Administration Annutization and Accumulation Accounting and General Ledger Statements

Platforms
WIN NT / WIN 2000 / WIN 98 / WIN XP / WIN 2000 Server IBM Mainframes

Tools
Test Director, WinRunner and LoadRunner

Cognizant Case Studies, 2004

Team Organization

Client

+ Cognizant

Cognizant

Sponsors

Overall Program Coordinator

Account Manager

Test Manager

Client Testing Core Team

Client Manager

Cognizant Test Cognizant Test Analyst Lead

Test Lead

Domain Experts

ClientTest Lead

Cognizant Test Coordinator

Test Coordinator Support Services Quality Network

Automation Tool Expert Team

Client Support Team

Quality, Training Network, DBA Client Team (s) Developers SME Testers Testers Business Analysts Testers

Technical Architecture Team

Testers

Cognizant Case Studies, 2004

Team Organization (cont)

E-Commerce

SME
E SM

SM E

Actuary

Core Testing Team

SME
SM E

Tr ai ni ng

y lic Po ic rv Se e

Customer Acquisition

SME SME

SMEs from different business user groups are a part of the requirements management team which interfaces with the Core testing team

du ct

gt .

Cognizant Case Studies, 2004

Underwriting

E E SM SM

SME

Pr o

M Ag an en ag cy em en t

Knowledge Transition & Management

Start- Up
Objectives Understand Client IB products and proprietary technology Knowledge Transition Understanding Client IB testing processes and procedures Establish Acceptance Criteria Conduct Sample Test Runs Establish engagement management & Communication Structure

Steady State Knowledg e Repositor y Transition

Objectives Transition knowledge gained offshore Establish processes and procedures in line with the clients IB requirements Set-up Offshore Test Environment Provide proof of concept and achieve sign-off as per Critical Success Factor established Acceptance Objectives Provide flexible Resourcing options to manage testing peaks and troughs Fine tune test processes and procedures Deliver test case preparation and script development services as per Client IB requirements Conduct Test Execution and Analysis Provide timely test results and recommendations

Cognizant Case Studies, 2004

Functional Testing On-site Offshore Touch Points

Onsite QA Team Onsite Review (Business Analyst, QA Lead, AD Lead) Onsite Review (Business Analyst, QA Lead, AD Lead) Onsite Review (QA Lead) Onsite Acceptance (Business Analyst, QA Lead, AD Lead)

Sign Off

Sign Off

Sign Off

Offshore Review

Offshore Review

Test Planning Offshore QA Team

Test Case Generation

Script Generation Changes due to Bug Fixing

Execution

Development Team

Cognizant Case Studies, 2004

Defect Reports Test Logs Status Report Release Report

Defect Reports Test Logs

Test Cases

Test Plan

Test Scripts

Fix es

Planning & Execution

Ensure QA Leads with similar prior experience (in relevant line of business) are involved from the requirements phase Individual product owners coupled with multiplexing of resources Ensure core team (BA, QA and Dev) has unified view of the system Clean handshake with IT and BA
QA team delivers ambiguity review reports that can be turned over to IT and BA BAs collaborate with QA in preparation of test strategies / plans and test cases

Requirements prioritization
Clear segregation of must have from nice to have for Risk Based Testing

Use customized templates for data and test scenarios preparation Multiple shifts in on-site / offshore model to increase throughput Tap offshore testing pool for ramp up in support for peak release cycles

Cognizant Case Studies, 2004

Planning & Execution (cont)

Testing early in the life cycle to save cost and time

Instantiate requirements with specific real life business scenarios and validate Ensure all alternate flows and exceptions scenarios are captured Use hand calculations & rule sheets for up front scenario validation

Get the QA team to review and sign-off on requirements document

Good test for effectiveness of requirements documentation - Requirements

should be testable, if not, it is not a requirement

Ensure test cases have one to one mapping with requirements

Test Approach and acceptance criteria defined before the actual test cycle start date
Develop system test cases early in the life cycle and get sign-off from BA/IT

Walkthrough final document with ALL stakeholders to ensure sign-off

Sign-off for Test Strategy, Test Plan, Automation Design and Test Logs

Cognizant Case Studies, 2004

10

Tool Based Test Management

Tracing Defects through Requirements for efficient Root Cause Analysis and Process Improvement Integration of Test Cases with Automated Test Suite for efficient tracking and scheduling in an unattended mode

Cognizant Case Studies, 2004

11

Transparent 24x7 Web-based On-site / Offshore Defect Management

Testing Team

Request for Testing

Replicate Build

System Test Code

Bugs?

Development Team
Direct Bug to Development Team

Yes Log Bugs Test Director Log Bugs

No

Client Environment
Replicate Build Acceptance Test Code

Bugs? No Move To Production

Yes

Cognizant Case Studies, 2004

12

Configuration Management Process

Build ONSITE

OFFSHORE Test Scripts Script Script Reposi Reposi tory tory

Test Plan/ Test Cases

Document Repository

Configuration Management

Test Reports

Metrics Repository

Cognizant Case Studies, 2004

13

A few basics
Communication
Problem Statement
Lot of rework caused, effort and time wasted because of improper communication between team members and lack of proper issue tracking.

Version Control
Problem Statement
There are scenario in which test cases or documents are accidentally modified or deleted.Moreover we are not able to maintain a proper version of the documents.

Domain Training
Problem Statement
Quality of work and test execution speed affected due to improper/incomplete knowledge of the application.

Objective
To avoid the redundancy of questions asked. To avoid rework, effort and time wastage due to lack of coordination and communication.

Objective
To organize all data in to prevent accidental deletion and modification of test cases. one directory structure..

Objective
A continues knowledge transfer should be done before the test cycle to achieve the complete coverage of Test cases with in the estimated effort, in time. The testing team must have proper knowledge of application so that accurate defects can be logged. Client IB training session was conducted for 2 weeks which was attended by associates from Onsite and Offshore and it imparted a solid knowledge of the application to all attendees.

Benefits
Tracking of requirements and activities. Communication between team members. Proper Issue tracking and telecon tracking Time sheet for time management. Effort Wastage & Schedule Slippage avoided

Benefits
Prevent scenario in which any of test case or document is accidentally deleted or modified. Easy access to all data since they exist in same data structure. Restricted access to members.

Benefits
Associates got a better understanding of the application. Helped to create a good knowledge repository of the application.

Training calendar
Cognizant Case Studies, 2004
14

3tiered Organization

Tier I - Generic tester pool

Shift around based on work requirements to handle ramp-ups and ramp downs

Tier II - Dedicated test analysts oriented around business communities and sound automated tools knowledge(e.g., underwriting, claims, etc.)
Deep SME expertise in specific areas Excellent Automation Tools Knowledge

Tier III Test Architects, Data Managers, Domain Experts

Plan and integrate execution with upstream business, downstream IT and

program management (PMO)

Review, mentor, guide and lead testing teams

Cognizant Case Studies, 2004

15

Test Automation Best Practices

Decide to automate after,

Documenting current testing process and benefits Assess areas of improvement, modules to automate and finalize automation

requirements
Evaluate and finalize automation option Cost / Benefit Analysis

Automation using WinRunner, Certify, QTP, etc. to ensure efficient execution of large number of test cases Parameterize the scripts to achieve re-usability across different builds

Cognizant Case Studies, 2004

16

Key Test Automation Performance Measures

Goals: Increased Test Effectiveness 1. 2. Test Coverage Test Efficiency

Reducing Cycle Time 1. Manual Versus Automation

Reduction in Cost

Key Metrics
Productivity Defects found Defects found due to improper Test Cases Defects due to misinterpretation of requirements Customer Reported Defects
Cognizant Case Studies, 2004
17

Effort Variation Schedule Variation Size Variation Load Factor Automation Trend

Manual Vs Automation Effort Test Coverage through Automation Causal Analysis

(For Process improvement for the next cycle)

Results

Independent QA Test Team

Total Objectivity Avoid change management nightmare as they do not touch code True assessment for product readiness for market

Core Testing Team

Business continuity through knowledge management Boot-strap ramp-ups through core domain expertise 7 on-site and 21 offshore team ramped up to 12 on-site and 40+ offshore in 2-3 weeks timeframe Effective utilization monitoring Long term savings in testing effort and cycle time through automation

Flexible cost-effective sourcing

Phased ROI based Automation

Cognizant Case Studies, 2004

18