Agenda

➢ Coles’ s QA journey with Thinksoft

➢Commitments Vs. Current state

➢Effectiveness & Metrics

➢ Efficiency &Metrics

➢Next Steps
 Coles’s QA journey with Thinksoft

QA JOURNEY SO FAR …
Successfully completed the
Defined Current State and testing of 3 Short term Demonstrated ADPART –
target state with implementation Implemented ADPART Personal Lines AUTO
projects and 4 Long term
road Map for QA for Personal Lines
Demonstrated Automation LOB
projects
Scripts Dry Run for Demonstrated
On boarded QA team and kick Automation Framework
Implemented Metrics Personal and Commercial
started Commercial Lines Maintenance Approach
Management Framework for QA Lines
projects

Set up dedicated Successfully completed the

Standardized QA Process and Provided Training on Mobile Automation Team for testing of 4 Short term
Templates across various Cloud to Coles Business Personal Lines and projects and 5 Long Term
phases Users Commercial Lines projects
On boarded QA team Designed Robust Successfully completed the Successfully completed the
and kick started Knowledge Provided UAT support testing of 3 Short term testing of 3 Short term
Personal Lines Management Portal for for Mobile Testing projects and 1 Long Term projects and 4 Long Term
transition QA project ▪ Robust Knowledge Management projectsPortal

▪ QC Standardization and Customization

▪ One Click Automated Metrics Dashboard

SERVICE SPECTRUM PROJECTS BEST PRACTICES IMPLEMENTED
▪ Templates and Status Reports standardization

▪ Metrics Driven Delivery Management

36 ▪ TCP - Scientific Estimation Methodology

▪ AQI / Project Quake for gauging Application Stability

21 ▪ PL Base Regression suite for Automation

9
7 2 0

2012 2013
Planned Delivered Deferred In progress Yet to start
 Commitments Vs. Current state
Graph depicting the number of RTS considered for estimation for SOW and the actual count of RTS executed

Assumptions :
Estimated RTS/ month for LOB based on 2012
trend : 20

90 RTS delivered
more than planned

Graph depicting the productivity improvement on RTS delivered per month

Steady improvement in PL and

action plan to replicate this in CL
 Effectiveness & Metrics
➢ Implemented effective Test design strategy resulting in optimal number of test cases

o Best in Class design techniques / concepts viz BVA, ECP , OATS

o Knowledge gained over a period of time on application and its stability

➢Increased test effectiveness with reduced defect rejection and leakage to subsequent phase.

o Additional test coverage based on production trend and business critical flows

Graph showing the reduced test set size by functional area through test optimization techniques

Nearly 40-50% volume of test

cases reduced through test
optimization techniques
 Effectiveness contd
Graph showing the test effectiveness on the defects raised by release
Test effectiveness better than
that committed for steady state Insufficient documentation
on the current state
application leading to more
defects rejection
Test Effectiveness - PL Test Effectiveness - CL

Knowledge gap resulted in

Graph showing the defect leakage to UAT/Prod by release incomprehensive coverage
Insufficient documentation
resulting in Knowledge Gap and
incomprehensive coverage
Defect Leakage- PL Defect Leakage- CL

* RC was Knowledge Gap and Incomprehensive coverage. Action items were identified and implemented in subsequent releases
 Efficiency & Metrics
Continuous efficiency improvement across LOB/Projects through the BP’s/ Value adds implemented

o Automation - Test Data automation, Test automation and Regression automation

o ADPART – Test Design automation

o Metrics Dashboard – Test Reporting automation

o Non Conventional Tools – FMR request automation, DEC validation tool, Quick Query Tool and XML Validation Tool

o Knowledge Portal – Robust knowledge management framework

Reduced unit Cost

Graph showing the cost per test case for PL & CL through increased
productivity and
efficiency

* Cost per Test Case - Cost / No of TCs prepared (Calculated for Offshore Core Team)
Efficiency & Metrics contd

$ 38 K Cost Savings due

to Efficiency
Efficiency - Best practices & Value Adds
Graph showing the cost saving through Best practices / Value adds

24786

38272.5
$91K

17758.874

10935

KM Non Conventional Tools Resource Onboarding Efficiency Improvements

 Improvements in Estimation methods
Two fold solution
• Change in Estimation Approach and Testing Strategy
• Change in Estimation Workflow

Parameters Current Method Proposed Method

▪ Testable Requirements, Complexity of Test Cases/

▪ Testable Requirements, Complexity of Test Requirements, Project Specific Parameters,
KEY COMPONENTS Cases/ Requirements, Project Specific Application Stability, Estimation for Testing based
Parameters on Developmental Changes (Table/Coding), Failure
probability (Defect Prone areas)

▪ Offshore Estimates Scenarios  Offshore ▪ Walkthrough TCP template to Coles and finalize
Feeds into TCP Template  Arrives at Effort the factor weightage based on RTS nature
WORKFLOW
Review and Sign off from Coles  Resource ▪ Provide the detail list of testable conditions with
Loading complexity to Coles QA for review as part of
requirement analysis of RTS
▪ Finalized conditions would be taken for estimation
per TCP guidelines. Ambiguous or missing
requirements evolved out of analysis would be
taken with BA for clarification
▪ Final estimation would be shared with Coles QA for
approval along with identified risks if any
▪ Resource loading based on approved estimates
▪ Test conditions are further shared with BA for
* Improvements brought in the proposed method business critical rating for implementing RBT
 Cycle Time Reduction

Requirement Phase Test Design Test Execution

Create Requirements/
Use Cases Baselined Release
Involve Offshore in
Req.
Walkthroughs and
Demos
Provide a demo of
Release specific
requirements to dev
and QA teams
Rate Business
Detailed and Coles BSUs to review test
Criticality
conditions and give feedback
Timelines
Defect triage meetings with
dev and QA teams
Incorporate review
comments
Macro Test Cases UAT and Sign Off
Design Test Cases

Clarify queries
Leverage
ADPART/OATs to
Queries Define SLA for supplement Design
Clarifications and Validate results and
Analyze use cases resolve within SLA Design Test
Conditions log defects
Queries Coles BSU

Identify application flows for testing Review Test Cases

based on RTS requirements Development Team

Implement Risk
Based Testing

Thinksoft QA onsite
Legend Automation of
Test Data
Analyze use cases and Regression Test Execute the Test
raise queries Accelerators Cases Cases
Thinksoft QA offshore
Optimizers

* Implementation of Accelerators and Optimizers across STLC