Audit of the Management of

Call Centre Operations

Audit Report

Project Number: 17031/09-10

SP-1036-12-12E

October 2012
Audit of Management of Call Centre Operations

Paper
ISBN: 978-1-100-21218-0
Cat. No.: HS28-205/2012E

PDF
ISBN: 978-1-100-21219-7
Cat. No.: HS28-205/2012E-PDF
Audit of Management of Call Centre Operations

Table of Contents

EXECUTIVE SUMMARY ................................................................................................. i

1.0 BACKGROUND ....................................................................................................... 1
1.1 Context ...................................................................................................... 1
1.2 Audit Objective ......................................................................................... 1
1.3 Scope ......................................................................................................... 1
1.4 Methodology ............................................................................................. 2
2.0 AUDIT FINDINGS ................................................................................................... 3
2.1 A governance framework is in place and is working well ........................ 3
2.2 An effective risk management approach is in place .................................. 5
2.3 Controls over quality assurance could be strengthened ............................ 6
3.0 CONCLUSION .......................................................................................................... 9
4.0 STATEMENT OF ASSURANCE ............................................................................ 9
APPENDIX A: Audit Criteria .......................................................................................... 10
APPENDIX B: Call Centre SMSM organizational structure ........................................... 13
APPENDIX C: Glossary ................................................................................................... 14
Audit of Management of Call Centre Operations

EXECUTIVE SUMMARY
Human Resources and Skills Development Canada’s (HRSDC) network of 14
specialized call centres is the key point of contact for citizens/beneficiaries to
access their client-specific information under the Employment Insurance (EI),
Canada Pension Plan (CPP), Old Age Security (OAS) and Employer Contact
Centre programs. In fiscal year 2011–2012, these specialized call centres
answered 8.69 million calls from Canadians.

The call centre network is managed through the four regions and National
Headquarters (NHQ). The Call Centre Directorate (CCD), within the Processing
and Payment Services Branch (PPSB) in NHQ, is responsible for the overall
planning and scheduling strategy, and developing the call centre network budget
and forecasts. NHQ also oversees program delivery and provides guidance
relating to quality and harmonization. The delivery of programs and services
within each region is governed by the regional Executive Head, Service
Management (EHSM).

Audit Objective

The objective of this audit was to provide assurance that the management control
framework supporting the EI and CPP/OAS call centres is in place and applied
consistently across all call centre operations.

Summary of Key Findings

 Roles and responsibilities are clear and accountabilities are defined.

 A performance management strategy, comprised of a suite of indicators
related to the network’s speed of service is in place. It is not clear how these
indicators and related performance data align with the network priority of first
contact resolution.
 Risk management principles and practices are embedded in call centre
operations.
 A workload management strategy is in place, but call centres continue to face
significant challenges in meeting current service levels.
 Scheduling of call centre resources supports service delivery and people
management.
 Training regime, procedures and systems are in place to support call centre
agents.
 A quality assurance program is in place to monitor calls and generate reports
for coaching. Guidelines are not consistently applied. Opportunities exist to
implement a methodology that takes into account operational changes.

Internal Audit Services Branch, HRSDC i

 Audit of Management of Call Centre Operations

Audit Conclusion

Overall, a management control framework supporting the EI and CPP/OAS call

centres is in place and applied consistently across all call centre operations.
However, the audit found areas for improvement related to quality assurance.

Recommendation

1. To ensure that relevant and useful information is used in its National Quality
Assurance Program (NQAP), PPSB should:
a. Implement a call monitoring methodology that will reflect a reasonable
distribution of monitored calls over time and consider operational changes,
e.g. recent updates to procedures may call for increased monitoring.
b. Ensure timely and complete coaching sessions between Citizen Service
Officers (CSOs) and Team Leaders (TLs) are conducted and documented.

Original signed by:

_________________________________________________
Vincent DaLuz, CA, CIA
Chief Audit Executive
Department of Human Resources and Skills Development

Audit Team Members:

Senior Director – Brigitte Marois, CGA, CMA

Audit Principal – Isabelle Barrette, CMA
Audit Principal – Rajni Patel
Nathan Ferguson
Emmanuel Ouellette
Amanda Elliott

ii Internal Audit Services Branch, HRSDC

Audit of Management of Call Centre Operations

1.0 BACKGROUND

1.1 Context

HRSDC network of 14 specialized call centres is the key point of contact for
citizens/beneficiaries to access their client-specific information under the EI,
CPP/OAS and Employer Contact Centre programs and to address enquiries
regarding the status of their files. Call centres also process transactional requests
such as changes of address, direct deposit or marital status, renewals and/or
cancellation of benefits. In fiscal year 2011–2012, these specialized call centres
answered 8.69 million calls from Canadians. Of this, approximately 800 EI call
centre agents answered over 5.3 million calls and approximately 400 CPP/OAS
call centre agents answered over 3.0 million calls.

The call centre network consists of four regions along with NHQ. The CCD in
NHQ is responsible for the overall planning and scheduling strategy and
developing the call centre network budget and call volume forecasts. NHQ also
oversees program delivery, provides guidance relating to quality and
harmonization, and leads the implementation of any enhancements to the
network. The delivery of programs and services within each region is governed
by the regional EHSM.

1.2 Audit Objective

The objective of this audit was to provide assurance that the management control
framework supporting the EI and CPP/OAS call centres is in place and applied
consistently across all call centre operations.

1.3 Scope

The scope of this audit included the CCD, PPSB at NHQ and the 14 EI and
CPP/OAS specialized call centres located across Canada (five EI sites, three
CPP/OAS sites, six sites housing both EI and CPP/OAS). The following call
centres were visited in February and March 2012:
 Edmonton EI & CPP/OAS call centres
 Montreal EI & CPP/OAS call centres
 Bathurst EI & CPP/OAS call centres
 Toronto EI & Scarborough CPP/OAS call centres

Internal Audit Services Branch, HRSDC 1

 Audit of Management of Call Centre Operations

1.4 Methodology

A number of methodologies were employed during the conduct of this audit, such
as:
 Interviews/discussions with management and staff from PPSB at NHQ and
from specialized call centres;
 On-site observation at regional call centres;
 Comprehensive review and analysis of relevant documentation; and
 File review and analysis of quality assurance reports and related action plans.

2 Internal Audit Services Branch, HRSDC

Audit of Management of Call Centre Operations

2.0 AUDIT FINDINGS

2.1 A governance framework is in place and is working well

Overall, roles and responsibilities are clear and accountabilities are well defined.
A performance management strategy, comprised of a suite of indicators related
to the network’s speed of service, is in place. Frequent updates on these
indicators and performance against them are provided. However, it is not clear
how these indicators and related performance data align with the network priority
of first contact resolution.

Analysis

Roles and Responsibilities

Since 2010, the specialized call centre network has undergone two key
organizational changes. The first has been the implementation of the Service
Management Structural Model (SMSM) – see Appendix B for call centre SMSM
organizational structure. The second organizational change to the network has
been the implementation of a co-managed model which delegated more
management responsibilities to the regions. Under the co-managed model,
EHSMs became accountable for a broader spectrum of call centre management
responsibilities.

Overall, interviews with call centre management and staff indicated that roles and
responsibilities were clear and understood. However, they also identified
uncertainty about the role of the call centre Business Expertise Consultant (BEC).
The audit team noted that roles and responsibilities for the BEC varied amongst
regions. In some instances, the BEC was seen as a technical expert who
provided advice to the Business Expertise Advisor (BEA), whereas in others, the
BEC was seen as a generalist that was primarily concerned with business
enhancements, process improvements and trends analysis. Management in the
regions reported that this inconsistency has been a cause of confusion or
uncertainty about the BEC’s role in call centres. There may be a need to review
existing roles of the BEC in order to improve consistency and clarity of the
position.

On the other hand, the TL’s role is consistent across the network. TLs’ range of
responsibilities include taking CSOs attendance, monitoring indicators, providing
feedback to CSOs, conducting coaching sessions based on BEAs’ quality
assurance reports and handling call escalations. The audit team noted that TLs
are not required to have program specific knowledge and this poses certain
challenges. For example, while conducting feedback sessions with CSOs based

Internal Audit Services Branch, HRSDC 3

 Audit of Management of Call Centre Operations

on the monitoring reports completed by BEAs, TLs are playing an intermediary

role between the CSO and the BEA. The TL might have to liaise with the BEA to
obtain advice and clarification which creates a duplication of efforts. There may
be a value added opportunity to equip TLs with the skills needed to better support
CSOs and advance the Department’s vision of first contact resolution.

In November 2011, the CCD developed and disseminated a draft Accountability

Framework and Performance Measurement Strategy to the regions. The
objective of this framework was to identify and distinguish NHQ and regional call
centre accountabilities within the new reporting structure. The strategy clearly
outlined network targets as well as regional and national accountabilities for
workload management, budget, human resources, quality monitoring and the
day-to-day management of the network. Interviews with regional management
and staff indicated that communication is in place and has been facilitated
through various national calls such as the National Operations Committee and
Call Centre Managers conference call.

Performance Management Strategy

The CCD has developed a comprehensive suite of key performance indicators
which are related to the network’s speed of service. These indicators, and
performance against them, are systematically provided to the regions, often on a
daily basis. However, the audit team could not find clear alignment between
these indicators and the current mandate and priorities for the network as defined
in the 2011–2014 PPSB National Integrated Plan and the 2012–2013 Report on
Plans and Priorities. One of the Department’s priorities for the call centre network
focuses on first contact resolution whereby citizen’s needs are fully resolved at
the call centre level.

The CCD provides the regions with a daily report on different performance
measures. Data included in these reports are forecasted call volume compared to
number of answered calls, service level (percentage), average speed of answer,
abandoned calls (percentage), average handle time, vacation and other leave
(percentage). In discussions with management, the audit team was informed that
first contact resolution of transactions is in fact currently measured, but these
results are not published. Regional call centres are also provided reports on
performance against expected call handling capacity, i.e. what is deemed to be
achievable with their current level of funding. Management informed the audit
team that these reports facilitate discussions with the CCD about improvements
to a call centre’s performance.

4 Internal Audit Services Branch, HRSDC

Audit of Management of Call Centre Operations

The current service level target is to have 80% of callers speak with a live agent
within 180 seconds. The audit team reviewed monthly performance trends dating
from April 2009 to March 2012 and noted that the network has not met this
service target in the three fiscal years reviewed. The fiscal year average for CPP,
OAS and EI call centres can be seen in the chart below.

Average service level performance by year (target 80%)

Fiscal year CPP-OAS call centres EI call centres
2009–2010 67% 42%
2010–2011 37% 51%
2011–2012 58% 32%

Regional management indicated that meeting current performance targets, in

particular service levels, poses a major challenge for the network. Reasons cited
include insufficient levels of resourcing, high staff turnover in some call centres
resulting in less experienced CSOs, and increased complexity of the calls
received. EI call centres have seen an increased number of calls as a result of
the backlogs in processing EI claims. CPP/OAS is also seeing an increasing
client base every year as more Canadians reach retirement.

2.2 An effective risk management approach is in place

Call centre operations are supported by risk management principles and

practices. A workload management strategy communicates call handling capacity
and comprises a number of tactics to address call volume. A central process for
scheduling of CSOs is in place to support both service delivery and people
management.

Analysis

Risk Management
Risks are being identified at the branch level as part of the Regional and National
Integrated Business Planning Processes. However, Regional Integrated
Business Plans do not necessarily identify risks at the call centre level. Although,
call centres have not documented their own risk registers, the audit team noted a
variety of measures that CCD has implemented to mitigate operational risks.
Minor opportunities for improvement have already been brought to
management’s attention during debrief meetings. The audit team also reviewed
Business Continuity Plans at the regional and CCD levels and found they include
detailed action plans in the event of a number of scenarios.

Internal Audit Services Branch, HRSDC 5

 Audit of Management of Call Centre Operations

Workload Management
The CCD applies a Resource Determination Model to determine the appropriate
mix of resources to meet forecasted demand and uses the draft accountability
framework to establish what is achievable within the network’s funding envelope.
The CCD, in consultation with the regions, has developed additional strategies to
manage the network’s workload, i.e. having a mix of part-time and full-time
resources to build latent call handling capacity into the network. While call
centres are not meeting their current service levels, workload management
strategies are in place to best manage national demand and workforce
availability within the current network’s funding envelope.

Scheduling of Call Centre Resources

A central process is in place for scheduling CSO resources that takes into
account information about planned off-phone activities, planned vacation time,
and incorporates breaks, lunch, and time to review any procedural updates. This
process strives to create schedules that will optimally serve the network. The
CCD has also developed a leave management strategy with the goal of
promoting planning at the regional level and recognizing the needed flexibility by
call centre management teams.

2.3 Controls over quality assurance could be strengthened

There is a training regime in place for newly hired CSOs as well as seasonal
refresher training. To support them in their role, agents access a web-based
reference tool which is also used to communicate updates or changes to
procedures. Our interviews with agents indicated that they would prefer more in-
class training sessions to validate their understanding of updated or new
procedures.

Call centres have a NQAP in place to monitor calls and generate reports for
coaching sessions. The audit team identified instances where the NQAP
guidelines were not consistently applied or followed. Taking into account the
interview and audit results as well as the way the NQAP is currently carried out, it
is not clear to what extent the quality assurance program provides relevant and
useful information to achieve consistency and improve client service.

6 Internal Audit Services Branch, HRSDC

Audit of Management of Call Centre Operations

Analysis

Training
Training is a shared responsibility between NHQ and the regions. The CCD is
responsible for coordinating the development of national program training while
the regions are responsible for delivering program and legislative training.

The New Hire Call Centre Curriculum (NHCCC) is a training program that
teaches agents technical and soft skills to deliver quality service to citizens. Staff
indicated that the NHCCC training packages adequately prepare agents for their
duties at the call centres. In addition to the initial training, “refresher” sessions are
provided, for example annual T4 and Guaranteed Income Supplement training is
provided to OAS/CPP agents. Staff expressed that they would benefit from
additional training sessions such as “current issues” and “soft skills”.

To support them in their day-to-day functions, call center agents access the
Common Reference Tool (CRT), a web-based reference tool that includes
policies, procedures and other information related to EI and CPP/OAS in a
structured manner. The CCD is responsible for maintaining and updating the
CRT.

Training material is increasingly being provided to agents in short memos as

opposed to holding traditional in-class settings. It is expected that agents
assimilate this information and any updates to the CRT during specifically allotted
reading periods totaling one hour per week. Agents expressed that it can be
difficult to go through and ensure proper understanding of the material received.
Majority of agents interviewed said they would prefer more in-class or group
training to better assimilate the material.

In addition to the CRT, call centres agents have access to a regional Business
Expertise support line manned by BEAs when they require assistance. Most
regions have taken the initiative of independently developing tracking tools to
collect information about the use of their support line with the goal of analyzing
the information for trends and training needs. At the time of this audit, this
regionally collected information was not available to the CCD for analysis. We
were informed that NHQ is currently working with the regions to implement a
national support line for agents.

Internal Audit Services Branch, HRSDC 7

 Audit of Management of Call Centre Operations

Quality Assurance
In 2008, the CCD implemented the NQAP with the objective of achieving
consistency and improving client service. The program is effectively a live call
monitoring regime. This program was revised in 2010 following observations and
comments gathered during the evaluation phase.

The NQAP requires each agent to have five calls monitored per quarter. Calls are
evaluated by BEAs using a scorecard of 26 technical and soft skills criteria. The
criteria cover all technical aspects of the call such as the greetings and
authentication, fact finding, resolving needs, promotion of services and the
closing. In addition, soft skills aspects such as an appropriate tone, vocabulary,
rate of speech, and demonstration of confidence are evaluated. A monitoring
report containing general observations on the agent’s strengths and areas for
improvement is written by a BEA. Within one week of monitoring, the report is to
be discussed between the agent and his or her TL in a coaching session.

The audit team reviewed a total of 156 monitoring reports from across the four
call centres visited. The results are as follows:
 For 37 reports (23.7%), the coaching session with the TL took place later than
one week. The range in which sessions were late was two weeks to two
months following the monitoring of calls.
 In 22 instances (14.1%), there was no record of the coaching session being
conducted by the TL.

Management indicated that the NQAP has been designed to serve as a coaching
tool. However, staff at call centres confirmed that the feedback received from
monitoring reports is not always useful from their perspective since the small
number of calls monitored during a quarter (versus the number of calls
answered) may not provide a representative view of their performance.
Additionally, these monitored calls are known to occur on the same day, at times
consecutively. The audit team is of the opinion that this methodology may not
provide a realistic depiction of the agent’s performance.

The NQAP guidelines provide for a standard of quality and consistency. Taking
into account the interview and audit results as well as the way the NQAP is
currently carried out, it is not clear to what extent the quality assurance program
provides relevant and useful information to achieve consistency and improve
client service.

8 Internal Audit Services Branch, HRSDC

Audit of Management of Call Centre Operations

Recommendation

1. To ensure that relevant and useful information is used in its NQAP, PPSB
should:
a. Implement a call monitoring methodology that will reflect a reasonable
distribution of monitored calls over time and consider operational changes,
e.g. recent updates to procedures may call for increased monitoring.
b. Ensure timely and complete coaching sessions between CSOs and TLs
are conducted and documented.

3.0 CONCLUSION
Overall, a management control framework supporting the EI and CPP/OAS call
centres is in place and applied consistently across all call centre operations.
However, the audit found areas for improvement related to quality assurance.

4.0 STATEMENT OF ASSURANCE

In our professional judgement, sufficient and appropriate audit procedures were
performed and evidence gathered to support the accuracy of the conclusions
reached and contained in this report. The conclusions were based on
observations and analyses of the situations as they existed at the time against
the audit criteria. The conclusions are applicable only for the Audit of the
Management of Call Centre Operations. The evidence was gathered in
accordance with the Internal Auditing Standards for the Government of Canada
and the International Standards for the Professional Practice of Internal Auditing.

Internal Audit Services Branch, HRSDC 9

 Audit of Management of Call Centre Operations

APPENDIX A: Audit Criteria

The conclusions reached for each of the audit criteria were developed according to the
following definitions.

Numerical Conclusion on Definition of Conclusion

Categorization Audit Criterion
1 Significant
Improvements
Required
Requires significant improvements (at least one of
the following three criteria need to be met):
 financial adjustments material to line item or
area or to the Department; or
 control deficiencies represent serious
exposure; or
 major deficiencies in overall control structure.

Note: Every audit criterion that is categorized as a

“1” must be immediately disclosed to the Chief Audit
Executive and the client Director General or higher
level for corrective action.
2 Moderate Issues Has moderate issues requiring management focus
(at least one of the following two criteria need to be
met):

 control weaknesses, but exposure is limited

because likelihood of risk occurring is not
high;
 control weaknesses, but exposure is limited
because impact of the risk is not high.

3 Controlled  well managed, but minor improvements are

needed; and
 effective.

4 Well Controlled  well managed, no material weaknesses

noted; and
 effective.

10 Internal Audit Services Branch, HRSDC

Audit of Management of Call Centre Operations

The following table outlines the audit criteria and examples of key evidence and/or
observations noted which were analyzed and against which conclusions were drawn. In
cases where significant improvements (1) and/or moderate issues (2) were observed,
these were reported in the audit report.

It is expected that: Conclusion Observations/Examples of Key Evidence

A defined and
communicated
governance framework is
in place to support call
centre operations,
outlining the decision-
making, delivery
mechanisms, roles,
responsibilities and
accountabilities.
3  Overall roles and responsibilities are clear and
understood with the exception of the BEC.
 Accountabilities are documented in the draft
Accountability Framework and Performance
Measurement Strategy. These accountabilities
have been communicated to regional
management.
 Key performance indicators have been
developed and performance is regularly
communicated to call centre management.
However, the alignment between these
indicators and the current mandate and focus
on first contact resolution is not clear.

A documented approach 3  Risk management is embedded in the roles

to risk identification, and responsibilities of management within call
assessment and formal centres and CCD.
response is in place and
 Within the CCD, the Intraday unit is ensuring
being used to support call
that the network’s call volumes are monitored
centre operations.
in real time to identify any issues quickly and
take actions appropriately.
 Call centre operations have implemented a
number of strategies that mitigate risk.
However, there is no documented risk
assessment (risk registers) at the regional call
centres or CCD level that presents risks
specific to call centre operations.
 Business Continuity Plans for call centres as
well as supporting systems are developed and
contain the appropriate amount of detail.
 Strategies are in place to best manage national
demand and workforce availability of the call
centre network within the current network’s
funding envelope.
 The scheduling of resources is done in an
appropriate and effective manner and strives to
support good people management.

Internal Audit Services Branch, HRSDC 11

 Audit of Management of Call Centre Operations

Appropriate controls are in 2  There are policies, procedures and supporting

place to facilitate provision systems in place to support call centre
of timely, accurate and operations.
consistent services to o CSOs would benefit from improvements to
citizens. the search functionality of the CRT and/or
its structural layout.
 In-class training initiatives are developed and
implemented appropriately.
 Procedures and guidelines are in place to
ensure security and protection of personal
information.
 A quality assurance program is in place and
monitoring activities are conducted.
o The NQAP guidelines provide for a
standard of quality and consistency.
o Sample tests show that follow-up activities,
namely coaching sessions and action
plans, could be strengthened.
o Given the resource intense nature of
NQAP, call centres may benefit from an
approach that is more focused on
operational changes and reasonable
distribution of calls to monitor calls.

12 Internal Audit Services Branch, HRSDC

Audit of Management of Call Centre Operations

APPENDIX B: Call Centre SMSM organizational structure

Internal Audit Services Branch, HRSDC 13

 Audit of Management of Call Centre Operations

APPENDIX C: Glossary

BEA Business Expertise Advisor

BEC Business Expertise Consultant
CCD Call Centre Directorate
CPP Canada Pension Plan
CRT Common Reference Tool
CSO Citizen Service Officer
EHSM Executive Head, Service Management
EI Employment Insurance
HRSDC Human Resources and Skills Development Canada
NHCCC New Hire Call Centre Curriculum
NHQ National Headquarters
NQAP National Quality Assurance Program
OAS Old Age Security
PPSB Processing and Payment Services Branch
SMSM Service Management Structural Model
TL Team Leader

14 Internal Audit Services Branch, HRSDC