SAI Performance Measurement Framework

Presented by: Yngvild H. Arnesen, INTOSAI Development Initiative

ICGFM Winter Conference, Washington, DC, 10-12 December 2012

1

Presentation Outline
1. Introduction 2. Background and rationale for developing a SAI Performance Measurement Framework (PMF) 3. Responsibilities and reporting lines 4. Indicative content and design 5. Looking ahead 6. Further information
2

1. Introduction

What is a Performance Measurement Framework?

Tool for measuring performance
of an individual, system or organization against an agreed performance scale accommodating local circumstances
Not everything that matters can be measured

To enable
Better management of performance Monitoring over time Evaluation and lesson learning

Contributing to
Understanding performance Improving performance
4

What is the SAI Performance Measurement Framework (PMF)?

A Performance Measurement Framework tailored to SAIs

Intended to give a high level assessment of SAI performance against established INTOSAI good practice:
The International Standards of Supreme Audit Institutions (ISSAI) framework The Framework on the Value and Benefits of SAIs Other INTOSAI guidance material

Combination of objective measurement and qualitative assessment Analyzes 7 domains of SAI performance in the country context and aims at assessing the value and benefits of the SAI to society (impact)
5

SAI PMF How will it be used in Practice?

1. Using the SAI PMF will be voluntary for the SAIs the tool will be a public good which SAIs can make use of if it helps them achieve their objectives 2. The SAIs themselves will make the key decisions regarding the assessment 3. The SAI will also decide whether or not to publish results 4. Because it is voluntary, the SAI PMF is not intended to replace any internal performance measurement procedures

Who decides to initiate a SAI PMF? Who decides when to do a SAI PMF? Who decides how to do a SAI PMF? Who decides if, when and how to publish a SAI PMF?

Head of SAI Head of SAI

Consulting with INTOSAI regions & donors as appropriate Every 3-5 years recommended

Head of SAI
Head of SAI

Approach selected to meet the SAIs objectives 3 broad approaches Quality assurance to promote objectivity and credibility

3 Approaches to Conducting a SAI PMF Assessment

There are three main approaches to conducting an assessment using the SAI PMF. The PMF will be equally well suited for all three:
1. SAI Self Assessment 2. INTOSAI Peer Review 3. External Assessment

Trained Team from within the SAI

Peer SAI INTOSAI Regional Body

Professional PFM Body Donor Agency Consultant External auditor

SAI Involvement The approach will not necessarily affect how the assessment is carried out but the SAI may use the results for different purposes depending on the approach 7

2. Background and rationale for developing a SAI Performance Measurement Framework (PMF)

Background
Full set of International Standards of Supreme Audit Institutions (ISSAIs) adopted by INTOSAI in 2010

INTOSAI Framework on the Value and Benefits of SAIs adopted 2010

Increased focus on use of country public financial management systems within the international donor community

Mandate: Johannesburg Accords (INCOSAI XX, 2010)

Responsibility
Working Group on the Value and Benefits of SAI

Task
Collaborating with Cognizance of

Develop further credible measurement tools to support evaluations in terms of the framework for Communicating and Promoting the Value and Benefits of SAIs

INTOSAI Capacity Building Committee INTOSAI-Donor Steering Committee

Tools that are already available at the level of international organizations

Timetable

For consideration at XXI INCOSAI in 2013

10

Rationale for INTOSAI to Develop a SAI Performance Measurement Framework

SAI PMF Stakeholders and Purpose
Supreme Audit Institutions Improve SAI approaches to capacity development through use of performance management Identify opportunities to strengthen SAI performance Stronger SAI-led capacity development programs SAIs become model learning organizations

Benefit to the SAI

Domestic Parliaments and Citizens

SAI

Promote transparency and accountability through credible public reporting of SAI performance (ISSAI 20)

Enhanced SAI integrity and credibility with domestic stakeholders

INTOSAI Regions and INTOSAI Global Community Enable better peer learning International Development Community (Donors and SAI Providers of Support) Demonstrate SAI performance changes to key stakeholders

Evidence-based policies and better targeted global and regional programs

Maintain external support for SAI capacity development 11

Why should SAIs use the PMF (1)

As a step towards ISSAI implementation
Increased understanding in the INTOSAI community of what good practice for SAIs entails Individual SAIs can benchmark their performance against good practice and get an indication of where there is room for improvement They can use the assessment as a basis for more detailed ISSAI compliance assessments

To demonstrate progress and value and benefits to society

The framework will be well suited for measuring progress over time and demonstrating this to external stakeholders Show to stakeholders how the SAI contributes to strengthening public financial management
12

Why should SAIs use the PMF (2)

Internal performance measurement / annual reporting
Help SAIs better manage their own performance ISSAI requirement to assess and report on own performance lead by example and become a model organization

Inform and secure support for capacity development initiatives

Identify strengths and weaknesses, increase understanding of reasons for good and weak performance Help build a culture for performance improvement SAIs can use assessment results to get support from donors and other external stakeholders
13

The Role of SAI PMF Assessments in SAI Capacity Development

Implement SAI Development Action Plan Formulate or Update SAI Development Action Plan High Level Performance Overview Progress Monitoring

Formulate or Update SAI Strategic Plan (Long Term)

(Medium Term)

SAI PMF Assessment

Recommend SAI Capacity Building Measures

Identify Main SAI Strengths and Weaknesses

SAI Capacity Building Needs Assessment

Investigate Underlying Causes

SAI Performance Measurement Framework (SAI PMF) & ISSAI Compliance Assessment Tool (ICAT)
1. Monitor change over time or assess capacity building needs? 2. High level overview or detailed drill down tool? 3. Assessment against good practices or against ISSAI framework?
ISSAIs
Human Resources Strategic Planning Value & Benefits Framework

4. Score levels of performance or assess compliance with standards and investigate causes?

SAI PMF

ISSAI Framework
Level 1 Level 2 Level 3 Level 4
Founding Principles Lima Declaration Prerequisites for the Functioning of SAIs Fundamental Auditing Principles Auditing Guidelines

ICATs

ISSAI Framework
Level 4 Level 3 Level 2 Level 1 What needs to be in place to implement level 4?
Auditing Guidelines Fundamental Auditing Principles Prerequisites for the Functioning of SAIs Founding Principles Lima Declaration

International Standards of Supreme Audit Institutions

3. Responsibilities and reporting lines

16

Responsibilities and Reporting Lines

INTOSAI Congress
Approval

In Consultation with: INTOSAI Capacity Building Committee INTOSAI-Donor Steering Committee Other Relevant INTOSAI Committees

INTOSAI Working Group on the Value and Benefits of SAIs

Oversight and Direction

WGVBS SAI PMF Task Team

Development of SAI Performance Measurement Framework
Coordinator: INTOSAI-Donor Secretariat @ IDI
INTOSAI Capacity Building Committee Members INTOSAI-Donor Steering Committee Members

SAI PMF Reference Group

Technical Advice, Review and Quality Assurance
INTOSAI Capacity Building Committee Members INTOSAI-Donor Steering Committee Members
Other Groups: PEFA Secretariat, International Budget Partnership

17

Task Team
INTOSAI Regional Organizations: 1. African Organization of English Speaking SAIs(AFROSAI-E) 2. African Organization of Francophone SAIs (CREFIAF) SAIs: 3. Bahrain 4. Brazil 5. Chile 6. Germany 7. India 8. Mexico 9. Russia 10. United Kingdom Development Agencies: 11. Inter American Development Bank Coordinator: 12. INTOSAI-Donor Secretariat @ IDI

Reference Group
INTOSAI Regional Organizations: 1. Pacific Association of SAIs (PASAI) SAIs: 2. Canada 3. Denmark 4. Estonia 5. European Court of Auditors 6. Japan 7. Netherlands 8. Norway 9. Peru 10. Sweden Development Agencies: 11. African Development Bank 12. OECD 13. European Commission 14. World Bank Civil Society Organizations: 15. International Budget Partnership 16. Global Reporting Initiative (GRI) 17. PEFA Secretariat

18

4. Indicative content and design

19

INTOSAI Working Group on the Value and Benefits of SAIs:

Criteria For an Effective SAI PMF

1. Comprehensiveness: Broad coverage of all key aspects of an SAIs work and its contribution to good governance, accountability and transparency 2. Objectivity: Indicators to measure performance and progress are objective 3. Subjectivity: Subjective factors can also be captured, e.g. through a narrative performance report 4. International agreement: performance indicators and measurement scales reflect agreed international standards and good practices where available 5. Relevance: Relevant to all SAIs, regardless of their administrative heritage or level of development 6. Performance: Enables analysis of SAI performance and identification of development challenges and opportunities 7. Progress: Facilitates the consistent measurement of SAI performance over time 8. Consistency: Consistent with pre-existing tools (both SAI coverage within broad public financial management measurement frameworks, and SAI specific performance measurement tools) 9. Compliance: Measures actual audit practices, as well as the quality of the SAIs legislative / regulatory framework and internal guidelines / manuals 10. Brevity: Comprises the minimum number of performance indicators possible to cover key aspects of an SAIs performance, so as make the tool of practical benefit to SAIs 11. User-Friendly: facilitates easy use by SAIs as self assessment or peer assessment tool
20

SAI PMF Task Team: Initial Work and Decisions

Mapping of 20 Existing SAI Assessment Tools
No existing framework fully meets INTOSAI needs Develop a new tool drawing heavily on existing ones (AFROSAI-E, PEFA, INTOSAI Peer Review Guide, IDI Capacity Building Needs Assessment Handbook)

What to Measure Performance Against?

ISSAI Framework where possible recognizing it will continue to evolve over time The SAIs mandate INTOSAI guides & other generally accepted international standards or good practices

What is performance? What are we going to measure?

External performance delivery of core audit activities (audit coverage, submission of audit reports, etc.) Internal enablers of performance SAI strategy, audit processes, SAI support services SAIs operating environment (i.e. independence and legal framework) recognizing this is largely beyond the control of the SAI Reported in the context of the specific country governance and public financial management environment and its impact on the SAIs performance HOLISTIC MEASUREMENT OF THE SAI AND ITS ENVIRONMENT
21

SAI PMF: Evidence Based Assessment of Performance

Holistic assessment of the SAI and its environment including factors which are not covered by the indicators

Qualitative assessment (Performance Report)

Objective measurement to inform qualitative assessment and track progress over time

Measurable indicators

22

SAI Performance Measurement Framework Structure

SAI Environment SAI Capability SAI STRATEGY C. Strategy for Organizational Development 1 indicator SAI CORE BUSINESS D. Audit Standards and Methodology 3-5 indicators
E. Management F. Human & Support Resources Structures 2 indicators 3 indicators G. Communication and Stakeholder Management 3 indicators

SAI Performance A. SAI Performance 3 indicators

Impact

B. Independence and Legal Framework 3 indicators

Contribution to Public Financial Management, Governance and Tackling Corruption

SAI SUPPORT SERVICES

Public Financial Management Environment

23

SAI Performance Measurement Framework Structure

Note: all indicators should try to capture both what is written in legislation and procedures (de jure) and what happens in practice (de facto)

Foundational factors set by the legislative and/or executive, but which can be influenced by the SAI in the long term

SAI Environment

SAI Capability
Under the control of the SAI

SAI Performance
Under the control of the SAI

Impact
Measured using other assessments. Achievement of impacts can not be wholly or directly attributed to SAI results. However, the way in which the SAI results are likely to contribute to high level impacts should be summarized in the narrative SAI performance report.

SAI STRATEGY
C. Strategy for Organizational Development Strategy for organizational development A. SAI Performance
Financial Audit Results Compliance Audit Results Performance Audit Results SAI Value Added Services SAI Annual Report

B. Independence and Legal Framework Independence of the SAI Independence of the Head of the SAI and its Officials Mandate of the SAI

D. Audit Standards and Methodology

SAI CORE BUSINESS

Audit Planning for the SAI Financial Audit Compliance Audit Performance Audit Quality Control & Quality Assurance of Audit Processes

E. Management &

F. Human Resources
Management of Personnel Professional development Leadership

Support Structures Management and Internal Control Support Structures & Infrastructure

G. Communication and Stakeholder Management Communications Strategy, and Internal Communication Communication with the Government Branches Communication with the Media, the Citizens and the Civil Society

SAI Contribution to Aggregate Fiscal Discipline Efficient Resource Allocation Efficient Service Delivery Good Governance, Accountability and Tackling Corruption

SAI SUPPORT SERVICES

Public Financial Management Environment

Influences the effectiveness of the SAI but not under direct control of the SAI. Measured through other assessments but referred to explicitly in the narrative SAI performance report.

Quality and Timeliness of Annual Financial Statements (PEFA PI-25) (Including completeness, timeliness and accounting standards used) Legislative Scrutiny of External Audit Reports (PEFA PI-28) (Including timeliness & extent of hearings, recommendations of the legislative & implementation by the executive)

SAI Performance Report

- end product from an assessment (30-40 pages?)
Enable SAI heads and senior management to
Have a high level overview of the SAIs performance Promote and communicate the value and benefits of the SAI Identify barriers to improved performance Track performance changes over time

Explores how SAI external performance is constrained by

Country context and PFM environment SAI environment SAI internal capability

Provides a coherent and holistic assessment of SAI performance

Informed by objective and evidence based performance indicators Analyzed within context Considered as a whole not the sum of its indicator scores
25

SAI Performance Report

i. Quality Assurance Statement ii. Observations on the SAIs Performance and Impact (4-5 pages?)
Integrated assessment of SAI performance: Linkages between the country context, SAI environment, SAI internal capability and SAI external performance Value and benefits of SAIs: making a difference to the lives of citizens Enablers or constraints to stronger impact from SAI performance Analysis of the SAIs capacity development efforts and prospects for further capacity development

iii. SAI Management reaction 1. 2. 3. 4. Introduction Country and SAI Background Information Assessment of the SAIs Environment, Capability and Performance SAI Capacity Development Process

Assessment against seven domains of SAI performance (evidence based indicator scores) Impact of the country context and PFM system on SAI effectiveness

Annex 1: Performance Indicator Summary & Tracking Performance Over Time Annex 2: Sources of Information & Evidence to Support Indicator Scoring

26

Indicator Design
Each indicator provides an objective measure of current, not planned, performance through 1-4 dimensions Measured on a scale of 0-4 Translated into a single indicator score
Measurement Scale
Accountability for Government performance SAI enables continual improvement LEVEL 4

Managed Level
LEVEL 3

Established Level
(Follows ISSAIs)
LEVEL 2

Accountability for use of public funds

Development Level
LEVEL 1

Executive agents held to account

Founding Level
LEVEL 0

Each level provides a basis for accountability

Activity not Established or does not Function

27

Quality Assurance

Quality assurance crucial for objectivity and credibility of assessments Enables a single assessment to meet different stakeholder needs Most appropriate approach depends on purpose of the assessment Good practice:
SAIs select QA process to meet the assessment purpose QA process is transparently disclosed in the performance report Separation between those undertaking the assessment, and those performing QA

Quality Assurance Statement Who has prepared the assessment? Who has provided quality assurance of the assessment? What were their quality assurance responsibilities?

QA should cover:
Is the report factually correct? Has the SAI PMF tool been applied correctly? Is there sufficient evidence to justify the indicator scores?

How have matters raised in the quality assurance process been addressed in the final report?
28

4. Looking ahead: Responsibilities and timelines

29

Potential risks related to having a SAI PMF

Revelations of performance weaknesses It will be up to the SAI to decide if results are to be shared Openness and commitment to reform likely to encourage external partners to support capacity development May help draw attention to constraints outside the SAIs control

Possible negative effect of scoring: Use of performance league tables Intention is not to aggregate scores to the level of the entire SAI as all indicators may not be of equal importance Scores partly linked to achievements of SAIs own mandate Narrative report provides context for scoring Scoring is already applied in INTOSAI (AFROSAI-E) Scoring very useful for measuring progress over time within an SAI
Profusion of Assessment Tools Risk that the SAI PMF becomes an additional tool Donors and SAIs continue to develop & use other tools
30

Identified Challenges in the Development of the SAI PMF (1)

Different SAI models, mandates and country context make it challenging to develop a global tool
ISSAIs and other established good practice is common benchmark Not measure at a too detailed level Assess against SAIs own mandate Different SAI models and regions represented in Task Team and Reference Group Pilots to identify challenges and address them Possible to answer N/A and to add local indicators

31

Identified Challenges in the Development of the SAI PMF (2)

Scoring / objective measurement not possible
Performance report will provide qualitative assessment and put scoring into context Scoring useful for measuring progress over time

Challenging timetable
Johannesburg accords look for tool to be presented at INCOSAI XXI, China, 2013 WGVBS looking to approve Exposure Draft for INCOSAI Global piloting and consultations will extend beyond 2013

32

SAI PMF Timetable

2011, Q3 2011, Q4 2012, Q1 2012, Q2 2012, Q3 2012, Q4 2013, Q1 2013, Q2 2013, Q3 2013, Q4

WGVBS: Set Direction

Mapping Existing SAI Measurement Tools

WGVBS: Approve way forward

Concept Note for SAI PMF

Develop SAI PMF Exposure Draft

Develop SAI PMF Version 2.0

Phase 1 Pilots in 3 SAIs & Comments from Stakeholders

WGVBS: Approve Exposure Draft

Phase 2 Piloting and Pre-Consultation (ongoing into 2014 & 2015)

Exposure Draft at INCOSAI 33

Promotion and Awareness Raising of SAI PMF Training on Use of SAI PMF Assessments

Implementation and Maintenance (proposed)

INTOSAI Donor Community Community INTOSAI-Donor Secretariat INTOSAI-Donor Secretariat INTOSAI-Donor Secretariat INTOSAI-Donor Secretariat INTOSAI-Donor Secretariat

Present SAI PMF at INTOSAI and other stakeholder events

Development and refinement of training material Training of trainers within INTOSAI regional bodies Training of SAI staff Approved Trainers INTOSAI Regional Bodies Training of other stakeholders (donors, consultants, CSOs) Approved Trainers

Provide Guidance and Support to Conduct of SAI PMF Assessments

Host and maintain guidance materials on internet
INTOSAI-Donor Secretariat

Monitor Progress on Use of SAI PMF and Maintain Consolidated Indicator Data
Maintain and update global summary of the status of SAI PMF assessments, to enhance INTOSAI-Donor Secretariat coordination Database of SAIs Task Force Manage an e-library of published SAI PMF assessments Maintain database on (published) SAI PMF indicator scores for benchmarking, evaluation and Database of SAIs Task Force research

Technical Development and Refinement of SAI PMF

Decision to modify or update the SAI PMF (e.g. as ISSAI framework develops) Develop and refine the SAI PMF
INTOSAI WGVBS SAI PMF Task Team 34

6. Further Information
www.idi.no
Mapping of Tools for Assessing Performance of Supreme Audit Institutions IDI, May 2012 Supreme Audit Institutions Performance Measurement Framework Concept Note IDI, May 2012

E-mail: INTOSAI.Donor.Secretariat@idi.no

35