Audit Proposal Manual

Audit Proposal Manual

June 2005

Netherlands Court of Audit, Lange Voorhout 8, Postbus 20015, 2500 EA The Hague, The Netherlands

Audit Proposal Manual

Contents

1 2 3 4 4.1 4.2 4.3 4.4 5 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 6 6.1 6.2 6.3 7 7.1 7.2 8 8.1 8.2 9

Introduction Roles and responsibilities Framework for audit proposals Importance of the audit Framework Subject Relevance Auditees response Details of the audit Objective and problem Terms and standards Audit questions Audit design Bridges Impact Expected results and form of publication Informing and consulting the Board Timetable and cost Organisation Cost Completion Risks and quality assurance Risks Quality assurance Proposal Positive recommendation Negative recommendation Annexes to the audit proposal

2 3 4 5 5 6 7 7 8 8 8 9 9 10 10 11 11 13 13 13 14 15 15 16 18 18 18 20

Audit Proposal Manual

Introduction

This manual contains guidelines for drawing up audit proposals, and is also a useful tool for assessing audit proposals. It is intended for use primarily by project managers and auditors working for the Netherlands Court of Audit. All audits performed by the Court of Audit begin with the formulation of an audit proposal, on the basis of which the Board decides whether or not to perform the audit in question. The main purpose of an audit proposal is therefore to provide information enabling the following question to be answered: Should the Court of Audit invest resources in the proposed audit during the specified period? The subjects of audit proposals are either described in the Courts work programme or, in the case of audit proposals submitted between the publication of work programmes, adopted during a Board meeting. An audit proposal provides fairly detailed information on the structure and cost of the audit in question, its expected results and the potential risks. This manual sets out a framework for the formulation of audit proposals and lists the type of information they should contain. In principle, the manual covers all types of audit. Please note that the general requirements set out in this manual relate to audit proposals as submitted to the Board. For managerial or other reasons, directors may request additional information. As a rule, this information is not included in the audit proposal. Audit proposals are not only intended for internal use, but also serve an external purpose. Once the Board has approved an audit proposal, the auditee is informed of certain matters referred to in chapters 4 (with the exception of section 4.4), 5 (Details of the audit) and 6 (Timetable and cost) of this manual. The manual is divided into the following chapters. Chapter 2 describes the roles and responsibilities during the formulation of an audit proposal. Chapter 3 presents a framework for audit proposals. Chapters 4 to 9 form the actual framework for audit proposals. Each section indicates the type of information required under each heading.

Audit Proposal Manual

Roles and responsibilities

The project manager is responsible for preparing an audit proposal, including the agreements made with the bureau head on the human and material resources to be deployed, and on quality assurance. The audit proposal serves as a contract between the project manager and the director. The bureau heads are keen to encourage the use of the standards database. The directors are responsible for checking whether the standards database was used when preparing the audit proposal. All audit proposals are first presented to and discussed by the management team (MT). The director decides whether to incorporate the MTs recommendations in the final audit proposal. It is the director who submits the audit proposal to the Board and enters into a contract with the Board. The Board is responsible for approving the audit proposal.

Audit Proposal Manual

Framework for audit proposals

To decide whether an audit should be performed, the Board requires a general understanding of: the importance of the subject and the expected results; the details of the audit and how the quality of the results is to be assured; the audit cost and timetable, including the associated uncertainties; specific risks relating to the audit.

These four aspects make up the framework for every audit proposal. In other words, the core of an audit proposal consists of the following four sections: 1. 2. 3. 4. Importance of the audit Details of the audit Timetable and cost Risks and quality assurance.

The information that these four core sections should contain is discussed in chapters 4 to 7 below. Chapter 8 goes on to describe the final proposal submitted to the Board (with a recommendation on whether or not to perform the audit in question). Lastly, chapter 9 lists the annexes that should be included with an audit proposal. The basic rule is that audit proposals should be as concise as possible.

Audit Proposal Manual

Importance of the audit

This chapter describes the type of information you should include in the section of the proposal entitled Importance of the audit. The sections in this chapter correspond with the relevant section headings to be included in the audit proposal.

4.1

Framework
The audit proposal should start with a reference to the relevant part of the strategy (please indicate which output target is relevant in this connection), the work programme and any other relevant documents. A copy of the format from the work programme should be enclosed. If the results the audit is expected to generate do not correspond with those specified in the work programme, you should explain why. A proposal for an annual regularity audit (covering the central government annual financial report) should refer to the instructions on risk analysis and the audit proposal published each year at the start of a new annual financial report cycle. Objection audits, follow-up audits and new (i.e. unforeseen) audits are subject to a number of supplementary requirements: If the audit in question is an objection audit, the audit proposal should refer to the collection of reports on the central government annual financial report, which contains a list of objection audits planned for the coming year. You should also refer to the decision to perform the objection audit in question. If the audit in question is a follow-up audit, you should specify the previous audit and list its main conclusions and recommendations. If the subject of the audit was not included in the work programme, you should explain why the proposal is nevertheless being made. Requested audits are examples of new subjects. You should also comply with the Regulations on Audits on Request (HANDAR) in this case.

Audit Proposal Manual

4.2

Subject
In describing the subject, you should discuss the current situation, in so far as it is relevant. Has new legislation been introduced? Has the ministry recently published policy documents and/or evaluations? Are there any major social trends that need to be mentioned? Where necessary, you can also indicate the relationship with other policy areas (and/or organisations). Please note, though, that you only need to specify the main points. There is no need to include detailed background information. Name the ministers and state secretaries concerned and describe their field of responsibility (make use, if necessary, of the manual on ministerial responsibility for this purpose). Depending on the period covered by the audit, it may also be worth naming their predecessors. You should also specify the policy departments covered by the audit. Section 91 of the Government Accounts Act 2001 states that the Netherlands Court of Audit is entitled to perform audits of certain non-state organisations. Where relevant, you should specify which non-state organisations are to be included in the audit, mentioning the relevant subsection of section 91 in doing so. You should describe the responsibilities of these organisations boards. The relevant manual gives detailed information on the specific powers and procedures involved in performing a section 91 audit. You should also indicate whether previous audits of annual government reports have contained comments about regularity, financial management, the reliability of policy information or policy accountability in general. You should take account not just of the previous years report, but of past reports over a period of several years. If necessary, get in touch with the project manager at the ministry in question responsible for the annual government report. This type of information should also be included in proposals for audits of outputs and outcomes. Proposals for regularity audits should also contain comments previously made about efficiency and effectiveness. Although such comments may be taken from audits performed by the Court of Audit, they may also be taken from evaluations performed by or on behalf of the ministry in question. The Civil Service Evaluation Review may be consulted for this purpose. The importance of the audit subject may also be illustrated by referring to trends in spending (such as the pattern of annual cash spending and end-ofyear spending sprees).

Audit Proposal Manual

4.3

Relevance
You should summarise the main arguments for carrying out the audit. The audits relevance depends on the degree to which it will help achieve the Courts strategic objectives. The section on the audits importance should refer to the Courts work programme and the objectives set out in it. Your proposal should contain an explicit discussion of the criteria used in selecting the audit subject, in accordance with the Courts strategy (page 23). These criteria are: the social value of the Courts activities, complex policy chains, the existence of risks, unique powers, structural interventions and financial value. As far as the financial value is concerned, you should also specify the budget articles that are at issue. If the auditee is a legal person with statutory tasks that is funded by premiums or tariffs, these money flows should also be described. You should also state whether there is any link between your audit proposal and the domain plans and development plans. In the case of objection audits, you should refer to the arguments put forward by the Board in deciding to perform an objection audit (see the relevant minutes).

4.4

Auditees response
In this section of the audit proposal, you should briefly describe the auditees response to the audit. The points concerned are also mentioned in the Manual on Tripartite Consultations. You should include a description of the auditees response to the audit standards. As the idea is to reach agreement with the auditees on the standards, any disagreements should be explicitly mentioned in the audit proposal.

Audit Proposal Manual

Details of the audit

This chapter presents the information you should give in the section entitled Details of the audit. Again, the sections in this chapter correspond with the relevant section headings to be included in the audit proposal. There are a number of aspects about which the auditees themselves should be notified; these are described in sections 5.1 to 5.4.

5.1

Objective and problem

The objective is a statement of the audits intended effect. It is based on the aim of the audit as defined in the work programme, since this information provided the basis for deciding to invest time in formulating an audit proposal. Do not hesitate to consult the auditees and other stakeholders when formulating the objective and the problem to be addressed. Use the tripartite consultations to make agreements on whether other stakeholders might be consulted and, if so, which ones. The problem to be addressed is defined with reference to the objective; the two must of course be consistent with each other. Whereas the objective is described in terms of the desired effect, the problem is defined as the question that needs to be answered in order to achieve that effect. In other words, defining the problem is not simply a matter of turning the objective round and adding a question mark. The objective and problem form the core of the audit and must therefore be formulated very carefully.

5.2

Terms and standards

Every objective and problem includes references to certain terms and standards, e.g. of particular policy instruments (such as a grant) or the orderly and auditable standard. The audit proposal should clarify the meaning of these terms and standards. Sources of definitions include HANDAR, parliamentary paper (budgets),

Audit Proposal Manual

ministerial documents, past audits performed by the Court of Audit, the professional literature and the standards database. You should consult the standards database every time you prepare an audit proposal. The proposal should include a brief report on the results of this exercise. The essential point is whether the standards on which the proposed audit is to be based are consistent with those previously used by the Court of Audit, or whether the audit standards have since been refined, extended or otherwise altered. This section of the audit proposal should also describe the role played by the context. In other words, you should specify the standards and weighting factors on which the final judgement is to be based. Experience shows that specifying standards and determining weighting factors is a process that can and should take up a great deal of time. The more clearly the standards are formulated, the more smoothly the audit will go.

5.3

Audit questions
Audit questions are the questions that must be answered in order to solve the problem defined. It is important to formulate a series of audit questions that will paint a full picture and will provide sufficient information for you to form a judgement and draw conclusions. You can make use of an AIQ chart 1 to operationalise the problem. Past experience shows that AIQ charts often help to clarify the situation for the client (i.e. the director or the Board). If you decide to use such a chart, it should be included in the annexe to the audit proposal.

5.4

Audit design
The audit design (i.e. the audit methods and techniques used) largely determines the quality of the solution to the problem defined, as well as the cost of the audit. You should therefore exercise great care in choosing the audit design.

Data collection In order to provide information on the quality of the audit design, you should briefly outline in this section the methods and techniques you intend to use for collecting and analysing data. You should also explain in general terms why you

AIQ stands for aspects, indicators and questions.

Audit Proposal Manual

have decided to use these methods. Obviously, you should check in advance whether the methods are indeed feasible. If, for example, you have opted to perform a file study, you should ascertain beforehand whether the files are available and what they contain. If you are intending to perform a quantitative performance audit, 2 you should consult the checklist for quantitative audits in HANDAR (which is part of the Performance Audits Manual).

10

Personal data You should also indicate in this section whether you intend to compile databases of personal data during the audit, or whether you will be using thirdparty databases. If so, the Court of Audits standard regulations on personal data apply, and you will need to comply with the relevant regulations (see the Manual on Personal Databases for Audit Purposes).

Multi-step plan for second-pillar audits If you are intending to perform a second-pillar audit, you should follow the new multi-step plan (see the Courts intranet site).

5.5

Bridges
The strategy defines four bridges, i.e. ministerial responsibility, EU aspects, enforcement and operational management (i.e. administrative burden, ethical standards and IT). You should explain in this section whether the bridges have a bearing on the subject of the audit and, if so, how you intend to deal with them. The Courts government-wide bureaus have drawn up a checklist to help you decide which particular bridges to include in the audit. You should make use of this checklist in arriving at a decision.

5.6

Impact
Audits are intended not just as a means of assessment, but also as a way of actually improving matters. In order to achieve our mission, audits must make an impact. You should indicate under this heading what you intend to do during the course of the audit to ensure that it has an impact on the auditee. You can use the Effective Court of Audit menu on the intranet for this purpose. The menu also contains guidelines for ensuring that you adopt the most effective tools. This involves analysing the policy context, the actors and the interests at stake.

These audits use statistical methods (mainly variants of multivariate regression analysis) and large

databases (over 50 observations) to assess policy efficiency and effectiveness.

Audit Proposal Manual

11

You should also indicate how you intend to make an impact during the reporting and follow-up stages.

5.7

Expected results and form of publication

In this section you should describe the message the audit is expected to generate for the authorities and the intended effect of publication. After all, the relevance of the message will affect the Boards decision on whether or not to approve the audit proposal. You should also indicate the appropriate form of publication. The form you choose depends on the expected results and the intended effects. Please note that this is only an indication, and may therefore be changed in the final report proposal.

5.8

Informing and consulting the Board

In this section you should indicate how often consultations should be held with the Board. This should be raised in the tripartite consultations. The following models should be followed: Model 1: audits of annual government reports In addition to the normal tripartite and quadripartite consultations, the Board should be informed each month how the audit of the annual reports for a particular year is progressing, beginning in September of the preceding year. The principle of reporting by exception should preferably be followed; audit subjects that are scrapped due to time constraints or for substantive reasons should be mentioned. Model 2: audits requested by parliament Audits of this type require intensive coordination. This is often due to serious time constraints or the politically charged nature of the request. In addition to the normal tripartite and quadripartite consultations, the Board member acting as rapporteur, the director and the project manager agree at the start of the audit how the rapporteur should be kept informed about the audit and how frequently.

Audit Proposal Manual

Model 3: all other audits The normal tripartite and quadripartite consultations will not always be sufficient. If more intensive coordination is required, this should be agreed in the tripartite consultations. For that purpose, an estimate should be made of the audits risk profile. More specifically, this includes: the political sensitivity of an issue; the risk of ministers concerning themselves with the audit at too early a stage; the time pressure.

12

Audit Proposal Manual

Timetable and cost

13

This chapter presents the information you should give in the section entitled Timetable and cost. Again, the sections in this chapter correspond with the relevant section headings to be included in the audit proposal. There are a number of aspects about which the auditees themselves should be notified; these are described in sections 6.1 and 6.3.

6.1

Organisation
You should name the project manager or managers who will be in charge of the audit, as well as the auditors who will be working on it. You should mention the Artemis project number. You should also say what type of skills are required for the audit, including both those already available at the Court of Audit and those to be obtained from third parties. If third parties are to be engaged, you should explain what arrangements you intend to make with them and state what aspects of the audit these third parties will be working on.

6.2

Cost
Your estimate of the cost should include both the time to be spent on the audit (in days) and the cost of engaging external experts, for example. Your estimate should be based on the entire implementation stage (i.e. up to and including the production of a report of findings and a report proposal). You should include an itemised list of the number of days to be spent on the audit by: the project manager; auditors (including senior auditors); management assistants; staff from other Court of Audit departments (statisticians, communication experts, lawyers, etc.). You should contact the staff involved in order to calculate the amount of time they are likely to be spending on the audit. Once you have made a timetable for the audit, it can be entered in Artemis as provisional. After the Board has approved the proposal, and the timetable has

Audit Proposal Manual

been adjusted as desired or required, the first version of the timetable may be frozen as version 1. If you need funding for the audit, you should say so here. You should state the amount of money you are intending to spend on foreign travel, seminars, external experts and/or the cost of travel and accommodation in the Netherlands.

14

6.3

Completion
The audit proposal should contain a copy of the project timetable as recorded in Artemis, showing the various stages of the audit and the milestones you have set. It is absolutely vital to take account of the commitments of all those involved when planning the audit. You can solve this problem with the aid of the institutional calendar. When you draw up the timetable section in your audit proposal, you should get in touch with the chair of the institutional calendar working group to set a provisional publication date and the relevant circulation periods. Once the audit proposal has been approved, you should ask your director to apply to the working group for a definite publication date. He should use an infringement form for this purpose. See the Institutional Calendar Manual for further information. Audits of annual government reports are subject to certain fixed time limits, and you should take these into account when allocating manpower resources in your own audit proposal. If you are intending for whatever reason to supply the Board with interim reports during the course of the audit, you should take these into account when planning the timetable for the audit.

Audit Proposal Manual

Risks and quality assurance

15

This chapter presents the information you should give in the section entitled Risks and quality assurance. Again, the sections in this chapter correspond with the relevant section headings to be included in the audit proposal.

7.1

Risks
Every audit may entail certain risks that can prevent it from achieving its objectives. You should therefore describe in this section the risks (both internal and external) associated with the audits critical success factors. The latter are: usefulness; argumentation; cost and timetable.

Usefulness The Court of Audits strategy for 2004-2009 sets out its views on influence. The Court is aware that, whilst it is capable of exerting influence, it is not up to the Court to initiate and implement changes. Parliament and the government are responsible for setting the pace in this respect, followed by the ministries and other executive bodies. It is the Courts job to disseminate its findings, conclusions, recommendations, knowledge and information in such a way that parliament and the government, as well as the ministries and other interested parties, can make the best possible use of them. This view of the Courts role cannot be divorced from the concept of usefulness. An audit may be regarded as useful if its final results enable one or more of the target groups to implement certain changes. This is guaranteed if the audit relates to a subject in which the target groups are interested or clearly ought to be interested, or if the standards are presented in a cogent manner to the target groups. Examples of risks in relation to usefulness are subjects where it is difficult to trace specific errors in the auditees actions. If there seems to be a lack of support for the standards, publication and communication will have to be particularly effective to ensure that the audit leads to action.

Audit Proposal Manual

16

Argumentation Audits need to generate data that are valid and reliable. After all, if the target groups have reason to question the strength of the evidence, all the work may have been in vain. For example, the quality of the information available (e.g. in files) must be sufficient to answer the audit questions. Cost and timetable The need to control costs and meet deadlines may also entail certain risks. If, for example, certain non-state organisations (as defined in section 91 of the Government Accounts Act 2001) refuse to cooperate and/or if the only means of collecting certain information is by using expensive methods, this may make it difficult to control costs and meet deadlines. As well as being crucial in terms of cost and completion time, these aspects may also adversely affect the audits external impact (e.g. publication after a crucial meeting of the House of Representatives). If you believe that the audit entails certain risks that could adversely affect one of the above-mentioned factors, you should briefly state this here.

7.2

Quality assurance
The main groups of quality requirements relate to: strategic product quality (especially usefulness); technical product quality (especially argumentation); process quality (especially cost and timetable).

The audit proposal should specify the types of material quality assurance you intend to use to ensure that both the product and the process meet the quality requirements. Indicate for each main group of quality requirements which mechanisms you intend to use and what you intend to achieve with them. The Manual on Quality Assurance should give you a picture of the mechanisms involved. It gives specific examples of quality assurance mechanisms for each main group of quality requirements. At this stage you should consider whether external experts need to be called in. This may involve experts in a certain policy area or in the field of audit methods and techniques. It can take various forms, such as a workshop, a permanent focus group or the hiring of specific expertise (see also the Effective Court of Audit menu on the intranet).

Audit Proposal Manual

For each audit, a peer reviewer (of senior auditor rank or above) from outside the bureau should be invited to assess the audit proposal and the draft report of findings. The audit proposal indicates who this is. The reviewer and the project manager should jointly submit the findings together with the audit proposal or draft report of findings to the relevant director, so that he can take the findings into account when discussing the products with the audit team. In the case of an audit proposal, the peer review always looks at: the consistency between problem definition and audit questions; the audit approach; quality assurance; the impact strategy.

17

In the case of a draft report of findings, it looks at: the arguments underlying the findings; the consistency between findings, conclusions and recommendations.

The audit risks should be taken into account when deciding on the type of quality assurance you intend to use. For example, more safeguards need to be built into a complex audit involving major risks than into a more routine audit. Interim Board briefings and external focus groups can act as safeguards for complex audits. The director has a key role to play in the quality assurance process, and you should bear this in mind when drawing up a timetable for the audit. Make sure that he is given enough time to read the audit report; if necessary, make agreements on this matter. After the audit has been concluded, you should evaluate the effectiveness of the quality assurance mechanisms as part of the self-assessment.

Audit Proposal Manual

Proposal

18

The audit proposal culminates in the submission of a proposal to the Board. This is the subject matter of the section headed Proposal. Although we have assumed that the proposal will include a recommendation that it should be approved, the preparatory stage may in fact culminate in a negative recommendation. This chapter presents the information required in the event of either a positive recommendation (i.e. to implement the proposal) or a negative recommendation (i.e. to reject the proposal). The information to be included applies to all audits.

8.1

Positive recommendation
Making a positive recommendation basically means asking the Board to approve the activities specified in the proposal and the resources required to achieve the intended results. In the interests of clarity, you should give a brief summary (about half a page) of the following: 1. 2. 3. 4. the audit subject; the expected results (the message for the authorities the audit is expected to generate); the cost and duration of the audit; the risks (political and administrative, and in relation to timing).

Under point 1 you should describe the audit as given in the work programme (including the project number) and the intended results.

8.2

Negative recommendation
Even if you advise against performing the audit in question, the Board must still take a decision, as the audit is part of the approved work programme. This means that different information is required, as there is no point spending a great deal of time describing the details of the audit, for example. A negative recommendation should therefore contain information on the following aspects:

Audit Proposal Manual

1. 2. 3. 4.

the intended subject; the reasons for not performing the audit; the lessons to be learned; the proposal.

19

The information given under the first point consists of a description of the audit as given in the work programme (including the project number) and the results it was intended to achieve. You should then briefly state your reasons for not performing the audit despite its inclusion in the work programme. For instance, the ministry might recently have carried out an evaluation of its own to which you feel that the Court of Audit has little to add. You should then list any lessons that have been learned, before concluding with your recommendation to the Board not to proceed with the audit.

Audit Proposal Manual

Annexes to the audit proposal

20

Every audit proposal should include the project format from the work programme as an annexe. In addition, the proposal should include at least the following three annexes: notification of auditees (see section 9.1); report on preparatory stage (see section 9.2); report on tripartite consultations (see section 9.3).

The inclusion of all the annexes referred to above is mandatory. In the case of audits that, according to the work programme, relate to the second pillar of the strategy, a completed checklist for assessing products relating to the second pillar of the strategy should be enclosed (see section 9.4). If you have drawn up an AIQ chart, please also enclose this as an annexe.

9.1

Notification of auditees
The first annexe should state when the auditee or auditees are to be notified of the proposed audit, stating the ranks of the staff who should be informed. The auditees should in any event be informed of the following elements of the audit proposal: the subject of the audit; the problem; the relevant standards and weighting factors; the data collection method; the timetable.

If the Court of Audit will be using its powers under section 91 of the Government Accounts Act 2001, the relevant details should be included (see the Manual on Section 91 of the Government Accounts Act 2001).

Audit Proposal Manual

9.2

Report on preparatory stage

The second annexe should consist of the following items: a statement of the amount of time spent on the preparatory stage (in days) and the total duration; a description of the quality assurance mechanisms used during the preparatory stage; the problems encountered and the lessons learned.

21

The information given under the first point consists of a statement of the amount of time spent on the preparatory stage, i.e. the number of days planned and the number of days actually spent. If there is a discrepancy between the two figures, please explain why. The same applies to the total duration of the preparatory stage: the information should include both the estimate made beforehand and the time actually taken, with an explanation of any discrepancy. The second item is a description of the quality assurance mechanisms used during the preparatory stage. As a third point, you should list any problems you may have encountered during the preparatory stage. Where relevant, you should also make recommendations for solving these problems. These recommendations should be as specific as possible.

9.3

Report on tripartite consultations

You should enclose a report on the tripartite consultations with the audit proposal.

9.4

Checklist for assessing products relating to the second pillar of the strategy
In the case of audits that, according to the work programme, relate to the second pillar of the strategy, a completed checklist should be enclosed as an annexe. It should make clear whether the audit can help to improve the link between policy and practice, and if so, to what extent. The checklist can be found on the performance audit intranet site, which also presents detailed instructions on filling it in.