ECM 523: CONSTRUCTION

PROJECT MANAGEMENT
Chapter 7.0 :
Project Evaluation and Auditing
Presented by: Suman Mishra
MSc. in Civil Engineering
University of Calgary, Canada
 Project Evaluation
• Evaluation is an objective and systematic judgmental
process for determining relevance, efficiency,
effectiveness and impact of project performance.
• It is an assessment of project during implementation.
• Evaluation is done to improve project implementation
and to improve future project planning and decision
making.
Project evaluation is a systematic and objective
assessment of an ongoing or completed project. The aim
is to determine the relevance and level of achievement
of project objectives, effectiveness, efficiency, impact
and sustainability.
It focuses on the Scope, Quality, Schedule, Budget, Resources
and the Risk of Construction Project.
 Evaluation
• What? Evaluation is an assessment that refers to assessment of design,
implementation and results of completed or on-going project / program / policy.
• How? Evaluation should be systematic and with objectives. It should consider
relevance, fulfillment of objectives, developmental efficiency, effectiveness, impact
and sustainability.
• Why? Evaluation should provide credible and useful information to enable the
incorporation of lessons learned into the decision-making process.

Evaluation comprises three things:

Whether we are doing the right things
Whether we are doing it right
Whether there are better ways of doing it
 Need and Importance
• The objectives of project evaluation are :
• To verify whether the project implementation progress is as planned.
• To take corrective measures for deviations in performance.
• To ascertain that actual costs are within the budgets.
• To ensure that quality standards are being attained.
• To identify unexpected problem areas and manage them.
• To bring about overall improvement in project performance to achieve
project objectives.
 Evaluation and purpose
• Determine relevancy of the project and decide go and no-go with the project
• See efficiency and effectiveness of the project and improve project
implementation
• Observe project performance and correct deficiencies
• Observe impact of the project on beneficiaries
• Improve future project planning
• Lessons for future projects for better planning - a learning strategy
• Enhance decision making for future project
• An effective tool for project management
• Track project progress and take corrective action for deviations
• An objective look on the planned operations
 Project Evaluation Myth
Myth: a widely held but false belief or idea.
Myth # 1: Evaluations only take place in the middle or at the end of a project
• The truth is that an evaluation can take place at any point before, during or after the
project cycle.
• Formative evaluations take place in the lead up to the project, as well as during the
project in order to improve the project design as it is being implemented (continual
improvement).
• While summative evaluations occurs at the end of a program cycle and provides an
overall description of program effectiveness.
• In recent years real time evaluation has emerged.

Just as the name suggests, Real time evaluation (RTE) is an approach with 'the primary objective is to provide
feedback in a participatory way in real time (i.e. during the evaluation fieldwork) to those executing and managing
the programme'
Myth #2: The types of evaluation are limited
The truth is that there are different types of evaluation available depending on
context and the available information and resources.
• Formative evaluation
• needs assessment determines who needs the program, how great the need is,
and what might work to meet the need
• evaluability assessment determines whether an evaluation is feasible and how
stakeholders can help shape its usefulness
• structured conceptualization helps stakeholders define the program or
technology, the target population, and the possible outcomes
• implementation evaluation monitors the precision/accuracy of the program or
technology delivery
• process evaluation investigates the process of delivering the program or
technology, including alternative delivery procedures
• Summative evaluation
• outcome evaluations investigate whether the program or technology caused
demonstrable effects on specifically defined target outcomes
• impact evaluation is broader and assesses the overall or net effects -- intended or
unintended -- of the program or technology as a whole
• cost-effectiveness and cost-benefit analysis address questions of efficiency by
standardizing outcomes in terms of their dollar / rupees costs and values
• secondary analysis re-examines existing data to address new questions or use
methods not previously employed
• meta-analysis integrates the outcome estimates from multiple studies to arrive at
an overall or summary judgment on an evaluation question
Myth # 3: Only the success stories from an evaluation have value
• Oftentimes the motivation behind an evaluation is a desire to show that a project
was a success.
• This is to prove that public/private funds or donor moneys were well spent and
targeted people / communities were well served. However, most evaluations show
mixed results and the 'brilliant failures' of a project are just as useful as the
successes.
• One example comes from a health, education and water program in Mali. The
evaluation findings revealed that failure to establish terms of engagement led to
divisions and miscommunication affecting the program's implementation.
• As a result of this finding, the different stakeholders developed a Memorandum of
Understanding.
• The moral of the story is that a 'failure' that is unearthed by an evaluation can (and
should be) be used for learning and program improvement.
6 Ways of Measuring Project Success
Measuring the success of a project once it’s brought to completion is a
valuable practice. It provides a learning opportunity for future undertakings,
and, the opportunity to assess the true effectiveness of the project.
1. Scope
2. Schedule
3. Budget
4. Team satisfaction
5. Customer satisfaction
6. Quality
 Project Success
“As Project Manager, juggling all of the
balls is important, but keeping your eye
on the right ball is the key to delivering
truly successful projects” –
Robert Goatham
• Tier 1 – The project was a success if it
delivers all or most of what it said it
would (the scope), regardless of
schedule or budget performance
• Tier 2 – The project was a success if it
delivers what it said it would, on
schedule and/or within the agreed
budget
• Tier 3 – The project was a success if it
delivers what it said it would, on
schedule, within the agreed budget
and to the expected quality standards
• Tier 4 – The project was a success it
delivers on all agreed project
objectives, be they scope, schedule,
budget, quality or outcomes based
(i.e. goals to be achieved or strategic
positions to be attained)
• Tier 5 – The project was a success if the
product produced by the project
creates significant net value for the
organization after the project is
completed.
Project Success criteria
Factors for Project Success and Failure ?
Since projects are temporary in nature, the success of the project should be
measured in terms of completing the project within the constraints of scope, time,
cost, quality, resources, and risk as approved between the client and contractor
(executing and construction bodies)

The project manager is responsible and accountable for setting realistic and
achievable boundaries for the project and to accomplish the project within the
approved baselines. The success of the project will depend on how well project time,
cost, scope and quality meet the requirements established at the beginning of the
project.
Success or Failure Factors in Relation to the Initial Project
Definition

• project scope (the extent of work required) is not clearly stated and understood;
• the technical requirements are vague;
• estimates of cost, timescale or benefits are too optimistic;
• risk assessment is incomplete or flawed;
• the intended project strategy is inappropriate;
• insufficient regard is paid to cash flows and the provision of funds;
• the interests and concerns of stakeholders are not taken into account;
Success or Failure Factors in Relation to the Initial Project
Definition

• undue regard is paid to the motivation and behavior of the people who will
execute the project;
• particularly in management change projects, insufficient thought is given to how
all the managers and workpeople affected by the project will adapt to the changes
expected of them;
• approval to proceed with the project is given for political, personal or intuitive
reasons without due consideration of the business plan.
 Types of evaluation based on time
Ex-ante evaluation - is performed before programme implementation and its
objective is to assess whether the planned intervention is accurate with regard to
needs as well as coherent with reference to planned aims and how they will be
implemented.
On-going evaluation - has a supplementary character for all above-mentioned
evaluation types and can be conducted independently. The on-going evaluation is
carried out throughout the period of implementation of an intervention; however it
can not be taken for monitoring, as it consists of the in-depth assessment of the
chosen problems that have appeared during other evaluations.
Mid-term evaluation is the evaluation performed towards the middle of the
implementation of an intervention. This evaluation critically considers the first
outputs and results, which enable assessing the quality of programme
implementation.
 Types of evaluation based on time
Thematic evaluations focus on the analysis of a selected part of the policy, and this
analysis is of the cross-sectional and/or comparative character.
Ex-post evaluation is the evaluation of an intervention after it has been completed.
It should be carried out not later than three years after the end of the
implementation period. The ex-post evaluation aims at examining long-lasting
effects of a programme and their sustainability.
Evaluator and Purpose
Role of Evaluator
• To track project progress and take corrective actions.
• To check the efficiency and effectiveness of the project and improve project
implementation.
• To suggest improvement for future planning.
Purpose
• To observe project performance
• To observe impact of project on beneficiaries
• Lessons for future project for better planning- a learning strategy
• Enhance decision making for future project
Method of Evaluation
Data collection methods
The most popular data collection methods for the evaluation needs are:
document analysis,
individual interviews,
questionnaires,
focus groups,
observation and group techniques
 Project Audit
-A formal review of any aspect of a project
- An audit is systematic, independent, documented
assessment using standards and set criteria.

Project auditing can be defined as the

process of detailed inspection of the
management of a project, its
methodology, its techniques, its
procedures, its documents, its properties,
its budgets, its expenses and its level of
completion.
A project audit is a key step in the process
of closing a project. This audit evaluates
the total project processes and outcomes.
 Purpose and Goals
• The main goal of an audit is to
-inspect and evaluate the current state of project realization,
-find out to what extent it complied with defined criteria for project success
and
-identify opportunities for improving the project realization and
management.
• This inspection can be performed after the project completion or after the end of
one project realization stage, or it can be performed during the project
realization.
Benefit of Project Audits
Project audits can help identify when a project is about to go off-course. In
addition, a project audit can provide the following benefits:
• Improve project performance.
• Increase customer and stakeholder satisfaction.
• Save costs.
• Control scope and avoid scope creep.
• Provide early problem diagnostics.
• Clarify performance/cost/schedule relationships.
• Identify future opportunities for improvement.
• Evaluate performance of the project team.
• Inform client of project status/prospects.
• Reconfirm feasibility of/commitment to project.
 Need of Project Audit
• Audits can be triggered when:
• Situations come about which do not give a clear picture of the happenings on a
project.
• A doubt arises on whether the project deliverables are being achieved.
• Technical gap seem to be apparent.
• The progress on the project is vastly in variance with the project plan.
• There are doubts about the organization of the project and its ability to produce
results.
• A need for technical changes is felt, to see whether they conform to best
practices in the industry.
• Needs to pinpoint defects and suggest the best methods to move forward to
proper implementation.
• Projects that have a number of milestones and deliverables are best audited at
regular intervals.
• Existing documents and data would have to be studied.
• Key team members would have to be interviewed to understand the processes
which they have followed for execution of the project.
• When audit completed reports would have to be prepared which would give a
summary, the background that initiated, and risks identified.
• It must also lay down clearly the effects of the risks perceived and their likely
outcome on the project in case they are ignored.
Components of Audit Planning Process
1. Research the Audit Area
2. Maintain Open Communications Throughout the Planning Process
3. Conduct Process Walk-Throughs
4. Map Risks to the Organization, Process, or Function
5. Obtain Data Prior to Fieldwork
Types of Audit
Basically two types of auditing
• External auditing is the verification process involving the independent
appraisal of financial accounts and statements.

• Internal audit is performed by the internal auditing staff of the

organization itself to ensure that its assets are properly safeguarded and
its financial records reliably kept.
 Types of Project Audit
• Financial – Financial audits typically involve a focus on financial controls as they
relate to reporting. These audits focus on accounting controls present in the
general ledger or sub-ledger systems. Financial statement auditing is the focus of
our external auditors. Internal Audit will complement the work they perform
based on an agreed plan.
• Operational – Operational audits focus on the review and assessment of a
business process. The activities of the business process may result in a direct or
indirect financial impact to the organization such as the collection of student
tuitions or patient account balances. Internal Audit primarily focuses on
operational audits but can extend the scope to include accounting procedures
that can impact financial reporting.
• Compliance – Compliance audits review the level of compliance with internal
policies or external regulatory requirements.
 Types of Project Audit
• Investigate Audit- This is an audit that takes place as a result of a report of
unusual suspicious activity on the part of an individual or department. It usually
focus on specific part of work of a department or individual.
• Follow-up Audit- These are audits conducted approximately six months after an
internal or external audit report has been issued. They are designed to evaluate the
corrective actions that has been taken on the audit issues reported to the original
report.
• Integrated Audits – Integrated audits look at controls that address financial,
operational, compliance and information systems risks. These audits are typically
centered on a business cycle or a specific part of a cycle or process.
 Purpose of audits will be:
• Determining
• What is going right and why?
• What is going wrong and why?

• Identifying
• Forces and factors that have prevented or may prevent achieving cost,
schedule, technical performance goal
• Evaluating
• Efficacy of existing project management strategy including organizational
support, policies, procedures, practices, techniques, guidelines, action plans,
funding patterns, human and other resources utilization
• Providing opportunity
• To exchange ideas, information, problem, solutions and strategies
 Key components of project audit
• Engineering
• Manufacturing
• Finance and accounts
• Contracts
• Purchasing
• Marketing
• Human resources
• Organization and management
• Quality
• Reliability
• Tests
• Logistics
• Construction etc.
 Depth of Audit
Time and money limit the depth of an audit.
Audits are distracting to those working on the project.
A poor audit result will lower morale on the project.

Planning & Design Procurement/ Implementation Impact of Project

Purchase
•Need analysis & Demand •Reduction in operation cost or
analysis •Timely completion of improvement in operation.
•Technical Specification •Compliance with project •Increase market share
preparation and Financial Manual •Provision of liquidity •Improvement in Key
authorization •Independent evaluation damage in case of Performance Indicator (KPI)s
•Quality Assurance and of received proposal. delay in project •Increase customer loyalty or
Experience of Supplier completion. decrease customer churn
required in the project •Acceptance Test •Increase Revenue of the
Report company.
 Major task of Project Audit
• Identifying the need for evaluation and selecting the topics/themes to be
evaluated.
• To continue project implementation as planned or to re orient or in the
worst case to stop the project.
• Ensuring the production of a high quality evaluation report and of the
dissemination of evaluation findings and recommendations.
• Concerning the questions, whether or not, in future similar projects should
be initiated.
Life-cycle of Project Audit
• Project Audit Initiation
• Project Baseline Definition
• Establishing an Audit Database
• Preliminary Analysis of the Project
• Audit Report Preparation
• Project Audit Termination
 Auditor’s responsibility

• General
• The auditor must maintain independence in mental attitude in all matters
related to the audit.
• The auditor must have adequate technical training & proficiency to perform the
audit.
• The auditor must exercise due professional care during the performance of the
audit and the preparation of the report.
 Auditor’s responsibility

• Standards of Field Work

• The auditor must adequately plan the work and must properly supervise any
assistants.
• The auditor must obtain a sufficient understanding of the entity and its
environment, including its internal control, to assess the risk of material,
misstatement of the financial statements whether due to error or fraud, and to
design the nature, timing, and extent of further audit procedures.
• The auditor must obtain sufficient appropriate audit evidence by performing
audit procedures to afford a reasonable basis for an opinion regarding the
financial statements under audit.
• The auditor must assign qualified personal to the engagement.
• Standards of Reporting
• The auditor must state in the auditor's report whether the financial statements
are presented in accordance with generally accepted accounting principles.
• The auditor must identify in the auditor's report those circumstances in which
such principles have not been consistently observed in the current period in
relation to the preceding period.
• When the auditor determines that informative disclosures are not reasonably
adequate, the auditor must so state in the auditor's report.
• The auditor must either express an opinion regarding the financial statements,
taken as a whole, or state that an opinion cannot be expressed, in the auditor's
report.
• In all cases where an auditor's name is associated with financial statements, the
auditor should clearly indicate the character of the auditor's work, if any, and
the degree of responsibility the auditor is taking, in the auditor's report.
Step of Project Audit
• Critically review project documents
• Interview project team and stakeholders – to gain insights project
perceptions and affairs
• Participate in some project activities – to gain an appreciation of project
opportunities and problem
• Prepare and submit audit report
 Technical Audit
A systematic analysis of a company’s Research & Development activities conducted
by an outside expert can be called a technical audit. The audit examines all current R
& D activities, to make an honest appraisal of their potential and worthiness. After
an audit, priorities on all projects should be clear to all involved.

Technical Audit (TA) is an audit performed by an auditor, engineer or subject-matter

expert evaluates deficiencies or areas of improvement in a process, system or
proposal. Technical audit covers the technical aspects of the project implemented in
the organization.

In general, technical audit means regular monitoring and examination (continuous

audit) of construction works adherence to the sustainability, quality improvement,
cost effectiveness, schedule effectiveness.
 Technical Audit
• It is a tool to monitor the overall (work/activities) performance of the project.
• It help to maintain the construction work adherence to the appropriate quality,
sustainability of construction work, on schedule, in time, correction of technical
deficiencies and defects if any.
• It target to achieve the economy, efficiency, transparency, accountability, zero
tolerance on corruption and increment in value for money.
In Nepal, we don’t have long history on technical audit. However, in 2056 B.S
amendment on financial administration regulation (FAR) introduced the technical
audit and its management section at NPC (National Procurement Commission).

World bank had supported the ministry of physical planning and works for capacity
enhancement related activities and technical audit works from the public entity.

To avoid the conflict of interest in technical audit within the same public entity, it
has been transferred to the National vigilance Centre (NVC) by Shrawan 1st 2060
B.S. however, in 2063 B.S as per PPA this responsibility has been assigned to (Public
Procurement Monitoring Office) PPMO but PPMO is only responsible for
Procurement related issues rather than technical audit of construction works. So
then after the responsibility has been transferred to NVC.
Objectives of Technical auditing

• The technical operations are being performed as per requirement.

• Sound framework of control is in place to sufficiently mitigate the potential risk.
• The procured technical equipment is technically suitable for the purpose.
• Authority and responsibility for operating activities are assigned properly.
• Information system is adequate to provide assurance of operating activities
being performed properly.
 Scope of Technical Audit
• Cost effectiveness
• Timeliness
• Quality assurance
• Efficiency and effectiveness
• Outcome
• Value for money
Stages of technical audit
In general, technical audit shall be carried out it four different stages of the project
• Design phase
• Procurement phase
• Construction phase
• Operation phase
Design Phase
In this phase, technical audit regarding the effectiveness and relevancy of
conceptual study, feasibility study, work plan, data collection, site selection, cost
estimates etc. are examined.

Procurement Phase
In this phase, technical audit regarding procurement plan, procurement strategies,
selection of procurement method, procurement process, use of PPA, PPR, PPMO
guidelines, preparation of bidding documents, bid or proposal evaluation, contract
agreement, effectiveness in implementation of agreement, timeliness, quality,
transparency, competitive process, accountability etc. are examined.
Construction Phase
In this phase, technical audit regarding the use of technical norms, quality
standards, drawings and designs, cost estimates, work schedule, human resources
of contractor, material, fund mobilization, workmanship in terms of effectiveness in
cost , quality, time etc. are examined.

Operation Phase
In this phase, technical audit regarding sustainability, relevancy of design,
environmental study, effectiveness of cost and benefit analysis, effectiveness of
outcomes, operation cost, effectiveness of maintenance cost, participation for
people etc. are examined
Process of TA and Criteria to be an Technical Auditor
• In Nepal, NVC usually schedules and defines the programmes related to TA.
• The project implemented by GON and Donor Agencies are taken for TA in sample basis which shall
be programmed by NVC.
• NVC mobilizes the third party consultant for the TA work.
• NVC prepares the standing list of third party consultant.
• NVC sets the criteria for Technical Auditor as:
• Bachelor in engineering + 10 years experience in relevant field
• Registered in Nepal engineering Council

• NVC manage the training programme for short listed TA

• NVC open for the proposal (technical and financial) submission from short listed TA and select
Lowest evaluated substantially responsible proponent.
• NVC prepare the TOR and undergoes the agreement with the selected consultant
TA shall examine the following during technical auditing

• Effectiveness of cost estimates, timeliness, quality and result including following details
• Relevancy of design
• Possibility in getting proposed result
• Effectiveness and sustainability of project
• Relevancy of Conceptual study, feasibility study, procurement of services, works, goods,
quality standards
• Budgetary provisions and work plan
• Goal setting and objective setting
• Work schedule (start of activities and end of project)
• Progress of work schedule and use of work schedule
• Quality related issues and its handling
• Budgetary provision for operation and maintenance
TA shall follow following Methodologies for above mentioned TOR

• Questionnaire and Document reviewing

• Examination
• Reporting
• Immediate communication
 Financial Audit
• A financial statement audit is conducted to provide an opinion whether the financial
statements (the information being verified) are stated in accordance with specified
criteria.
• The primary purpose for financial audits is to give regulators, investors, directors,
and managers reasonable assurance that financial statements are accurate and
complete.
• In providing an opinion whether financial statements are fairly stated in accordance
with accounting standards, the auditor gathers evidence to determine whether the
statements contain material errors or other misstatements.
• The audit opinion is intended to provide reasonable assurance, but not absolute
assurance, that the financial statements are presented fairly, in all material respects,
and/or give a true and fair view in accordance with the financial reporting
framework.
Environmental Audit
• Environmental auditing is essentially an environmental management tool for
measuring the effects of certain activities on the environment against set criteria or
standards.
• Depending on the types of standards and the focus of the audit, there are different
types of environmental audit. Organizations of all kinds now recognize the
importance of environmental matters and accept that their environmental
performance will be scrutinized by a wide range of interested parties. Environmental
auditing is used to
• investigate
• understand
• Identify
These are used to help improve existing human activities, with the aim of reducing
the adverse effects of these activities on the environment.
Environmental Audit
• Environmental auditing should not be confused with environmental impact
assessment (EIA). Both environmental auditing and EIA are environmental
management tools, and both share some terminology, for example, 'impact',
'effect', and 'significant', but there are some important differences between the two.
• Environmental impact assessment is an anticipatory tool, that is, it takes place
before an action is carried out (ex ante). EIA therefore attempts to predict the
impact on the environment of a future action, and to provide this information to
those who make the decision on whether the project should be authorized. EIA is
also a legally mandated tool for many projects in most countries.
• Environmental auditing is carried out when a development is already in place, and is
used to check on existing practices, assessing the environmental effects of current
activities (ex post). Environmental auditing therefore provides a 'snap-shot' of
looking at what is happening at that point in time in an organization.
Social Audit
A social audit is a way of measuring, understanding, reporting and ultimately
improving an organization's social and ethical performance. A social audit helps to
narrow gaps between vision/goal and reality, between efficiency and
effectiveness.

Social auditing creates an impact upon governance. It values the voice of stakeholders,
including marginalized/poor groups whose voices are rarely heard. Social auditing is
taken up for the purpose of enhancing local governance, particularly for strengthening
accountability and transparency in local bodies.

The key difference between development and social audit is that a social audit focuses
on the neglected issue of social impacts, while a development audit has a broader focus
including environment and economic issues, such as the efficiency of a project or
program.
Social Audit
A social audit looks at factors such as company’s record of charitable giving, volunteer
activity, energy use, transparency, work environment and worker pay and benefits to
evaluate what kind of social impact a company is having in the location where it
operates.

Social audit is based on the principle that democratic local governance should be
carried out, as far as possible, with the consent and understanding of all concerned. It is
thus the process not an event.
Objectives of Social Audit

• Assessing the physical and financial gaps between needs and resources
available for local development.
• Creating awareness among beneficiaries and providers of local social and
productive services.
• Increasing efficacy and effectiveness of local development programmes.
• Scrutiny of various policy decisions, keeping in view stakeholder interests and
priorities, particularly of rural poor.
• Estimation of the opportunity cost for stakeholders of not getting timely
access to public services.
Advantages of social audit

• Encourages local democracy.

• Encourages community participation.
• Benefits disadvantaged groups.
• Promotes collective decision making and sharing responsibilities.
• Develops human resources and social capital
DIFFERENCE BETWEEN EVALUATION AND AUDIT

• Evaluation is a part of the Management Cycle(planning implementation

evaluation)
• Audit is independent from Management Cycle since it comes into scene
after the completion of Management Cycle.
• Evaluation talks about doing the right thing while audit talks about how the
things are done.
• Evaluation involves doing it right while audit involves managing it right.
• Evaluation is about sustainability while audit is about efficiency.
• Evaluation approach follows good practices while audit works against
norms.
Similarities

Though audit and evaluation differ in the skills and methods used and
practiced, there are also some basic similarities between the two. Both
methods analyze the data objectively and within defined standards and
protocols. Both are conducted by professionals and fall under committees.
These are mutually supportive and organizationally close. The results of both
should be posted and easily accessible.
THANK YOU