Beruflich Dokumente
Kultur Dokumente
RESULTS-BASED EVALUATION
OF
THE KABUL PROCUREMENT MARKETPLACE
(KPM)
PILOT PROJECT
EVALUATION REPORT
PREPARED BY
ASIF CHIDA
JANUARY 2008
1
Acknowledgements
This evaluation report forms part of the Canadian International Development Agency’s
(CIDA) Afghanistan Task Force’s (ATF) performance monitoring cycle. The evaluation
involved extensive interactions with CIDA and Peace Dividend Trust (PDT) officials in
Ottawa and Kabul, Afghanistan, whose valuable cooperation was indispensable and
greatly beneficial. The evaluator would like to express gratitude to Elizabeth Lee, Senior
Development Officer, ATF, CIDA, Ottawa and George Saibel, Michel Tache and Abdul
Qadir Danish, CIDA Kabul’s office. My special thanks to Scott Gilmore, Ainsley Butler
Ostojic and Keith Forsythe of PDT, Ottawa and Ms Shirine Pont, Barb Everdene, Nita
Colaco, Akbar Ahmadzai, Omid Delbary and Ahmad Asiyan of PDT field office in
Kabul.
During the mission, meetings were held with officials of government departments,
private sector and other stakeholders to whom I wish to express my thanks. The list of
persons met is provided in Appendix B.
Special thanks are to my PSI colleagues Pamela Branch and Hubert LeBlanc for giving
useful comments and support.
Value added from this evaluation of the PDT’s “The Kabul Procurement Marketplace”
will result from the sharing of what is learned from such investments, and the
development of more efficient and effective allocation strategies for consideration of
future CIDA projects.
Asif Chida
Senior Consultant
Project Services International
Ottawa, Canada
January 2008
1
TABLE OF CONTENTS
EXECUTIVE SUMMARY .......................................................................................................................... 1
1.0 INTRODUCTION.......................................................................................................................... 9
1.1 BACKGROUND .............................................................................................................................. 9
1.2 PROJECT SCOPE ............................................................................................................................ 9
1.3 PROJECT GOAL ............................................................................................................................. 9
1.4 PURPOSE OF THE REPORT ........................................................................................................... 10
1.5 KEY AUDIENCES ........................................................................................................................ 10
1.6 SCOPE OF CIDA’S SUPPORT TO PDT .......................................................................................... 10
2.0 PROJECT PROFILE .................................................................................................................. 11
2.1 KEY CONTEXTUAL FACTORS ..................................................................................................... 11
2.2 PRIVATE SECTOR DEVELOPMENT IN AFGHANISTAN................................................................... 11
2.3 DOING BUSINESS IN AFGHANISTAN............................................................................................ 12
2.4 LOCAL PROCUREMENT AND PRIVATE SECTOR DEVELOPMENT .................................................... 15
2.4.1 A brief overview of Donor funding in Afghanistan and local procurement.......................... 16
2.5 REASONS FOR LOW LOCAL PROCUREMENT:................................................................................ 17
3.0 EVALUATION PROFILE .......................................................................................................... 19
3.1 EVALUATION OBJECTIVE ........................................................................................................... 19
3.2 EVALUATION FRAMEWORK ........................................................................................................ 19
3.3 EVALUATION METHODOLOGY ................................................................................................... 19
3.4 DOCUMENTATION REVIEW ......................................................................................................... 19
3.5 FIELD VISIT ................................................................................................................................ 19
3.6 ADDITIONAL ANALYSIS ............................................................................................................. 20
3.7 LIMITATIONS .............................................................................................................................. 21
4.0 EVALUATION ANALYSIS AND FINDINGS.......................................................................... 22
4.1 EFFECTIVENESS .......................................................................................................................... 22
4.2 EFFICIENCY ................................................................................................................................ 26
4.3 COST EFFECTIVENESS ................................................................................................................ 26
4.4 RELEVANCE ............................................................................................................................... 27
4.5 SUSTAINABILITY ........................................................................................................................ 27
4.5.1 Local national partners ........................................................................................................ 27
4.1.1 Donor support....................................................................................................................... 28
4.1.2 Financial viability................................................................................................................. 28
4.1.3 Capacity Development.......................................................................................................... 28
4.1.4 Institutional Memory ............................................................................................................ 29
5.0 LESSONS LEARNED AND RECOMMENDATION .............................................................. 30
5.1 LESSONS LEARNED ................................................................................................................... 30
5.1.1 Focus .................................................................................................................................... 30
5.1.2 Partnerships are difficult to develop: dependencies are not ................................................ 30
5.1.3 What gets measured gets done.............................................................................................. 30
5.1.4 Monitoring and Evaluation................................................................................................... 30
5.2 RECOMMENDATIONS ................................................................................................................ 31
5.2.1 Policy Issues ......................................................................................................................... 31
5.2.2 Operational Matters ............................................................................................................. 32
5.2.3 Sustainability Issues ............................................................................................................. 33
6.0 CONCLUDING REMARKS....................................................................................................... 35
APPENDIX A ............................................................................................................................................. 36
APPENDIX B.............................................................................................................................................. 49
2
ABBREVIATIONS AND ACRONYMS
3
EXECUTIVE SUMMARY
Although no significant Foreign Direct Investments (FDIs) have come into Afghanistan,
the huge amount of donor aid (approx US$1.5 billion in 2005-2006)1 flowing into the
country provides substantial potential for the local private sector to flourish, expand and
diversify. Purchasing local goods and services is the mechanism by which the
operational funds disbursed for Afghanistan can be directly injected into the local
economy providing increased opportunities that serve to increase the confidence and role
of domestic private sector in their country’s reconstruction. This approach adopted by
the Peace Dividend Trust (PDT)2 and supported by CIDA underscores the importance of
spending money in Afghanistan, not only on Afghanistan. Increased spending in country
has multiple benefits, including job creation that draws workers into the formal labour
force, unleashing entrepreneurship, stimulating growth and encouraging domestic
investment.
This evaluation report provides feedback on progress to the CIDA’s Afghan Task Force
(ATF) and through it to stakeholders, identifies any obstacles to achieving objectives, and
provides advice on any changes needed to improve relevance, effectiveness, efficiency or
1
According to PDT’s research study, approximately $1.5b in official development assistance was
disbursed in Afghanistan during the Afghan fiscal year 1384 (21 March 2005 to 20 March 2006).
2
Peace Dividend Trust (PDT) is registered as a NGO in Afghanistan. PDT is a non-profit foundation
operating in post conflict areas of Sierra Leone, Timor-Leste and Afghanistan with offices in London
UK and Ottawa Canada. PDT’s mission is to make peace and humanitarian operations effective,
efficient and equitable resulting in cheaper, faster and more successful missions, delivering a stronger
peace and a more sustainable recovery.
1
sustainability. The evaluation methodology included primary and secondary data
collection through the combination of literature and document reviews, field visit to
Kabul, key stakeholders interviews, and physical verification. The information was
analysed to answer the key evaluation questions.
SUMMARY OF FINDINGS
For the purpose of the evaluation, the services and activities of PDT’s KPM pilot project
are broadly divided into the following five main streams:
i) Match Making
ii) Afghan Procurement Directory
iii) Policy Development
iv) Tender Dissemination
v) Capacity Development
Since its inception in January 2006, PDT’s pilot project services including activities and
principal achievements are provided below:
2
Tender Dissemination • Collects tender notices from • Customer visits since
international and national November 2006: 1250
organizations • 671 tenders collected and
• Distributes tenders and disseminated
RFQs via email, print media, • Helped businesses win
hard copy tenders of over US$1
• Assists business million
representatives with • Many repeat tenders won
preparing by businesses have not
tender documents been reported.
b) Efficiency: The five key project services have been quite efficient. All stakeholders
interviewed in the field have also expressed the positive contribution of the project
as well as its usefulness resulting from the achievement of the outcomes. Within the
budget and funding allocated, the PDT-KPM project has efficiently utilized its
resources. Most of the initiatives are cost effective and provided good return for
value of investment for each of the five streams of PDT services. (E.g. PDT held six
training sessions, the average cost per session for salaries was $9.035.92 and for hard
costs (including materials) was $2,887.33 for a total of $11,923.25 per session. Over 60%
(86 out of 134 companies) of training participants have won contracts worth $24 million,
which is an outcome of training dispensed by PDT. In other words, average training cost
per company trained amounted to approximately $534.)
3
facilitating matchmaking contracts to the tune of US$ 34.9 million, procurement
deals through tender dissemination US$ 4.7 million, while more than 3,500
businesses have been listed in the online directory after physical verification. All
the constituents have also welcomed the other PDT services such as training and
capacity development as well as promotion of Afghan First Policy.
RECOMMENDATIONS
Following are the key recommendations from the evaluation of the PDT KMP pilot
project:
1. Policy Issues
a) Partnerships: One of the successes of the project has been to develop and
maintain partnerships with stakeholders. Being a CIDA supported project, it is
important to demonstrate that similar partnerships exist with Canadian
institutions. Unfortunately, there is no evidence of local procurements by
either the Canadian embassy or the military. It is recommended that CIDA, in
collaboration with PDT headquarters, meet and brief the officials or ministers
of DFAIT and DND on the project’s objectives and seek support for local
procurement.
b) Institutional Capacity Development: PDT has obtained significant credibility
from international procurement agencies and military as well as local private
sector of being a neutral organization. Can similar credibility and neutrality be
expected if the activities are transferred to local institutions? This needs to be
addressed during the next project phase. It should be noted that PDT is still
refining activities dispensed during the pilot phase. While the pilot project
3
The other issues of financial viability and donor support are discussed as a part of the recommendations
4
has been highly successful in impact and in developing initial working
methods and tools, these have not yet reached the point where they can be
transferred to a partner organization mainly because the mid-term future of
both partner organizations (AICC and AISA) is currently unclear. It is
imperative; therefore, PDT starts collaborating with the proposed local
organization(s) very early in the next phase and gradually transfer its activities
depending on the readiness and preparedness of the national counterpart(s).
c) Impact Assessments: The focus of pilot project’s achievements has been on
the number of beneficiaries and total value of contracts signed by the local
private sector. It would have been ideal for the pilot project’s overall
outcomes to be measured by developing baseline data on the actual number of
jobs created (or protected) including for women, incomes generated, reduction
in poverty, taxes paid and other indicators which will help to develop better
strategies as a case for local procurements. However, it should be noted that
tracking information is a significant challenge to the PDT project team
because most of the data acquired remain estimates, despite follow-up with
buyers and suppliers. Compiling basic economic data for Afghanistan is
difficult in a continuously fluid economy but mechanisms need to be
developed for baseline data and impact assessments.
d) Crosscutting Themes (Gender and Environment): The Afghanistan
Marketplace proposal contains a gender strategy and an environmental
component that will be implemented. This strategy will allow CIDA to ensure
that the principles developed by PDT for integrating gender and environment
considerations in the project’s next phase are applied from the outset. Taking
into consideration the social and economic situation of Afghanistan, it should
be noted that the number of existing female entrepreneurs remains low. In
addition, businesses operating in one of the most difficult business and
security environments in the world may not direct scarce economic resources
to environmental interventions. Therefore, PDT should focus on providing
suggestions for environmental best practices that are linked to cost savings,
thereby proposing a realistic approach to getting Afghan businesses to
implement these measures. Additionally, if the national partner is deemed
deficient with respect to gender-sensitivity or lacking in the ability to manage
environmental impact assessment and mitigation, sufficient expertise needs to
be accessed, preferably from in-country sources such as local NGOs,
academia, and development-related institutions, or from the Canadian sources.
Sufficient financial resources directed toward gender mainstreaming and
environmental sustainability must be allocated in the next phase of the project
if gender and environment are to be integrated as cross cutting themes.
e) Exit Strategy: There is a clear future for PDT’s activities in Afghanistan
rather than creating a new institution. The activities of PDT are seen as great
benefit to the two existing institutions, namely AISA and AICC. Therefore, it
is possible that activities implemented by PDT can be transferred to these
institutions. The exit strategy proposed for the Afghanistan Marketplace
project allows for some flexibility in the transfer of activities since the
medium term future of any potential partner organization remains unclear. It is
5
recommended that a clear exit strategy be spelt out by the beginning of the
next phase by clearly dividing the roles and responsibilities.
2. Operational Matters
Updating the current accounting systems at the field office from basic Excel
sheets to Quick Books accounting method will help in simplifying procedures
(currently significant time is wasted in account posting and maintenance) and
providing management information systems.
6
c) Strengthening institutional memory:
3. Sustainability Issues
Following the recent post-training survey results wherein nearly 60% of the
past participants had indicated willingness to pay training fees, PDT in
conjunction with the national partner(s) should examine the possibility of
introducing nominal affordable fees. As most of the activities of PDT bring
direct commercial benefits to local businesses, it would be advisable to
recover the cost of some of the initiatives. This would help in future planning
of PDT’s training as well as other services.
c) Donor Support
The PDT-KPM project has demonstrated its relevance and effectiveness to all
its stakeholders. Donor support to maintain the continuity of the project’s
ongoing activities is critical. Previous experience of delay in CIDA’s funding
resulted in PDT scrambling for funds to keep the project running.
Sustainability and credibility of the project is likely to be affected if the
7
funding is not in place early enough for the next phase. It is recommended that
CIDA initiate a dialogue with relevant departments on local procurements and
with donors to seek further support to the project’s national partners to
continue PDT’s activities. This will ensure long-term sustainability of the
project especially due to limited resources available locally.
CONCLUDING REMARKS
After examining the documentation provided, and meeting with key stakeholders of
PDT’s Kabul Market Place, it can be concluded that this pilot project has both achieved
all its outputs and outcomes, and has far exceeded its goals. In particular, the project
provides core services such as local procurement, tender dissemination and training
which, no other agency provides in Afghanistan. The pilot project has rightly identified
and developed a special niche area of services that are critical for the development of
private sector in Afghanistan. Additionally PDT’s KPM project deserves special credit
for the accomplishment within the given timelines, especially working under difficult
situations in post conflict zones. PDT’s next phase of the project, namely Afghan Market
Place, deserves full support as the project intends to deepen and widen its activities with a
focus on both sectors-wide and countrywide.
Overall CIDA has to be commended for supporting this initiative and it can be the
flagship of ATF’s achievements and needs to be widely publicized particularly in leading
Canadian business magazines and newspapers.
8
1.0 INTRODUCTION
1.1 Background
In January 2006, PDT launched the Kabul Procurement Marketplace (KPM) pilot
project with financial support from CIDA. The project’s broad objectives are to:
The KPM project scope covers the whole country, with specific focus on areas
where international buyers and local suppliers operate. Currently the project’s
focus has been mainly in and around Kabul. The KPM is located within the
Afghanistan International Chamber of Commerce (AICC) in Kabul.
The goal of PDT’s KPM pilot project is to support the private sector development
in Afghanistan by channeling the operational spending of international agencies,
donor countries and enterprises into Afghan economy by enabling better
marketplace access and working with local companies to improve their chances of
winning new contracts. Broad activities of KMP project are as follows:
9
5. Train and assist entrepreneurs in areas that will help them improve their
ability to supply goods and services to the UN an international agencies,
6. Design and launch an on-line business directory of local SMEs to provide
good and services,
7. Assist women’s SMEs to market to international buyers.
This evaluation report provides feedback to the CIDA’s Afghan Task Force (ATF)
and through it to stakeholders on progress, identifies any obstacles to achieving
objectives, and provides advice on any changes needed to improve relevance,
effectiveness, efficiency or sustainability.
PDT’ KPM pilot project has received a total amount of CD$ 1,163,470 from
CIDA since its inception in January 2006.
10
2.0 PROJECT PROFILE
One of the principal achievements of the post-Taliban era has been the sustained
high level of formal economic growth and an exponential rise in trading volumes
with regional partners, including Pakistan, Iran, India, Uzbekistan and Turkey.
The IMF estimates that official GDP growth averaged 22.5% between 2002 and
2004 5 , while growth is predicted to reach 12% in 2007/2008 and 10% in
2008/2009. These growth rates are based on two key factors: rebounding
4
The HDR for 2006 does not include Afghanistan due to insufficient data.
5
World Bank, “The Investment Climate in Afghanistan”, Official Draft, December 2005.
www.aisa.org.af/PDF/Afghanistan%20ICA-Official%20Draft.pdf.
11
agriculture and a sustained level of activity in construction and services6. These
factors underscore the dual importance of growth stimulated by both the domestic
market and demand from international reconstruction partners.
6
“Statement of the IMF Staff at the Conclusion of the Mission for the First Review under the Poverty
Reduction and Growth Facility”, Kabul, November 26, 2006,
www.imf.org/external/np/ms/2006/112602/htm
12
13
Table below provides a World Bank’s graphic display of ranking and Afghanistan’s
position in relation to each of the indicators.
14
2.4 Local procurement and private sector development
In order to ensure the most effective and efficient use of funds being delivered to
Afghanistan, the international community and the Afghan Government committed
themselves to the Afghanistan Compact Agreement. Under the Compact, donors
agreed to channel an increasing proportion of their assistance through the core
government budget, either directly or through trust fund mechanisms by: (a) using
national partners rather than international partners to implement projects; (b)
increasing procurement within Afghanistan; and (c) using Afghan goods and
services wherever possible, rather than imported goods and services. This
approach is founded in the understanding that by using development assistance
locally it can provide the impetus for economic growth by injecting income and
wealth into the local economy. Some of the benefits that development assistance
projects can deliver are as follows:
Cost Savings
• Reduces freight, transport costs
• Savings in the range of 30-60 percent
Economic Impact
• Generates employment and revenue for the private sector and government
• Promotes and increases skills and capacity of local businesses
• Justifies capital investments to build infrastructure
• Develops a sustainable market
Environmental Benefits
• Reduces waste
• Reduces pollution and greenhouse gas emissions
15
A recent study7 on the "Economic Impact of Peace keeping" concluded that only
between 4%-9% of UN peacekeeping missions funds stay in the local economy.
External Expenditure
Local Procurement
MS A
National S taff
The study further analyzed the major channels of ODA disbursements including
international and local contracts, which are provided in the following table.
7
“Economic Impact of Peace Keeping”, - a study 10 UN Peace Keeping Nations, January 2006
8
PDT, “Afghanistan Compact Procurement Monitoring Project”, April 2007
16
The PDT study9 concluded the following:
9
ibid
17
b) From suppliers’ perspective:
18
3.0 EVALUATION PROFILE
This evaluation is carried out in accordance with the CIDA’s published Evaluation
Guidelines and it intends to improve performance by assessing progress of PDT’s pilot
project in Kabul, and to contribute towards decision-making, organizational learning, and
greater accountability and transparency. Lessons learned from this evaluation will also
allow more informed preparation of subsequent phase two of the Project “Afghan Market
Place (AMP)” which intends to widen and deepen its activities both geographically as
well as outreach to more local businesses sector wide and Afghan wide.
The key objective of this evaluation is to review the progress of PDT’s pilot project since
its inception, determine how it has evolved, and identify any issues or practical measures
that can be taken following the conclusion and recommendations with this evaluation.
The evaluation framework is the key element of this report, as it systemizes the
methodology, identifying the issues to be addressed, sub-questions that provide
elaboration; and the performance indicators. The evaluation issues reflect what was set
out in the TORs, but have been simplified and re–organized.
The evaluation framework is provided in Appendix A.
The methodology adopted for this evaluation is designed to meet the requirements and
expectations set out in the TORs for this assignment. The approach is considered credible
and appropriate for identifying the results attributable to the KPM project given the range
of information that is currently available, and the limited time and resource constraints.
3.4 Documentation Review
The process of identifying and reviewing available project documents began with the
awarding of the contract and a review of key project management documents was
instrumental in preparing this report.
The evaluation work started with meeting the ATF officials at CIDA, Ottawa and PDT
officials at the headquarters in Ottawa on December 4th followed by field missions to
Kabul Afghanistan from December 6th -18th, 2007. CIDA’s ATF and PDT's Kabul office
in consultation with its Ottawa headquarters had facilitated meetings with the following
organizations:
19
• PDT Country Office (Director and Key Staff)
• Local private sector (beneficiaries of PDT’s activities)
• International Buyers (including International Security Assistance Force (ISAF)
of NATO)
• CIDA staff based in Kabul
• Government Ministries (Ministry of Commerce, Ministry of Finance’s Public
Procurement Unit)
• Afghan International Chamber of Commerce (AICC) – PDT’s National
hosting organization
• Afghanistan Investment Support Agency (AISA) –PDT’s potential partner for
future operations
• Afghan Growth Finance – A key financing agency for SMEs in Afghanistan
The field visits were well coordinated by the PDT’s Afghan Office to make best
use of the limited time available, including arranging meetings with key contact
points of government ministries, international buyers and private sector. The field
mission started with a good professional presentation by PDT Afghan office
country director and key staff on its activities and accomplishments and the
cooperation with the team has been good, and interaction has been marked by a
strong degree of candour.
a) For the efficacy analysis, the evaluator assessed whether the program is
meeting its developmental objectives (or can be expected to reach its
developmental objectives by the end of the program) by reviewing the program
plans, performance measurement framework, and reports to ensure:
• Progress made towards the achievement of results at the output and outcome
levels
• Level of actual versus intended results
• If there is shared responsibility and accountability for results
• Whether there is active participation of local country partners and
beneficiaries in the design, implementation, monitoring and periodical
evaluation of the Program
• Is there an appropriate level of coordination between the various
stakeholders
• Is the coordination with other donors is adequate
20
• Does the local partners have the appropriate authority and tools that are
required to make decisions and take actions
b) For efficiency analysis, the evaluator briefly analyzed project’s schedules and
budgets to ascertain whether the program is achieving its objectives on schedule
and within budget generally, and answer the following principal evaluation
questions:
• Are resources and services delivered in a manner that effectively responds to
conditions, including risks, needs, opportunities or problems?
• Has the Performance Measurement Framework been applied during Program
implementation?
3.7 Limitations
Due to limited time during the field visit to Kabul, the evaluator had to depend on
the meetings arranged by the local PDT office. Most of the private sector
respondents were selected by the PDT office, though some additional meetings
particularly with the government departments were arranged at the request of the
evaluator.
KPM project’s accomplishment in terms of contracts won and success stories are
based on the information provided by the PDT Kabul office. The evaluator has
therefore depended on the PDT’s information and success stories matrix with a
couple of on the spot verifications. Triangulation was used to verify and validate
the findings.
21
4.0 EVALUATION ANALYSIS AND FINDINGS
4.1 Effectiveness
The table below provides the efficacy of PDT’s KPM pilot project by assessing the
outputs and outcomes.
PDT’s Kabul Market Place Pilot Project Assessment of Outputs and Outcomes
(Source: Peace Dividend Market Place LFA)
Results
• More than one dozen international agencies have
been identified, contacted and enrolled as the
potential procurement buyers.
Establishment of the Tender Distribution Point by PDT
has provided access to current tenders from international
organizations, private business, government line ministries,
and military organizations. This facility has assisted local
SMEs significantly in terms of cost saving and timely
information. (E.g. PDT’s tender distribution statistics
indicates that a construction company would have to visit
22
14 ministries, at least 5 international organizations, and
browse 10 different websites in order to access all of the
tenders open to bids in the span of one month). Locating
tenders in a single office therefore increases transparency,
enhances understanding of the procurement needs of major
buyers and facilitates greater access to information.
23
• Helped 86 companies win over US$ 24million
worth of construction contracts
• Delivered 6 training packages in 4 cities:
- Kabul (3), Mazar-e-Sharif, Jalalabad, Herat
5) Design and launch on- i) PDT launched country’s first online business directory
line business directory of “Afghanistan Procurement Directory”, it is the only
local SMEs to provide online directory that connects international and national
goods and services. buyers to Afghan companies. It currently features business
profiles of over 3,500 Afghan companies.
ii) All businesses are initially verified by PDT team with
regard to their registration, business address, and nature of
business including experience among other criteria.
Information is updated on a regular basis.
www.buyafghan.af
www.peacedividendmarketplace.af
Results
• Helped prime contractors identify construction
companies for tenders worth roughly US$4.7
million
Approximately 945,000 hits since December 2006
6) Assistance to women’s i) PDT’s pilot project key focus was on start-up, market
SMEs to market to research, networking and establishing of linkages, as well
international buyers. as promoting PDT’s activities to various stakeholders as a
result gender was not the main focus. Efforts were made to
include women in the training and capacity development
programmes. However, the next phase of Afghan Market
Place has given prominence to gender in most of its
initiatives.
24
2) Enable a business climate that draws i) There has been no evidence of informal
informal entrepreneurs into the formal entrepreneurs moving into formal
economy, increasing local tax revenues and economy. Most of the beneficiaries are
adherence to labour standards. already registered and operate within the
formal economy.
ii) No study or survey has been initiated to
know exact amount of tax generated or
local businesses adhering to labour
standards and environment regulations.
3) Build capacity of local entrepreneurs and i) PDT’s capacity building and training
improve their ability to attract international programmes have provided local
clients and investors in the wake of the entrepreneurs to understand and adapt to
peacekeeping economy by encouraging the the requirements of international bidding
development of commercially sustainable procedures.
business models that can be expanded and ii) lessons learned from pilot project
replicated. (namely: advocacy, systematic tracking of
results, relationships and partnerships with
both buyers and vendors) can be further
strengthened in the next phase
25
unpredictable utilities, non- availability of finance and credit while government
policies are still in the process of evolving, among other factors prevailing in
Afghanistan.
4.2 Efficiency
All stakeholders interviewed in the field have also expressed the positive
contribution of the project as well as its usefulness in achieving the outcomes. The
information provided by the project to local SMEs is also considered timely and
has been appreciated by all businesses. Meeting with the international buyers
particularly with ISAF indicated that PDT-KPM’s project has been efficiently
providing its services and information about the background of local vendors as
well as helping local businesses to understand the rules and procedures of
international procurement.
Within the budget and funding allocated, the PDT-KPM project has efficiently
utilized its resources. In fact, the project had earned quite good credibility among
other donor agencies that were able to provide bridging finance at the critical
juncture when CIDA’s financing was delayed. Most of the initiatives of PDT-
KPM are cost effective and it has provided good return for value of investment for
each of the five streams of services provided. An example of PDT’s cost effective
training programmes and its result is provided below:
26
Cost Effectiveness of the PDT’s KPM training Programme
During the pilot project, from April 2006 to January 2007, six training sessions were held in Kabul
and in regional areas of Afghanistan and 134 Private sector participants were trained mostly from
Construction and Printing sectors. In 2007, no trainings were provided, as there was no resourced
position for a trainer in place
It should be also noted that the previous training workshop were funded mainly by Bearing Point
(USAID) as well as DFID. CIDA-funded costs amounted ONLY to $28,390 or 40%. Salary scales
were determined by Bearing Point, although hiring was done by PDT. Here is an approximate
breakdown (based on available records) of the previous training program, which covered an 11
month period:
Item Description Cost
Personnel International Consultant and National $54,216.00
Training Associate
Materials Stationary, Manual (3,000 copies $2,680.00
printed) and CDs (1,000 copies printed)
Travel & Logistics Bulk cost of all other workshop costs, $14,644.00
including travel, venue, logistics, etc.
Total $71,540.00
Given that PDT held six training sessions and trained 134 companies at a total cost of $71,540,
therefore the average cost per session was $11,923.25. Average training cost per company trained
amounted to approximately $534 per company. In total, the fifty companies that responded to
PDT’s evaluation reported that they were able to obtain $ $24,207,446 in contracts directly because
of improved knowledge resulting from their participation in PDT’s training program. This amounts
to an average of $180,653 per company.
PDT training investment of $534 resulted in $180,653 of contract value on average for
companies that participated.
(Source: PDT Training Program 2006- Survey Assessment, December 2007)
4.4 Relevance
4.5 Sustainability
27
President of AICC also assured that the changes would not affect PDT’s
operations.
28
4.5.5 Institutional Memory
29
5.0 LESSONS LEARNED AND RECOMMENDATIONS
5.1.1 Focus
Matchmaking and successful tender bids have shown the total value of contracts
signed by local private sector. However actual impact needs to be measured
especially in terms of jobs created (including women employment), income
generated and retained within the country, wealth created through new
investments, taxes paid, number of women entrepreneurs, among others.
Additionally needs assessment has to be done both geographically and sectoral
basis for widening and deepening the next project phase.
30
monthly tracking of project indicators. Tracking systems could benefit from
moving away from Excel-based systems to an integrated MIS.
5.2 Recommendations
Following are the key recommendations from the evaluation of the PDT KMP pilot
project:
5.2.1 Policy Issues
a) Partnerships:
One of the successes of the project has been to develop and maintain partnerships
with stakeholders. Being a CIDA supported project, it is important to demonstrate
that similar partnerships exist with Canadian institutions. Unfortunately, there is
no evidence of local procurements by either the Canadian embassy or the military.
It is recommended that CIDA, in collaboration with PDT headquarters, meet and
brief the officials or ministers of DFAIT and DND on the project’s objectives and
seek support for local procurement.
c) Impact Assessments:
The focus of pilot project’s achievements has been on the number of beneficiaries
and total value of contracts signed by the local private sector. It would have been
ideal for the pilot project’s overall outcomes to be measured by developing
baseline data on the actual number of jobs created (or protected) including for
women, incomes generated, reduction in poverty, taxes paid and other indicators
which will help to develop better strategies as a case for local procurements.
However, it should be noted that tracking information is a significant challenge to
the PDT project team because most of the data acquired remain estimates, despite
follow-up with buyers and suppliers. Compiling basic economic data for
Afghanistan is difficult in a continuously fluid economy but some mechanisms
need to be developed for baseline data and impact assessments.
31
d) Crosscutting Themes (Gender and Environment):
The Afghanistan Marketplace proposal contains a gender strategy and an
environmental component that will be implemented. This strategy will allow
CIDA to ensure that the principles developed by PDT for integrating gender and
environment considerations in the project’s next phase are applied from the outset.
Taking into consideration the social and economic situation of Afghanistan, it
should be noted that the number of existing female entrepreneurs remains low. In
addition, businesses operating in one of the most difficult business and security
environments in the world may not direct scarce economic resources to
environmental interventions. Therefore, PDT should focus on providing
suggestions for environmental best practices that are linked to cost savings,
thereby proposing a realistic approach to getting Afghan businesses to implement
these measures. Additionally, if the national partner is deemed deficient with
respect to gender-sensitivity or lacking in the ability to manage environmental
impact assessment and mitigation, sufficient expertise needs to be accessed,
preferably from in-country sources such as local NGOs, academia, and
development-related institutions, or from the Canadian sources. Sufficient
financial resources directed toward gender mainstreaming and environmental
sustainability must be allocated in the next phase of the project if gender and
environment are to be integrated as cross cutting themes.
e) Exit Strategy:
There is a clear future for PDT’s activities in Afghanistan rather than creating a
new institution. The activities of PDT are seen as great benefit to the two existing
institutions, namely AISA and AICC. Therefore, it is possible that activities
implemented by PDT can be transferred to these institutions. The exit strategy
proposed for the Afghanistan Marketplace project allows for some flexibility in
the transfer of activities since the medium term future of any potential partner
organization remains unclear. It is recommended that a clear exit strategy be spelt
out by the beginning of the next phase by clearly dividing the roles and
responsibilities.
32
Evaluator finds since this is an important activity of PDT, and it should have
been part of the next phase of CIDA’s funding.
Updating the current accounting systems at the field office from basic Excel
sheets to Quick Books accounting method will help in simplifying procedures
(currently significant time is wasted in account posting and maintenance) and
providing management information systems.
33
• milestones and benchmarks;
• and a clear exit strategy.
Partner institutions need to be provided with financial planning, management
tools and training. Maximum responsibility should be transferred to the
national partner (s) as a component of overall capacity development /
institutional strengthening
b) Financial Viability
Following the recent post-training survey results wherein nearly 60% of the
past participants had indicated willingness to pay training fees, PDT in
conjunction with the national partner(s) should examine the possibility of
introducing nominal affordable fees. As most of the activities of PDT bring
direct commercial benefits to local businesses, it would be advisable to
recover the cost of some of the initiatives. This would help in future planning
of PDT’s training as well as other services.
c) Donor Support
The PDT-KPM project has demonstrated its relevance and effectiveness to all
its stakeholders. Donor support to maintain the continuity of the project’s
ongoing activities is critical. Previous experience of delay in CIDA’s funding
resulted in PDT scrambling for funds to keep the project running.
Sustainability and credibility of the project is likely to be affected if the
funding is not in place early enough for the next phase. It is recommended that
CIDA initiate a dialogue with relevant departments on local procurements and
with donors to seek further support to the project’s national partners to
continue PDT’s activities. This will ensure long-term sustainability of the
project especially due to limited resources available locally.
34
6.0 CONCLUDING REMARKS
After examining the documentation provided, and meeting with key stakeholders of
PDT’s Kabul Market Place, it can be concluded that this pilot project has both achieved
all its outputs and outcomes, and has far exceeded its goals. In particular, the project
provides core services such as local procurement, tender dissemination and training
which, no other agency provides in Afghanistan. The pilot project has rightly identified
and developed a special niche area of services that are critical for the development of
private sector in Afghanistan. Additionally PDT’s KPM project deserves special credit
for the accomplishment within the given timelines, especially working under difficult
situations in post conflict zones. PDT’s next phase of the project, namely Afghan Market
Place, deserves full support as the project intends to deepen and widen its activities with a
focus on both sectors-wide and countrywide.
Overall CIDA has to be commended for supporting this initiative and it can be the
flagship of ATF’s achievements and needs to be widely publicized particularly in leading
Canadian business magazines and newspapers.
35
APPENDIX A
Kabul Procurement Market Place (KPM) – Pilot Project Evaluation Framework
Evaluation Performance Indicators – Variables Evaluation Findings
Issues Sub-Questions
Comments to be Considered
A-1 Is the project Are continental, regional national and Afghan Government and Donor agencies are quite supportive
Relevance responsive to YES local policies and strategies supportive of KPM pilot project and fits within the national priorities of
Afghanistan’s of the Project? How is the project supporting PSD
development responsive to Afghanistan’s priorities?
challenges,
priorities and
objectives?
Does the YES Does the intervention make sense in Very practical hands-on intervention providing solutions and
program terms of the conditions, needs and addressing needs.
address the problems that it is intended to address?
real needs of Is the design (as defined in the Project Design suits the local context and needs.
targeted Program Approval Document and the
beneficiaries? Program Implementation Plan)
appropriate, and is it based on a sound
understanding of the local context?
Is the intervention still relevant to the YES, the project is quite relevant and useful
needs of the targeted beneficiaries and
local partners?
Are beneficiaries clearly identified and Pilot project has identified the target beneficiaries
are they targeted for benefits
throughout implementation?
Does the YES Do the results contribute to CIDA's YES. With local procurement from local SMEs, it is anticipated
program overall goals of poverty reduction and that jobs will be created, income generated and wealth created
support CIDA’s sustainable development? which will help in reducing poverty as well as create
policies, sustainable development
priorities, and
Canadian
foreign policy?
ii
Evaluation Performance Indicators – Variables Evaluation Findings
Issues Sub-Questions
Comments to be Considered
A-3 What are the KPM pilot Is there evidence of improved PDT did most of the initiatives but local partners were also
Efficacy development project has capabilities of local counterparts to involved. The next phase may include capacity development
Results and clearly deliver quality services to the local and of local counterparts when handover will be required, which
Success demonstrated international businesses? Of increased will likely to help in the delivery of services by local partners.
Factors? the potential capacity of partners to deliver services PDT’s local staff are being trained on the job
for PSD in to their members and diversify service
Afghanistan lines? PDT Survey has indicated that there is a possibility of
and that the Are there improved revenue streams, generating revenue from training programs. Similar analysis/
local SMEs in both cases? surveys have to be undertaken for other initiatives to ascertain
have the Were/are the pilot projects contributing whether a fee can be charged.
necessary to attainment of results (i.e. are they a
capabilities to useful mechanism towards achieving KPM project has clearly identified niche area of international
supply and project results and how so)? procurement locally and its impact in the host economy.
meet Is there evidence of policy and Govt. of Afghanistan’s Ministry of Commerce, PSD policy and
international regulatory impact or micro-macro strategy and Ministry of Finance’s Public Procurement Unit
standards linkages? have recognized the need to support local SMEs through local
provided they procurement in their policy and strategy documents.
are given: a)
necessary Similarly, Both Afghanistan Compact Agreement and National
information on Development Strategy have acknowledged the importance of
opportunities local procurement as well as support.
b) training and
c) enabling What is the level of actual versus Both national government and private sector are supportive of
environment. intended results with respect to all PDT initiatives. In fact, the have incorporated PDT's
advocacy? Are the partners happy to initiatives in their respective policies and programs.
go with the flow of both program
activities and national policies and
regulation? Alternatively, are the
partners increasingly proactive in their
engagement with government and
other stakeholders? In short, do the
partners drive the agenda?
Are there any unintended results that Multi million dollars business generated and recycled within
can be attributed to the Project? the local economy has far exceeded the initial expectations.
iii
Evaluation Performance Indicators – Variables Evaluation Findings
Issues Sub-Questions
Comments to be Considered
Is there evidence generally of PDT’s training and capacity development has provided new
enhanced skills, improved economic skills to local SMEs especially on tender bidding and business
means and prospects, capacities for plans. The success of businesses wining repeat international
being self-sustaining, empowerment, bids is evidence of sustainable economic development
and self-awareness?
To what extent KPM pilot Is there active participation of partners KPM pilot project was designed to test various initiatives in
have project’s and beneficiaries, including women, in consultation with stakeholders. Constant monitoring of its
partnerships success is the design, implementation, monitoring activities through surveys and call centre helped PDT to
and/ or linkages mainly due to and evaluation of the Project? design the next project phase namely Afghan Market Place.
between strong Within the partnership or linkages that KPM project has clearly gained significant respect from all
institutions and linkages with are facilitated, is there shared stakeholders. Sharing of responsibility and accountability is
organizations both national responsibility and accountability for likely to take place in the next phase of the project.
been and decisions and results? Is there a
encouraged international sufficient level of trust and respect
and supported? organization. between the partners?
PDT staff have Is there an appropriate level of YES
demonstrated coordination between the various
very good stakeholders?
interpersonal Do Afghan partners have the YES
skills of appropriate authority and tools required
working with to make decisions and take actions?
different In this project, how strong is the The interest shown on KPM project activities by the Ministry of
organizations partnership-ownership linkage for Commerce’s Afghan Investment Support Agency and the
sustainability of improvements to local private sector organization AICC demonstrates strong
government? partnership linkage as well as continuity of the project’s
activities. The partners have infused KPM project with a
climate of inclusion and togetherness that translates into a
culture of optimism and “can do”.
iv
Evaluation Performance Indicators – Variables Evaluation Findings
Issues Sub-Questions
Comments to be Considered
Are management structures effective in YES.
responding to ongoing challenges and Strengths: international expertise, adaptability to local
in promoting creativity and innovation? conditions, quick learning of local language and traditions,
What are the strengths and designing of various initiatives, and adapted to the local
weaknesses of management needs. Good Teamwork and involvement of all staff in future
structures? activities. Excellent public relations and Communications.
High level of commitment and vision.
Weaknesses: Management staff comprises of all external staff
on short term contracts and if they leave the project will have
serious impact on the project’s activities
Are the Resources Are human, financial and physical All resources have been appropriately used and financial
A-3 resources and were just resources used appropriately and information maintained. However, financial systems in Kabul
Efficiency services sufficient to financial information accurately and need to be upgraded. Currently financial records are
provided by initial design adequately maintained? Are the maintained manually in Excel sheets which, can be upgraded
PDT designed of the project. variances justified? to Quick Books.
to effectively However, PDT
respond to the had to source Is there an adequate level of Due to short term, contracts of the staff there has been
needs and additional management personnel and resources activities suspended due to non-availability of new staff due to
address the funding to in place for effective management and lack of funding. ( e.g. Training and capacity development)
problems meet the new monitoring of the Project?
identified in the project needs Has the Project identified and is it KPM pilot project got support from various international
design? like training, making effective use of Canadian and agencies including military. Unfortunately, there has been no
surveys. Etc. Afghan areas of competitive evidence of local procurement from either Canadian Embassy
advantage? What has been the benefit or the Canadian military.
of using a Canadian executing agency?
Are resources and services delivered in YES
a manner that effectively responds to
conditions, including risks, needs,
opportunities or problems?
v
Evaluation Performance Indicators – Variables Evaluation Findings
Issues Sub-Questions
Comments to be Considered
Is the project well coordinated with Donors in Afghanistan have supported KPM project. The
other donors? Is this coordination project has also become a reliable and credible partner of
based on a high level of knowledge of many donors including UNDP, World Bank, DFID, USAID
what other interveners are doing in among others
difficult contexts such as Afghanistan?
Does the project know and take into
account their views on key aspects
such as the absorptive capacity of
partners?
Are the Project office What evidence is there of prudence Both at the PDT field office and at Headquarters, evaluator was
accounting and in Kabul needs and probity being appropriately given all the required financial information. The quarterly
financial to upgrade the exercised? Is there adequate reports also demonstrate sufficient transparency
system accounting transparency at all levels?
adequate for software
effective project system similar
management? to Quick
Books
Is there effective sharing of KPM project has been promptly sharing the required
information? Is knowledge knowledge and information with its partners. This has been
management effective, in the sense highly appreciated by the stakeholders. However, knowledge
that the partners actively engage in management now needs to go further, play to its own
knowledge production, sharing, and strengths i.e. address elements of Monitoring and Evaluation
practice derived from knowledge? How where possible, and start monitoring development outcomes
prevalent is the KAP results chain: by using KAP results chain where feasible.
knowledge → attitude→ practice?
Are there YES Has the Performance Measurement PMF has been applied in this pilot project, work plans and
appropriate and Framework been applied during reporting on activities and outputs through quarterly reports
adequate Program implementation? What have
information the partners contributed?
systems in Number and nature of systems in place Responding to changes taking place in a volatile environment
place to identify Evidence that systems function and are like Afghanistan, there are on going changes to activities,
emerging risk, used effectively procedures, templates, timelines roles and responsibilities
vi
Evaluation Performance Indicators – Variables Evaluation Findings
Issues Sub-Questions
Comments to be Considered
developing Is there evidence of the success of Following an operational review of KPM IN October 2006 and
issues, and systems in responding to change? internal assessments there is evidence of systems have
project Application of lessons learned. changed responding to the needs especially the new training
performance Evidence of timeliness of using and capacity development program, Tender Distribution Point,
levels? information to manage effectively, Call Centre, Matchmaking and Policy Development
appropriateness of actions taken
Has the Project YES Did CIDA anticipate and respond to CIDA has been supportive to the changes made to suit the
anticipated and change based on adequate local environment.
responded to information?
risks identified Is the relationship between costs and Quite reasonable. In fact in certain areas it has far exceeded
during design results generally reasonable? the expectations i.e. minimum cost and high returns. E.g
or which may ( interim results demonstrate that a PDT training investment of
arise $534 resulted in $180,653 of contract value on average for
subsequently companies that participated.
during
implementation Are the expected results achievable YES
? with the level of Canadian and Afghan
resources available for
implementation?
Are resource Due to Is there demonstrated local There is a high commitment and ownership on the part of the
levels shortage of commitment to Project activities, to the government and private sector associations. Both national
adequate? financial attainment of results and the methods government and private sector are supportive of all PDT
resources chosen to achieve them? Does this initiatives and have incorporated in their respective policies
KPM sourced translate into proactive project of and programs. The next phase is likely to have the PDT’s
funds from activities driven by partners? programs can be shared by AISA & AICC to drive the agenda
other donors
(USAID and
DFID)
A-4 Extent of local The next Is there a commitment on the part of AISA is working on a new corporate plan and strategy to
Sustainabili ownership Phase will local partners to provide the necessary incorporate and integrated PDT’s activities. However in the
ty demonstrate and timely level of human resources case of AICC due to changes in the structure and
the exact and financial support to maintain management, it is not clear at this stage how such resources
nature of local benefits/results? will allocated for PDT’s initiatives.
ownership
vii
Evaluation Performance Indicators – Variables Evaluation Findings
Issues Sub-Questions
Comments to be Considered
Commitment of AISA is Is there adequate institutional capacity Both AISA and AICC need support and assistance in capacity
adequate working on the of local, national or regional partners to development.
resources commitment maintain results?
but not quite
clear form
AICC
Institutional Possible in the Is the policy and regulatory YES
capacity next phase environment conducive to maintaining
building results?
Conduciveness New policies To what extent did CIDA develop, CIDA’s support to the project has helped in raising the project
of international/ and strategies encourage and support new profile.
national are quite approaches and practices?
environment (e. conducive
g. domestic
policies)
How did CIDA CIDA’s What has been learned from project The level of participation and inclusiveness is very high
programming support for the implementation to date that could resulting in finding value to the services provided by the KPM
contribute to next phase strengthen the project’s capacity to project. Enhancing capacities of the local private sector
the ensures respond to its development challenges, through training and encouraging to bid for tenders on one
sustainability of continuity and priorities and objectives? side and working with international agencies to source locally
results? sustainability on the other has helped the project to achieve high level of
of the project. acceptance by all the stakeholders.
A-5 lessons What (if any) CIDA should What methods are successful? Direct and regular contacts
viii
Evaluation Performance Indicators – Variables Evaluation Findings
Issues Sub-Questions
Comments to be Considered
Learned main hold What needs to be done to further - direct and indirect impact assessment of the project in terms
implementation discussions achieve/enrich the project objectives? of jobs created, incomes generated, taxes earned by the Gov't,
and policy with the DFAIT etc.
issues need to and DND to - more support for gender equality and making sure it is more
be brought to encourage to strategic.
the attention of procure some - Strengthening B2B platform through increased local
the partners, of their procurement
Governments requirements - Increase local capacity of businesses through Business
and/or CIDA? locally and Development Services.
support the - Grading of local businesses on performance and
project. - better tracking system of local procurement
- establishing linkages to business development services with
finance
What changes (if any) to present CIDA initatie a dialogue with Donors to support PDT’s national
strategies and practices are partners to continue the initiatives and their institutional
recommended? capacity to deliver.
What should be future CIDA support to Subject/ Theme: Strengthening the institutional capacity of
Afghan Market Place? national partners for the provision of services and for
Subjects/themes advocacy
Geographic focus Geographic Focus should be first based on safe and secure
Macro-meso-micro level of intervention environment and based on the needs based survey of where
Balance between Canadian support most of the procurement goods and services are required as
agency/Afghan partner driven well as the capacity of local private sector to supply the needs.
Coordination with other donors Macro-meso or micro level interventions: No matter what level
is supported, the primary beneficiaries should be the local
private sector through national partners and local
governments. In the same vein, Afghans should progressively
become the principal orienters, planners and implementers,
with Canadian institution gradually assuming the role of arm's
length mentor, monitor and quality controller. Coaching
mentoring should gradually replace workshops.
Donor Coordination: can be further enhanced
ix
APPENDIX B
Kabul Contacts
Hamidullah Farooqi, CEO of Afghanistan International Chamber of Commerce
Al Haj Zabiullah Ehsan, President, Al Ehsan Construction Company
M.Ihsan Saffi, Managing Director, Melli Factories
Ahmad Jawid Omary, Procurement Policy Unit, Ministry of Finance
Mustafa Kazem, Director, Afghan Growth Finance
Siad Mubin Shah, Research and Policy Director, Afghanistan Investment Support Agency (AISA)
LTC William Lorey, ISAF HQ Procurement
Eng Ali, Managing Director, First Option Engineering Co.
Mrs Fazelah, Planning and Finance Manager, First Option Engineering Co.
Mohd. Homayun Azizi, Managing Director ETC Power Company
Hafizullah “Walirahimi”, General Director, Private Sector Development, Ministry of Commerce
& Industries