Sie sind auf Seite 1von 14

PICMET 2008 Proceedings, 27-31 July, Cape Town, South Africa (c) 2008 PICMET

Benefits Realisation Management in Information Technology Projects


D. C. Smith, H. Dombo, N. Nkehli
Department of Information Systems, University of Cape Town, South Africa

Abstract--This study evaluates the use of benefits II. IT VALUE


realisation management (BRM) in information technology
(IT) projects. Past research concludes that realising the Reference [13] defines value as the “impact of IT on
tangible and intangible benefits from IT projects is not easy organisational performance.” As such, value is often
and is often not carried out at all.
Fifty-four IT project managers in South Africa
defined in terms of strategic positioning, increased
successfully completed an online questionnaire. An analysis productivity, improved decision-making, cost-savings,
of the data showed that, whilst BRM processes exist in some benefits, payoffs or improved services, which are all
IT projects, benefits monitoring during the project is rarely measures of organisational performance [34]. The
applied even though planned benefits can change expected value to be delivered by IT often forms the
dramatically. It was also found that IT project managers are justification for large technology investments, and must
keen to improve BRM processes so that benefits are more therefore be measurable [13],[18]. However, IT value is
clearly defined and realised. They identify the project owner often embedded in activities and processes wherein IT has
as the person accountable for benefits realisation as the been integrated which makes it difficult to discern and
business benefits normally occur long after the project is
complete, but they argue that this is not happening. They also
measure the value added specifically by the
conclude that, because their performance is measured using implementation of IT [39].
project efficiency metrics (on time, on budget and on IT value is often measured using quantitative metrics,
specification), they are not required to focus on the project especially financial metrics such as return on investment
effectiveness (the longer term benefits from the IT (ROI), return on assets (ROA), return on equity (ROE),
investment). internal rate of return (IRR), net present value (NPV) and
payback period [5],[19],[32]. Quantitative measures of IT
I. INTRODUCTION value also include system performance metrics (e.g.
downtime per week; complaints per user; defects per user),
The success of IT projects is often based on whether IT operations metrics (e.g. response time for problem
objectives are met on time, on budget and on specification resolution; support staff per user; cycle time) and end-user
[38]. This focus is on the delivery of project artefacts metrics (e.g. amount of money spent per user for training;
rather than the targeted benefits that often form the actual IT usage) [15],[19]. However, these quantitative
justification for IT projects. As such, many IT projects methods have been criticised for their inability to measure
have failed to show the net benefits identified in the initial the intangible benefits that IT can deliver [19],[33]. These
project justification [13],[22],[34]. Reference [40] cites the measures also do not account for the risk associated with
following as some of the reasons for not attaining the net IT investments, nor variability in time, specifications and
benefits: budget that often accompany IT projects [32].
• The immediate results of an investment are rarely the
expected benefits III. PROJECT SUCCESS
• Benefits do not occur where and when they are planned
• The “right” benefits are difficult to identify upfront There are a number of factors that are essential for
every project’s success. These factors, sometimes known
Reference [41] suggests a benefits management as critical success factors (CSFs), are selected by the
approach that manages the planning and implementation relevant project stakeholders as the agreed metrics on
of IT projects such that the potential benefits from the which the project’s success will be measured [9]. As such,
projects are actually realised. Such an approach would project success factors are not universal for all projects.
involve the identification, planning, realisation and review Not all CSFs are used at the same time to measure each
of the benefits from IT projects [41]. project’s success and they may not be of the same
This research seeks to determine the extent to which IT importance [10]. This is because the selected CSFs may be
projects have formalised processes in place to ensure that relevant at different stages of the project and thus measure
the potential benefits identified prior to undertaking an IT the project’s success at the different stages [30]. Selecting
project are actually realised. CSFs that will be used to measure an IT project’s success
can be challenging due to the subjective nature of what
defines success to different stakeholders [30]. According
to Pinto and Slevin, as cited in [9], an obvious approach

1442
PICMET 2008 Proceedings, 27-31 July, Cape Town, South Africa (c) 2008 PICMET

would be to select CSFs that are simple, unambiguous and duration of the project, the underlying risks and the
easy to measure. required resources [31].

A. The Iron-Triangle B. Benefits Identification


In general, IT projects are often deemed successful if Benefits identification identifies and documents the
delivered on time, within the budget and in accordance potential benefits that can arise from an IT investment [4].
with user requirements [26]. These three CSFs are also The benefits identified should be both tangible and
known as the Iron Triangle. According to [1], most IT intangible benefits, each with a defined metric so that the
projects fail because they have set unrealistic goals in success of the project can be measured [37].
terms of the Iron Triangle. Should the possibility of However, it is difficult to identify such benefits as
renegotiation of project specifications arise, the benefits may change and evolve over time, intangible
development team often then fails to inform the user of the benefits may be hard to identify and IT projects benefits
possibility of not meeting the promised Iron Triangle goals are complex and can therefore be hard to document [4].
[1].
C. Benefits Planning
B. Dimensions of IT Project Success The benefits planning process maps implementation of
References [9] and [30] identified four dimensions of the IT investment and the conversion of identified benefits
IT project success. These are Benefits to the customer, into realised benefits, against the required changes in
Meeting design goals, Commercial success and Future business processes [7]. As such, the main purpose of
potential. benefits planning is to develop an approved business case
These success dimensions account for various for the investment [41]. The plan should outline the
stakeholders’ different points of view of success [9]. More planned benefits from the IT project and the specific
importantly, the different success dimensions are relevant means by which these benefits will be realised [3],[4]. The
to different types of IT projects with different levels of plan should also identify the beneficiaries of the benefits,
importance [30]. The different dimensions can be assessed how the identified benefits are linked to the final project
at different times with respect to project completion, output, and of course, the feasibility of achieving the
making the relative importance of each dimension time- identified benefits [[4],[41].
dependant [30]. Once the different success dimensions The plan should identify a person in the organisation
have been agreed upon, the project stakeholders should who is responsible for the delivery of the benefits as set
assign measures and weightings to them, and rate each out by the benefits realisation plan [4],[41]. During
dimension relative to the allocated weights[10]. development and implementation, more benefits may be
identified, in which case the responsible person will have
IV. BENEFITS REALISATION MANAGEMENT to revise the benefits realisation plan in consultation with
the stakeholders, in order to accommodate the newly
As organisations become more aware of the importance identified benefits [41].
of realising benefits from IT, there is an increasing need
for formalised benefits management processes or D. Measuring the Planned Benefits
frameworks to be adopted by organisations [3],[4]. Key Performance Indicators (KPIs) are a means of
The Benefits Realisation Management (BRM) process measuring the identified benefits of an IT investment [28].
manages an IT investment from pre-project evaluation Reference [4] identifies the five advantages of developing
through to post-project evaluation. In addition, Various KPIs in projects. They enable stakeholders to assess
authors highlight the need for business process change and whether the planned benefits of IT projects have been
for changes to be made by users within the organisation, in delivered; they Identify the project benefits to measure,
order to facilitate benefits realisation [3],[26],[17]. The and when to measure them; they Facilitatate action based
benefits realisation management process can be viewed as upon KPIs measurements; they clearly link accountability
a 4-step model developed by [4], with the additional post- to measured benefits and they assists th project in being
implementation evaluation step added by [3]. funded.
Similar to CSFs, KPIs differ from project to project and
A. Pre-Project Evaluation should be agreed upon beforehand [28]. KPIs can also be
Pre-project evaluation, also known as screening, used to define and measure goals in the benefits plan other
selection or project appraisal, requires an organisation to than the identified benefits [29].
identify their objectives, and do preliminary analysis of
whether an IT investment is worth pursuing [12]. This E. Benefits Monitoring
process therefore provides the justification for investing in Benefits monitoring is defined as a process that
an IT project [28]. Pre-project evaluation considerations “compares project results with the benefits realisation plan
include factors such as the project size, the expected during the project and assesses if internal and external

1443
PICMET 2008 Proceedings, 27-31 July, Cape Town, South Africa (c) 2008 PICMET

changes have occurred that will affect the delivery of of business managers to the performance of IT projects
planned benefits” [4]. This process needs to occur that they sponsor [24],[38].
throughout the project lifecycle, particularly at the various The project sponsor is responsible for the planning and
milestones in the benefits realisation plan, and not just organisation of all aspects of the benefits management
when the benefits review takes place [8],[34]. Monitoring cycle [27]. As such, the sponsor must work closely with
ensures that the IT investment is achieving the objectives the change management team and should be able to
that will bring about the identified benefits by providing a address any uncertainties that may arise regarding the
feedback mechanism through which it can be controlled achievement of project benefits [27],[41]. Most
[12]. importantly, the benefits owner should be in a senior
position and be influential enough to ensure that the
F. Benefits Realisation project team executes their responsibilities within the
Benefits realisation consists of ensuring that the project [41]. The responsibilities of the sponsor continue
benefits outlined in the benefits realisation plan are after the project completion, as the benefits may only be
actually realised [21]. This involves a comparison of the realised later [27].
realised benefits to the benefits outlined in the plan [4].
The realisation process uses the metrics and expected V. RESEARCH METHODOLOGY
values defined in the identification process to benchmark
the success of the IT investment [26],[34]. The research aimed to determine the extent to which
External and internal factors can affect whether or not benefits realisation management techniques were used in
benefits are realised. The external factors include practice in South Africa. This was done by evaluating the
competitive forces, government regulations and changes in following research question which was then evaluated for
the technological environment (e.g. competitor’s adoption each of the 5 BRM processes:
of a similar IT project; pricing regulations; emergence of Do IT projects have BRM processes in place to
substitute technologies for the IT investment) [7]. The ensure business benefits are realised and IT value is
internal factors include the extent to which management maximised?
promotes the implementation and usage of the IT
investments, thus garnering organisational support for the A quantitative research method was followed. This was
investment [7]. facilitated by the use of a cross-sectional survey as the
research instrument. The survey method, in the form of a
G. Post-Project Evaluation questionnaire, was deemed appropriate due to the fact that
Post-project evaluation is defined as the “process by it allows for generalisations about a population to be
which the success of the project in terms of benefit drawn from analysing sample data [6]. The questionnaire
delivery is assessed; opportunities for the realisation of was constructed in the form of an online survey, which
further benefits are identified; and lessons learned and was open for participation over a four-week period. The
opportunities for improvement in future projects are research instrument used was constructed based on
identified”[3]. Post-project evaluations are conducted once previous surveys developed by [4] and [23].
the project is complete with the aim of using the lessons The research instrument utilised was the
learned to better estimate future projects [12]. These SelectSurveyASP Advanced tool. The tool provided useful
evaluations also enable an organisation to determine functionality that facilitated data analysis. This includes
whether the expected benefits have actually been realised summaries of the survey data, which are presented
[22]. graphically and numerically in the form of graphs and
In practice, most organisations fail to conduct proper basic percentages respectively. The tool also allows for
post-project evaluations, if at all [17],[25]. Project teams filtering of data according to specified criteria. Therefore,
are often disbanded upon project completion, which the data could be filtered based on the respondents who,
decreases the likelihood of a post-project evaluation for example, answered that their organisations had a
[3],[12],[25]. Benefits Realisation methodology in place. The tool
exports data in electronic format into an Excel
H. Assigning Accountability for Benefits Realisation spreadsheet, which reduced the number of potential errors
Effective accountability can be achieved by defining a that could have arisen during data capturing.
clear mandate and scope for the project; clearly defining The target sample for the survey was IT project
the lines of accountability; using relevant performance managers in South Africa. Emails inviting project
measures; and aligning accountability with the reward managers to participate in the survey were sent to 251
system [38]. Furthermore, accountability can be promoted potential respondents. 67 project managers completed the
through holding business units independently accountable survey resulting in a response rate of 27%. However, 13
for their IT expenditure, and by linking the compensation responses had to be discarded, as they were incomplete.
These responses were therefore not used in the final

1444
PICMET 2008 Proceedings, 27-31 July, Cape Town, South Africa (c) 2008 PICMET

analysis. The eventual sample size was 54 IT project used in the instrument for these questions may not be
managers. Of the 54 valid responses, 31 project managers reliable.
requested feedback on the findings of the research.
The data exported from the online survey tool was VI. FINDINGS AND IMPLICATIONS
stored in Excel format and exported into Statistica for
statistical analysis. The data collected was analysed at a Q1: Do IT projects have pre-project evaluation
high-level using response frequency i.e. the percentage of methodologies to identify and specify metrics for the
total responses that selected a particular answer. proposed business benefits?
To assess if the relationships between responses were The survey revealed that 50% of the participants’
statistically significant, the data was analysed using non- organisations have IT investment evaluation
parametric statistical tests, which are appropriate for methodologies in place. When participants were asked
ordinal data, such as data gathered from Likert-type scales how effective they thought evaluation methodologies
[16]. In addition, non-parametric tests do not require data were, those who had evaluation methodologies in place in
to be normally distributed and they can be used for data their organisations see them as being effective, with 63%
gathered from a relatively small sample [35]. seeing them as frequently effective, while 22% see these
An item reliability analysis was run for each of the sets methodologies as being extensively effective as illustrated
of questions pertaining to pre-project evaluation/benefits in Figure 1. Therefore, in aggregate, evaluation
identification, benefits planning, benefits monitoring, and methodologies are effective for 85% of participants’
post-project evaluation. The Cronbach’s alpha values for organisations who have evaluation methodologies in place.
pre-project evaluation, benefits planning and post-project The response from those participants who did not have
evaluation, were all above 0.7. This indicates reliable evaluation methodologies in place in their organisations
scales for those constructs. However, the Cronbach’s alpha was expected, with 74% citing evaluation methodologies
for benefits monitoring was 0.43, indicating that the scales as not being effective at all as illustrated in Figure 1.

Figure 1 : Effectiveness of Formal IT Investment Methodologies

All respondents with these evaluation methodologies organisations have formal evaluation methodologies in
confirmed that these methodologies have a process that place, and if these methodologies include intangible
ensures the alignment of the proposed project with the benefits. 70% of the participants confirmed that their
overall business objectives (Q5 in Figure 2). Figure 2 also evaluation methodologies identify all possible benefits and
shows that 81% of participants with evaluation that the relevant benefits were adequately quantified. A
methodologies incorporate intangible benefits in their Mann-Whitney U test run on these variables supports this,
appraisal processes. However, the Spearman rank order showing that statistically this group believed that their
correlation run on these variables revealed that there was evaluation process identifies all available benefits for a
no statistically significant correlation between whether the project and adequately quantifies all the relevant benefits.

1445
PICMET 2008 Proceedings, 27-31 July, Cape Town, South Africa (c) 2008 PICMET

Figure 2: Elements of Pre-Project Evaluation Methodologies

Figure 3 shows that 72% of the participants’ recent correlation test shows that this is not a statistically
projects assigned targets to the benefits. Furthermore, 59% significant correlation.
of the participants’ recent projects assigned KPIs to When asked what they considered to be the advantages
intangible benefits. Of those participants whose recent of using KPIs, over 80% of the respondents generally
projects assigned KPIs to intangible benefits, 64% agreed that KPIs enabled stakeholders to assess if planned
reported that their projects assigned non-financial KPIs for benefits have been delivered. Furthermore, as can be seen
the intangible benefits. However, the Spearman rank in Figure 4, over 60% of the respondents feel that KPIs
identify project benefits to be measured.

Figure 3 : Benefits Identification Metrics

A Wilcoxon matched pairs test was run to see if there and that they allow management to take action based on
was a significant difference in the average responses to the measurement of these KPIs. This was in comparison to the
factors that respondents viewed as advantages of average agreement with advantages such as linking
developing KPIs for project benefits. The results of the test accountability to benefits and facilitating a better
seem to indicate respondents agreed more strongly that understanding of the project itself. However, participants
KPIs enable stakeholders to assess if the planned benefits generally seem to agree with all the advantages of
have been delivered, identify the benefits to be measured establishing KPIs for benefits.

1446
PICMET 2008 Proceedings, 27-31 July, Cape Town, South Africa (c) 2008 PICMET

Figure 4 : Advantages of KPIs

The results indicate that even if an IT project has a Due to the equal distribution of IT projects with pre-
formal evaluation methodology in place, this does not project evaluations, and those without such methodologies,
necessarily mean that the plan will identify intangible it is unclear whether IT projects generally have pre-project
benefits. This could be an indication of the difficulty that evaluation methodologies in place. However, in
organisations have in identifying the intangible value that organisations with these methodologies, they are generally
IT can deliver, which was highlighted by [4]. About 36% seen to be effective and all possible benefits are identified
of participants do not assign non-financial KPIs for the and quantified through these methodologies. Furthermore,
intangible benefits identified in their pre-project organisations seem to specify metrics to the proposed
evaluations, which implies that even if IT projects specify business benefits, although at times the nature of the
the intangible benefits, they may not necessarily be using metrics (i.e. financial or non-financial) may not be suitable
appropriate metrics. Based on the literature, the expected to the benefits.
results of the Spearman correlation would be a positive
relationship between projects that identify intangible Q2: Do IT projects have benefits planning procedures
benefits, and the nature of the KPIs assigned to these in place?
benefits. The survey revealed that 59% of the participants’
IT projects with a formal evaluation seem more likely organisations do not prepare a benefits delivery plan.
to identify all the available benefits that the project can When asked about the frequency with which elements of
deliver and assign KPIs to these benefits. These projects benefits planning are articulated, over half the respondents
also seem to quantify all the benefits from the project stated that the recipient of the benefits and where the
adequately. Responses to the advantages of developing benefits will occur are articulated on most occasions as
KPIs seem to indicate a heavier emphasis on using KPIs to illustrated in Figure 5. The Wilcoxon rank sum test to
assess if the planned benefits have been delivered, identify evaluate these factors revealed that where the benefits will
the benefits to be measured, and allow management to take occur and who receives the benefits were significantly
action based on measurement of the KPIs. This could more frequently specified than how the benefits are linked
mean that KPIs are not extensively used to link to deliverables and the action required by stakeholders to
accountability to the delivery of IT value, as was deliver the benefits. About a third of respondents
suggested by [12] and [26]. indicated that when the benefits will be realised is seldom
articulated, if at all.

1447
PICMET 2008 Proceedings, 27-31 July, Cape Town, South Africa (c) 2008 PICMET

Figure 5 : Elements of the Benefits Delivery Plan

There should be no significant difference in how will be realised, the recipients of the benefits seem to be in
frequently elements of benefits delivery are articulated. If place.
all elements are to be specified in the benefits plan, then
these elements should receive equal attention in all IT Q3: Is accountability for realising business benefits
projects. However, it seems that IT projects neglect to assigned to a specified project owner in IT projects?
specify the link between the benefits and deliverables, the Figure 6 shows the different roles suggested by IT
action required in order to deliver those benefits, and the project managers as the person responsible for the delivery
recipients of those business benefits. There appears to be a of project benefits. The role suggested the most as being
lack of formalised benefits planning IT projects, as only a accountable for the delivery of IT benefits was the project
third of the IT project managers indicated that they prepare sponsor. A diverse range of suggested roles, such as
a benefits delivery plan. However, the planning procedures program management, the business owner and the steering
such specification of when, where and how the benefits committee were grouped under ‘Other’ as shown in Figure
6. Respondents occasionally suggested more than one role.

Figure 6 : Who is Accountable for Project Benefits?

1448
PICMET 2008 Proceedings, 27-31 July, Cape Town, South Africa (c) 2008 PICMET

The benefits plan requires specification of the person As shown in Figure 8, just over half of the participants
responsible for delivering the project benefits [41]. Figure agree that KPIs provide the benefit of linking KPIs to
7 shows that 52% of participants state that the benefits accountability to benefits realisation. About 79% of IT
delivery plan generally articulates the party responsible for project managers agreed that post-project benefits reviews
delivery of benefits, with 17% of that being always. ensure that the party responsible for realising project
benefits are held accountable.

Figure 7 : Accountability in the Benefits Delivery Plan

The results indicate that there is accountability for the suggested by project managers were varied. The literature
realisation of benefits from IT projects. This person is identifies the project owner. The project manager should
specified in the benefits planning process on most not be the accountable party, because project teams are
occasions. However, there does not seem to be a consistent often disbanded upon project completion [3].
view of who should be held accountable. The roles

Figure 8: Accountability in Pre-Project Evaluation and Post-Project Benefits Review

1449
PICMET 2008 Proceedings, 27-31 July, Cape Town, South Africa (c) 2008 PICMET

Q4: Do IT projects have benefits monitoring test, showing significantly stronger agreement with the
procedures in place? reasons that there is a lack of business awareness and there
The results showed that 63% of the participants’ are few tools to help manage benefits, than compared to
organisations hold formal reviews of activities. In trying to the reason that changing the benefits could threaten the
establish reasons why organisation do not conduct benefits viability of the project. IT projects seem to place heavier
monitoring, the results as depicted in Figure 9 show an emphasis on managing the deliverable rather than the
85% agreement with the statement that organisations do benefits of the project. This supports the view that projects
not monitor project benefits because the focus is on place greater emphasis on delivering the technology rather
managing the project deliverables. than the IT benefits [3]. Meeting the goals of the Iron
50% of participants generally agreed that lack of Triangle seems to be heavily entrenched in the objectives
experience, few tools to help manage benefits and lack of of IT projects.
focus on the people who enjoy the benefits are some of the In general, IT projects seem to have benefits
reasons why organisations do not monitor project benefits, monitoring procedures. However, there is strong sentiment
while roughly 10% more felt that a lack of business that where monitoring does not occur, it is due to a lack of
awareness contributed to a lack of benefits monitoring. a benefits-oriented focus.
This is supported by the results of a Wilcoxon rank sum

Figure 9 : Reasons for a Lack of Benefits Monitoring

Q5: Do IT projects have post-implementation reviews whether there is a formal process in place to identify
of business benefits derived from the project? further benefits after implementation, and whether
Results show that 56% of participants’ organisations organisations take action after implementation to realise
have a formal process in place that ensures that lessons these further benefits. This indicates a very strong positive
learned from successful (or unsuccessful) implementations relationship between the two factors.
are transferred to future projects. However, 74% of As shown in Figure 10, survey respondents show
participants’ organisations do not have a formal process in general agreement with the proposed reasons for
place that identifies any further benefits after performing post-project benefits reviews. In particular,
implementation. Of the 26% that had such a process in providing experience for future projects has the highest
place, all of them indicated that their organisations take agreement, with 89% of respondents showing either
action after implementation to realise these additional agreement or strong agreement. Respondents also tended
benefits. This is supported by the Spearman rank to agree that reviews should be conducted to identify the
correlation, which indicates a particularly strong positive project benefits attained, and those that were not achieved.
correlation coefficient (0.74) between variables measuring Statistically, project managers seem to agree more strongly

1450
PICMET 2008 Proceedings, 27-31 July, Cape Town, South Africa (c) 2008 PICMET

that organisations should conduct post-project evaluations When asked why organisations do not perform post-
in order to provide experience for future projects and to project benefits reviews, 56% of participants agree that
allow senior management to review the results of their there is pressure to deliver other projects and many IT
decisions for IT. The mean responses for these reasons benefits are intangible. Interestingly, roughly a third of the
were significantly higher than for reasons such as participants do not view the benefits review process as
maximising the benefits and identifying any unexpected being too costly, unnecessary or against organisational
benefits. culture. These results are shown in Figure 11.

Figure 10 : Reasons for Formal Benefits Reviews after Project Completion

Figure 11 : Reasons for NOT Reviewing Benefits after Project Completion

1451
PICMET 2008 Proceedings, 27-31 July, Cape Town, South Africa (c) 2008 PICMET

A Wilcoxon rank sum test showed that compared with strong grounds for conducting a post-project evaluation.
the rest of the reasons cited, the strongest agreement was However, it is interesting that the responses seem to be
with the reason that organisations are under pressure to skewed towards lessons learnt, rather than measurement of
deliver other projects. The responses to this had identified benefits against realised benefits.
statistically higher mean compared to the rest of the Just over half of the respondents have post-project
reasons, which according to the coding of the answers evaluations in place, which makes it difficult to develop a
indicates strong agreement. Project managers also showed general conclusion. In addition, about three quarters of the
statistically stronger agreement with the reason that many participants indicate that their organisations do not identify
IT benefits are intangible compared to their responses to further benefits after implementation. The expected
the reasons that it is too costly to perform a post-project benefits from an IT project rarely occur immediately upon
evaluation, it is against organisational culture or that it is project completion [38]. If the majority of organisations
not necessary to perform a post-project evaluation. In fact, are failing to track the benefits after project completion, it
IT project managers seem to disagree that post-project will be difficult to evaluate if the IT project has actually
evaluation are not conducted because they are added business value or not.
unnecessary. As the question was worded negatively, this
implies that they feel the opposite, i.e. post-project VII. GENERAL QUESTIONS
evaluations are necessary. This is evident by the fact that
the mean response to this reason is significantly lower than The general questions in the survey reveal that 56% of
the rest of the reasons. the participants’ organisations have benefits realisation
IT project managers seem to recognise that performing methodologies in place. Of those participants whose
post-project evaluations is a necessary and vital process. organisations have benefits realisation methodologies in
However, it surprising that there are statistically place, 47% and 23% think that these methodologies are
significant differences in the respondents’ level agreement frequently and extensively effective, respectively, as
with reasons for performing these evaluations. Surely, all shown in Figure 12. As expected, the majority, i.e. 71%, of
the reasons proposed, such as maximising the benefits, the participants whose organisations do not have benefits
providing experience for future projects and assessing realisation methodologies in place, think that these
whether benefits were actually realised should provide methodologies are not at all effective.

Figure 12 : Effectiveness of IT Benefits Realisation Methodologies

Where organisations have benefits realisation Figure 14 shows the presence of BRM elements in
methodologies in place, it was interesting to note which organisations that do not have a formal benefits realisation
elements of the benefits realisation management process methodology. 92% of organisations who do not have a
were actually present. Figure 13 shows that 83% of the benefits realisation methodology in place also do not have
participants’ organisations have pre-project evaluation a pre-project evaluation methodology in place.
methodologies in place, 53% of them have benefits Furthermore, 75% of these organisations do not have a
delivery planning, 73% of them perform benefits benefits delivery plan and 62% do not perform post-
monitoring during implementation, and 70% of the project reviews. However, there is a 50/50 split between
participants’ organisations perform post-project reviews. those with a monitoring process and those without it.

1452
PICMET 2008 Proceedings, 27-31 July, Cape Town, South Africa (c) 2008 PICMET

Figure 13 : The Presence of the Elements of BRM in Organisations WITH Benefits Realisation Methodologies

A Spearman rank correlation was run to determine in place, with a correlation coefficient of 0.75.
whether there is a relationship between if an organisation Surprisingly, there appears to be no significant relationship
has a benefits realisation methodology, and if the between the presence of a benefits realisation
organisation has the elements of BRM in place. There is a methodology and having a monitoring process in place.
particularly strong positive correlation between whether an These variables are all positively correlated, which
organisation has a benefits realisation methodology and if supports the research literature findings.
the organisation has a pre-project evaluation methodology

The Presence of BRM Elements in Organisations


without
92
Benefits Realisation Methodologies
100
Survey Responses (%)

75
80 62
60 50 50
38
40 25
20 8
0

BRM Elements
YES NO

Figure 14 : The Presence of BRM Elements in Organisations WITHOUT Benefits Realisation Methodologies

Figure 15 shows overwhelmingly that participants feel senior management doubts that IT delivers real business
that organisations could improve their benefits benefits and that benefits are often overstated to gain
management processes. In aggregate, 58% of participants project approval. This indicates stronger support for those
agree that senior management has doubts that IT delivers statements.
real business benefits. This is further supported by It is of concern that project managers agree with the
statistical evidence that respondents show a statistically fact that senior management doubts the value delivered by
stronger agreement with the suggestion that organisations IT projects. Senior management support was cited by the
can improve their benefits management process. Chaos Reports in both 1994 and 2000 as a project success
Compared to the perception that it is becoming factor [14],[36]. The literature also recommends a sponsor
increasingly difficult to predict benefits in advance, project with links to executive management [11],[34].
managers had statistically higher mean responses that

1453
PICMET 2008 Proceedings, 27-31 July, Cape Town, South Africa (c) 2008 PICMET

Management Considerations
Survey Question: Based on your experience, please rate the
following.

It is becoming increasingly difficult to


4 30 31 28 7
predict benefits in advance

Benefits are often overstated to gain


2 19 26 41 13
project approval

Senior management has doubts that IT


2 26 15 28 30
delivers real business benefits

Organisations could improve their benefits


07 56 37
management process

0 10 20 30 40 50 60 70 80 90 100
Survey Responses (%)
Strongly Disagree Disagree Neutral Agree Strongly Agree

Figure 15 : Management Considerations

Activities such as pre-project evaluation, benefits management process in place. The focus is still largely on
planning and post-project evaluation determine whether an delivering the artefacts of IT projects - on time, on budget
organisation has a benefits management methodology in and according to specifications – rather than on delivering
place. The positive nature of this relationship implies that business benefits. IT project managers seem to recognise
if these activities are performed in the organisations, it is the need for an improvement in benefits realisation
likely that the organisation will have a formalised benefits management techniques but that will obviously require a
realisation methodology in place. This is particularly true change in organisational mindset as to the value that IT
for those organisations with pre-project evaluation can deliver.
processes in place. However, monitoring whether the Further research could be qualitative in nature, looking
benefits plan is being adhered to and if the interim benefits specifically at how organisations have implemented BRM.
are being realised, does not seem to affect the presence of In addition, further research could also examine specific
a benefits realisation methodology. elements of BRM processes, such as how pre-project
evaluations account for intangible benefits, how
VIII. CONCLUSION accountability is assigned, why benefits monitoring does
not seem to be a priority, and the reasons for the lack of
The research aimed to determine the extent to which post-project reviews of the benefits.
benefits realisation management is applied in IT projects.
The theoretical basis to benefits realisation management REFERENCES
was established in the literature review, outlining key
aspects required in the management of benefits from IT [1] Ambler, S. W.; The "Broken Iron Triangle" Software Development
Anti-Pattern. Retrieved August 8, 2007, from
projects. The focus of the research was thus geared at http://www.ambysoft.com/essays/brokenTriangle.html#Impact
determining if these elements were actually used in [2] Andresen, J., M. Betts, C. Carter, A. Hamilton and E. Stokes, A
practice. Framework for Measuring IT Innovation Benefits. ITcon , Vol. 5,
The results of the research indicate that IT projects in pp. 57-72, 2000.
[3] Ashurst, C. and N.F. Doherty, Towards the Formulation of a 'Best
South Africa do not fully utilise the identified processes of Practice' Framework for Benefits Realisation in IT Projects.
benefits realisation methodology. Instead, projects seem Electronic Journal of Information Systems Evaluation , Vol. 6 (2),
to be selective in their application of the elements of pp. 1-10, 2003.
benefits realisation management. The results also indicate [4] Bennington, P. and D. Baccarini, Project Benefits Management in
IT Projects - An Australian Perspective. Project Management
that while organisations may have the selected benefits Journal , Vol. 35 (2), pp. 20-30, 2004.
management elements in place, this may not necessarily [5] Chan, Y. E.; IT Value: The Great Divide between Qualitative and
mean that they have a formalised benefits realisation Quantitative and Individual and Organizational Measures. Journal

1454
PICMET 2008 Proceedings, 27-31 July, Cape Town, South Africa (c) 2008 PICMET

of Management Information Systems , Vol. 16 (4), pp. 225-261, [24] Lohmeyer, D., S. Pogreb and S. Robinson, Who's Accountable for
2000. IT? The McKinsey Quarterly Special Edition: Technology , pp. 39-
[6] Creswell, J. W.; Research Design: Qualitative and Quantitative 47, 2002.
Approaches. California, USA: Sage Publications, 1994. [25] McAvoy, J.; Evaluating the Evaluations: Preconceptions of Project
[7] Davern, M. J. and R.J. Kauffman, Discovering Potential and Post-Mortems. The Electronic Journal of Information Systems
Realizing Value from Information Technology Investments. Evaluation , Vol. 9 (2), pp. 65-72, 2006.
Journal of Management Information Systems , Vol. 16 (4), pp. 121- [26] Peppard, J. and E.Daniel, Managing the Realization of Business
143, 2000. Benefits from IT Investments. MIS Quarterly Executive , Vol. 6 (1),
[8] DellaVecchia, T., S. Scantlebury and J.G. Stevenson, Three CIO pp. 1-11, 2007.
Advisory Board Responses to "Managing the Realization of [27] Reiss, G., M. Anthony, J. Chapman, G. Leigh, A. Pyne and P.
Business Benefits from IT Investments". MIS Quarterly Executive , Rayner, Gower Handbook of Programme Management. Aldershot,
Vol. 6 (1), pp. 13-16, 2007. England: Gower Publishing Ltd, 2006
[9] Dvir, D., S. Lipovetsky, A.J. Shenhar and A. Tishler, In Search of [28] Sakar, P. and C. Widestadth, Benefits Management - How to
Project Classification: A Non-Universal Approach to Project Realize the Benefits of IS/IT Investments. Unpublished Masters
Success Factors. Research Policy , Vol. 27, pp. 915-935, 1998. Thesis, Göteborg University, Department of Informatics, Göteborg,
[10] Dvir, D., S. Lipovetsky, A.J. Shenhar J. and A.Tishler, What is 2005.
Really Important for Project Success? A Refined, Multivariate, [29] Saloojee, R.; Investigating the Business Value of Information
Comprehensive Analysis. International Journal of Management Technology in South Africa. Unpublished Masters Dissertation,
and Decision Making , Vol. 4 (4), pp. 382-404, 2003. University of Johannesburg, Faculty of Business Management,
[11] Englund, R. L. and A. Bucero, Project Sponsorship: Achieving Johannesburg, 2006.
Management Commitment for Project Success. San Francisco, [30] Shenhar, A. J., D. Dvir, O. Levy, and A.C. Maltz, Project Success:
USA: Jossey-Bass, 2006. A Multidimensional Strategic Concept. Long Range Planning , Vol.
[12] Frame, J. D.; Lessons Learned: Project Evaluations. In P. W. 34, pp. 699-725, 2001.
Morris, & J. K. Pinto (Eds.), The Wiley Guide to Managing Projects [31] Shtub, A., J.F. Bard and S. Globerson, Project Management:
(pp. 1197-1213). New Jersey, USA: John Wiley & Sons, Inc., 2004. Processes, Methodologies, and Economics (2nd ed.). New Jersey,
[13] Goh, K. H. and R.J. Kauffman, An Industry-Level Analysis of the USA: Pearson Education, Inc., 2005.
Potential and Realized Value of IT. Twenty-Seventh International [32] Silvius, A. J.; Does ROI Matter? Insights into the True Business
Conference on Information Systems, (pp. 1841-1860). Milwaukee, Value of IT. The Electronic Journal of Information Systems
2006. Evaluation , Vol. 9 (2), pp. 93-104, 2006.
[14] Johnson, J., K.D. Boucher, K. Connors and J.Robinson, The [33] Sircar, S., J.L. Turnbow and B. Bordoloi, A Framework for
Criteria for Success. Software Magazine , Vol. 21 (1), pp. 3-11, Assessing the Relationship Between Information Technology
2001. Investments and Firm Performance. Journal of Management
[15] Kanungo, S.; Sustainable Benefits of IT Investments: From Concept Information Systems , Vol. 16 (4), pp. 69-97, 2000.
to Implementation. Proceedings of the Tenth Americas Conference [34] Smith, H. A. and J.D. McKeen, Developments in Practice VII:
on Information Systems, New York, (pp. 925-933). New York, 2004. Developing and Delivering the IT Value Proposition.
[16] Keller, G.; Statistics for Management and Economics (7th ed.). Communications of the Association for Information Systems , Vol.
California, USA: Thomson Brooks/Cole, 2005. 11, pp. 438-450, 2003.
[17] Kohli, R. and S. Devaraj, Realizing the Business Value of [35] StatSoft, Inc.; Electronic Statistics Textbook. Retrieved September
Information Technology Investments: An Organizational Process. 4, 2007, from StatSoft Statistica:
MIS Quarterly Executive , Vol. 3 (1), pp. 53-68, 2004. http://www.statsoft.com/textbook/stnonpar.html#index
[18] Kohli, R. and S.A. Sherer, Measuring Payoff of Information [36] The Standish Group International.; The CHAOS Report (1994).
Technology Investments: Research Issues and Guidelines. Retrieved August 31, 2007, from
Communications of the Association for Information Systems , Vol. http://www.standishgroup.com/sample_research/chaos_1994_2.php
9, pp. 241-268, 2002. [37] Thiry, M.; Program Management: A Strategic Decision
[19] Lee, I.; Evaluating Business Process-Integrated Information Management Process. In P. W. Morris, & J. K. Pinto (Eds.), The
Technology Investment. Business Process Management Journal , Wiley Guide to Managing Projects. New Jersey, USA: John Wiley
Vol. 10 (2), pp. 214-233, 2004. & Sons, Inc., 2004.
[20] Lin, C. and G.P. Pervan, A Review of IS/IT Investment Evaluation [38] Thorp, J.; A Benefits Realization Approach to IT Investments. In
and Benefits Management Issues, Problems and Processes. In W. W. Van Grembergen, Information Technology Evaluation Methods
Van Grembergen, Information Technology Evaluation Methods and and Management (pp. 25-43). Hershey, USA: Idea Group
Management (pp. 2-24). Hershey, USA: Idea Group Publishing, Publishing, 2001.
2001. [39] Tillquist, J. and W. Rogers, Using Asset Specificity and Asset
[21] Lin, C., G. Pervan. and K.H. Lin, A Survey on Evaluating and Scope to Measure the Value of IT. Communications of the ACM ,
Realizing IS/IT Benefits in Taiwanese B2BEC Companies. Vol. 48 (1), pp. 75-80, 2005.
European Conference on Information System, 2004. [40] Truax, J.; Investing with Benefits in Mind: Curing Investment
[22] Lin, C., G. Pervan and D. McDermid, A Framework on Evaluating Myopia. The DRM White Paper , pp. 1-6, 1997.
and Managing the Benefits of IS/IT. Pacific Asia Conference on [41] Ward, J. and E. Daniel, E., Benefits Management: Delivering Value
Information Systems, (pp. 644-658). Tokyo, 2002. from IS & IT Investments. West Sussex, England: John Wiley &
[23] Lin, K. H., C. Lin and H-Y Tsao, IS/IT Investment Evaluation and Sons Ltd., 2006.
Benefit Realization Practices in Taiwanese SMEs. Journal of
Information Science and Technology , Vol. 2 (4), pp 45-71, 2005.

1455