A REVIEW OF INFORMATION TECHNOLOGY INVESTMENT EVALUATION METHODOLOGIES: THE NEED FOR APPROPRIATE EVALUATION METHODS

A REVIEW OF INFORMATION TECHNOLOGY INVESTMENT EVALUATION
METHODOLOGIES: THE NEED FOR APPROPRIATE EVALUATION METHODS


Benny Ranti, ranti@infosolusi.co.id

Graduate Program in Information Technology, University of Indonesia
Salemba Campus, PUSILKOM UI Building, Jalan Salemba Raya No. 4, Jakarta 10430


ABSTRACT

For years, many academics and practitioners have been debating about the worth or business value of information
technology (IT) and information systems (IS) investment. Despite large investments in IT over many years, some
studies said that IT investment produces insignificant benefits known as the “productivity paradox”. Later, the
paradox has been declined by other studies reporting that there appears positive relationship between IT investment
and organization performance.
The difficulties in measuring especially benefits are often the cause for the uncertainty about the expected benefits of
IT investments. Organizations seeking positive relationship between organization performance and IT investment
have often disregarded the IT investment evaluation because of its intangible and difficult nature. Or if it was done,
the evaluation was performed inefficiently and ineffectively because of the absent of the appropriate evaluation
methods.
By taking into account the positive impact of IT investment, this paper reviews the model, type, and benefit of IT
investment, as well as the evaluation methodologies. This paper suggests that the development of a more complete
evaluation method is highly needed because it is said that “cannot measure, cannot manage”.

Keywords: IT investment, organization performance, IT evaluation method



1. MODEL, TYPE, AND BENEFIT
continue to be an important research concern for both

practitioners and academics [2].
1.1. Model of IT Investment
The most commonly cited causes of the difficulty to

appraise IT investments as brought up by Giaglis [3]:
The main logical notion of a company investing
1. The intangible nature of the benefits.
money
in
information
technology
(IT)
and
2. The benefits of IT are realized in the long run.
information systems (IS) solution is to see a positive
3. Strategic and competitive advantages are
impact to the organization performance. In the past
inherently difficult to quantify.
thirty years there have been some IT investment
4. The benefits of IT are indirect and therefore
models introduced by researchers. The Synthesized
indistinguishable from several confounding
Model of IT Investment developed by McKeen et al
factors.
[1] combined all the above mentioned models.
5. The theories and techniques available are
This model shows a chain of transformational process
inappropriate for understanding and capturing
of IT investment before impacting the organization
the value of IT.
performance. The process is started with the form of

IT governance that must be defined either on project
1.2. Type of IT Investment
or on-going basis. This leads to a specific investment

and deployment before IT can be used by the
From business perspective there are many different
organization to enhance its performance. The level
reasons for investing in IT which lead to some types
of conversion effectiveness (i.e., the mix of internal
of IT investments. Each type of IT investment
managerial and external environmental factors)
requires different evaluation criteria depending on the
determines the degree to which each of these
purpose of the investment and the benefits to be
variables translates into organizational performance.
achieved.
Determining the impact of IT investments on
organizational performance has been and will
Prosiding Konferensi Nasional Teknologi Informasi & Komunikasi untuk Indonesia
112
3-4 Mei 2006, Aula Barat & Timur Institut Teknologi Bandung

---

Based on the combination of IT Investment Matrix
introduced among others by Remenyi [6] and Parker
by Lucas [4] and Zee’s classification [5], the type of
[7][8], 4 (four) types of IT benefits called IT Benefit
IT investment can be defined as follows:
Matrix (following Remenyi’s term) have been
1. Mandatory IT. Required to satisfy regulatory
produced as follows:
requirements, to meet internal organizational
1. Easy-to-Quantify Tangible (EQT) or Hard
requirements, and to provide IT applications that
Benefit: IT benefits which directly affect the
are competitive necessity. For example: changes
company’s profitability and the effect can be
to the payroll system, financial consolidation
directly and objectively measured. For example:
system for a multinational company, and join an
reduced cost and increased revenue.
industry-specific EDI service.
2. Hard-to-Quantify Tangible (HQT): IT benefits
2. Efficiency and Effectiveness Improvement.
which can also be seen to directly affect the
Aimed at reducing or avoiding operational and
company’s profitability, but the precise effect
labor costs, increasing business productivity and
cannot be directly measured. For example: better
revenue, and monitoring business activities. For
information and improved the company’s
example: IT may contribute in various ways and
security.
its contribution can be evaluated with measures
3. Easy-to-Quantify Intangible (EQI): IT benefits
of cost, quality, and speed of internal business
which can be measured, but the impact does not
processes.
necessarily directly affect the company’s
3. Strategic IT. Aimed at gaining a sustainable
profitability. For example: increased customer
advantage over competitors and improving the
satisfaction and improved staff satisfaction.
organization’s share of, or position in, existing
4. Hard-to-Quantify
Intangible
(HQI)
or
and new markets. For example: the application
Strategic/Soft Benefit: IT benefits which cannot
of IT as a product or as a service, often involving
be easily measured and the impact does not
Internet functionality.
necessarily directly affect the company’s
4. IT Infrastructure. These are investments in the
profitability. For example: improved market
basis on which IT applications are built. They do
reaction and increased company image.
not offer direct benefits, but enable the benefits
HQT and EQI can be said as a derivation of quasi
of other IT investments to be realized. For
intangible.
example: WAN.

5. IT Research. Executed to ensure that the
2. EVOLUTION OF IT EVALUATION
business is not left behind by technological

progress. For example: in various ways.
In line with the development of IT investment model
6. Transformational IT. Must be combined with
in the past thirty years, researchers also proposed
changes in management philosophy, good for
methodologies for evaluating or measuring IT
fast response organization (risky to change
business value. There may be a number of objectives
structure, but high potential rewards). For
for IT investment evaluation including [2]:
example: Virtual Organization.

1. As part of the process of justification for a
1.3. Benefit of IT Investment
project.

2. Enabling an organization to make comparisons
In line with the strong hope of CEO and senior
of the merit of a number of different investment
management to see real improvements to the business
projects competing for limited resources.
performance as a result of IT/IS investments, the
3. Providing a set of measures which enable the
concept of IT benefit has been expanded to IT value.
organization to exert control over the investment.
The definition of IT value is to always relate IT
4. Enabling organizations to gain competitive
benefits to business performance. In other words, all
advantage, to develop new business, to improve
benefits produced must be quantified or measured
productivity and performance, as well as to
economically. Henceforth, the term IT benefit is used
provide new ways of managing and organizing.
interchangeably with IT value with the meaning of IT
In general, IT investment evaluation (or IT evaluation
value.
in short) methodologies have been influenced by
So far there have been 2 (two) types of IT benefits
quantitative, qualitative, or combined approaches.
commonly used or known, i.e., tangible and
Cronk and Fry [9] conducted a research in the
intangible. In between there are quasi intangibles.
evolution of IT evaluation methodologies. They
The increasing need to assess and measure IT benefit
roughly divided the evolution into four phases based
has produced a more detailed type of IT benefits.
upon:
Based on observation of types of IT benefits
Prosiding Konferensi Nasional Teknologi Informasi & Komunikasi untuk Indonesia
113
3-4 Mei 2006, Aula Barat & Timur Institut Teknologi Bandung

---

1. Type of measurement. Can be qualitative or
3. Balanced Scorecard (BSC). [11][12]
quantitative type of measurement.
BSC is a set of financial and operational
2. Level of measurement. Can be at system,
measures that provide a balanced presentation of
process, or organizational level.
both the financial and operational impacts of a
3. Degree of complexity.
system giving senior managers a comprehensive
Starting as early in the mid 1960s which focused on
view of a system's value. BSC focuses attention
efficiency at the system level, the fourth phase,
on four areas or perspectives that are most
starting at the end of 1990s, introduced the so-called
critical to any business: finance, customers,
multi-dimensional methodology. This methodology
internal processes, and employee learning and
combined the strengths of each type of measurement
growth. Additional information was taken from
(qualitative
and
quantitative)
and
level
of
Robinson [15].
measurement (system, process, and organization).
4. Economic Value Added (EVA). [11][14]
Some of multi-dimensional methodologies fit in this
As a metric, EVA equals net operating profit
phase
including
Information
Economics
and
minus appropriate capital charges. Using EVA as
Balanced Scorecard.
a yardstick to assess the performance of
individual departments, including IT, on a

monthly, quarterly and yearly basis can help with
3. IT EVALUATION METHODOLOGIES
decisions on new projects. Conflicting and

confusing goals (like revenue growth, market
The developments of IT evaluation methodologies in
share or cash flow) are replaced with a single
the last few decades have produced dozens of
financial measure for all activities. Additional
methods both unclassified and classified by authors
information was taken from Shand [16].
into a number of ways, e.g., qualitative, probabilistic,
5. Return on Management (ROM). [6][13]
tangible, intangible, and risk.
ROM measures the impact of IT on business unit
Based on observation of quite a number of
performance based on the added value to an
methodologies, the following briefly explained multi-
organization provided by management. The
dimensional methodologies are the most cited ones
assumption is that in the modern organization,
(excluding by the authors/creators):
information costs are the costs of managing the
1. Information Economics (IE). [11][12][13]
enterprise. If ROM is calculated before and after
IE is a variant of cost-benefit analysis, tailored to
IT is applied to an organization, then the IT
cope with the particular intangibles and
contribution to the business, which is so difficult
uncertainties found in IT projects. IE retains ROI
to isolate using more traditional measures, can be
calculations for those benefits and costs which
assessed. ROM defines the management’s value-
can
be
directly
determined
through
a
added as everything left after subtracting all the
conventional cost-benefit analysis. However, the
direct operating costs from the value-added due
decision making process is based on a ranking
to direct labor. ROM proposes an index of the
and scoring technique of intangibles and risks
total performance of management due to the
factors associated with the IT investment. IE
introduction of IT. The index is obtained by
identifies IT performance measures and use them
dividing the management value-added by the
to rank the economic impact of all the changes
costs of management. ROM focuses on the most
on the organization’s performance caused by the
important impact of IT, i.e., on the value-added
introduction of IT. Surrogate measures are often
by management generated by the IT in excess of
used for most intangible.
management’s total costs.
2. Real Options Valuation (ROV). [11][13][14]
6. Multi-Objective Multi-Criteria (MOMC).
ROV aims to put a quantifiable value on
[6][13]
flexibility. The technique was applied to leasing,
MOMC explicitly recognizes the existence of
mergers and acquisitions, and manufacturing.
many points of view and more than one set of
ROV uses three basic types of data: current and
values in the decision to invest in IT. MOMC
possible future business strategies; the desired
does not rely on monetary measurements of
system capabilities sought by the company, and
value, instead it works via an iterative procedure
the relative risks and costs of other IT choice that
to establish preferences and utilities. Where there
could be used. ROV can help assess the risks
are many stakeholders, the best IT investment is
associated with IT investment decisions, taking
that which will deliver the highest aggregate
into account that business strategies and system
utility or which provides the highest overall
requirements may change.
measure of satisfied preferences.


Prosiding Konferensi Nasional Teknologi Informasi & Komunikasi untuk Indonesia
114
3-4 Mei 2006, Aula Barat & Timur Institut Teknologi Bandung

---

4. CONCLUDING REMARKS
Contemporary
Approaches,
Idea
Group

Publishing, 1999.
Although IT evaluation methodologies have been
[2] Chad Lin and Graham P. Pervan, A Review of
progressing considerably in the last few decades,
IS/IT Investment Evaluation and Benefits –
moving from system level, quantitative efficiency
Management Issues, Problems, and Processes,
metrics to complex multi-dimensional ones, much
Chapter 1, Idea Group Publishing, 2001.
remains to be uncovered. For example, the causal
[3] George M. Giaglis et al, “The I.S.S.U.E
relationships issue brought by Cronk [10], which
Methodology for Quantifying Benefits from
suggests that the IT value outcome should be a result
Information
Systems”,
Department
of
of the interaction among the system, the user and the
Information Systems and Computing Brunel
organization (called Holistic Construal).
University, United Kingdom, 1999.
According to Remenyi [6], each methodology leads
[4] Henry C. Lucas, Jr., Information Technology and
to the development of a measure or metric which
the Productivity Paradox – Assessing the Value
allows IT to be evaluated. Sometimes metric is
of Investing in IT, Oxford University Press, 1999.
compared to a corporate or industry standard or
[5] Han van der Zee, Measuring the Value of
sometimes relative metrics are compared for
Information Technology, IRM Press, 2002.
competing systems. However, in most cases a single
[6] Dan Remenyi et al, Effective Measurement and
measure is not sufficient to make an evaluation. Two
Management
of
IT
Costs
and
Benefits,
or three metrics combined will usually be required
Butterworth-Heinemann, 1995.
but six or seven measures should be avoided.
[7] Marilyn M. Parker, Strategic Transformation
and Information Technology – Paradigms for
It can be concluded that the main constraint in
Performing While Transforming, Prentice-Hall,
evaluating IT investment is the lack of any single
1996.
appropriate method for understanding and capturing
[8] Marilyn M. Parker and Robert Benson,
the value of IT. This is to confirm Giaglis’ fifth
Information Economics – Linking Business
statement [3] as explained before. Thus, the research
Performance
to
Information
Technology,
in developing a more complete IT investment
Prentice-Hall, 1988.
evaluation method is highly needed. In doing that, in
[9] Marguerite Cronk and Graham Fry, “IT
order not to “reinvent the wheel”, combining some
Evolution:
how
far
have
we
come?”,
existing methods or improving one existing method
http://www.sabusinessreview.co.za, July 2001.
by adding the strengths from other methods to
[10] Marguerite Cronk, “Understanding Information
overcome its weaknesses, can be the effective way to
System Evaluation through Holistic Construal”,
move forward.
http://www.sabusinessreview.co.za,
December
Based on the writer’s experience in using Information
2000.
Economics (IE), the following brief scenario might
[11] Tracy Mayor, “A Buyer’s Guide to IT Value
be applied to improve or enhance IE to become a
Methodologies”, CIO Magazine, July 15, 2002.
more appropriate methodology:
[12] Vector Research Incorporated, “Guide for
1. Financial Approach. The original simple ROI
Managing Information Technology (IT) as an
calculation of IE can be replaced with a more
Investment Measuring Performance”, Prepared
rigorous EVA or ROV measurements.
for Department of Defense the United States of
2. Non-Financial
Approach.
The
original
America, 1997.
Business and Technology Domains of IE can be
[13] Joseph Wen and Cheickna Sylla, in Mo Adam
complemented with Human Domain to fulfill
Mahmood and Edward J. Szewczak (eds),
Cronk’s Holistic Construal [10].
Measuring Information Technology Investment
By limiting only to the 6 (six) above methodologies,
Payoff: Contemporary Approaches, Idea Group
there are numerous ways in combining strengths of
Publishing, 1999.
each methodology. It definitely needs an in-depth and
[14] Sarv Devaraj and Rajiv Kohli, The IT Payoff –
continuing research to produce the better, the next
Measuring the Business Value of Information
better ones, and so forth.
Technology Investments, Prentice-Hall, 2002.

[15] Robin A. Robinson, “Balanced Scorecard”,
5. R
Computer World, January 24, 2000.
EFERENCES

[16] Dawne Shand, “Economic Value Added”,
[1] James D. McKeen et all, in Mo Adam Mahmood
Computer World, January 24, 2000.
and Edward J. Szewczak (eds), Measuring

Information Technology Investment Payoff:

Prosiding Konferensi Nasional Teknologi Informasi & Komunikasi untuk Indonesia
115
3-4 Mei 2006, Aula Barat & Timur Institut Teknologi Bandung

---