A Balanced Scorecard Approach to Enterprise Systems Performance Measurement

Proceedings of the Twelfth Australasian Conference on Information Systems
A Balanced Scorecard Approach to Enterprise Systems Performance Measurement
Darshana Sedera, Guy Gable and Michael Rosemann
School of Information Systems
Queensland University of Technology, Brisbane, Australia
d.sedera@qut.edu.au, g.gable@qut.edu.au, m.rosemann@qut.edu.au,
Abstract
A range of influences, both technical and organisational, has encouraged the wide spread adoption of Enterprise
Systems (ES). Nevertheless, there is a growing consensus that Enterprise Systems have in many cases failed to
provide expected benefits. The increasing role of, and dependency on ES
(and IT in general), and the
‘uncertainty’ of these large investments, have created a strong need to monitor and measure ES performance.
This paper reports on a research project aimed at deriving an ‘Enterprise Systems benefits measurement
instrument’. The research seeks to identify how Enterprise Systems benefits can be usefully measured, with a
‘balance’ between qualitative and quantitative factors.
Keywords
Enterprise Systems, Information Systems, Performance Measurement, Balanced Scorecard, Governmental IS
INTRODUCTION
Enterprise Systems (ES) (synonymous with Enterprise Resource Planning Systems, Enterprise Wide Systems,
Integrated Vendor Solutions, Integrated Standard Software and Enterprise Application Systems) are
customizable, standard software solutions that have the potential to link and automate all aspects of the business,
incorporating core processes and main administrative functions into a single information and technology
architecture (Klaus et al., 2000; Bingi et al., 1999; Parr et al., 1999). Organisations have invested heavily in these
systems, with the expectation of; improvements in business processes, better management of IT/IS expenditure,
increased customer responsiveness, and generally, strategic business improvements (Li, 1999; Ross and Vitale,
1999). Despite growth in the ES market, recent research shows growing dissatisfaction with ES; that they have
failed to deliver the anticipated benefits (e.g. Bingi et al., 1999; Gable et al., 1998; Holland et al., 1998). High
profile ES implementations (e.g. Fox Myers, Unisource worldwide, Dell computers, Hershey Food Cooperation,
Whirlpool and Gore-text) have raised serious questions about the credibility of ES (Calogero, 2000). Some of the
organisations mentioned have incurred significant financial losses and also lost business opportunities worth
many millions of dollars (e.g. Bingi et al., 1999, Chung and Snyder, 1999; Songini, 2000).
STUDY OBJECTIVE AND DESIGN
Several research organisations, practitioners and academics have investigated the key issues related to low return
on investments in ES. These studies have identified the extent of the so-called ‘damage’ done by the ES, critical
success factors of ES, possible causes for low performance, and benefits realisation from ES. Many of these
studies have implicitly and explicitly referenced the importance of measuring the performance of ES.
Furthermore, it has also been identified that a high percentage of ES benefits are intangible and therefore harder
to evaluate financially (e.g. Baer, 1999; Davis, 1999; Deloitte consulting 2000; Knowles et al., 2000;
Levingston, 1999; Matthys and Shorter, 2000; Mabert et al., 2000; Radding, 1999; Thorp, 1994). This has further
undermined the position of ES; especially in the public sector, where the tangible benefits (i.e. Financial
benefits) are a relatively smaller proportion of total benefits sought or achievable.
The aim of this research is to address ‘how Enterprise systems benefits can be usefully measured’, with a
‘balance’ between qualitative and quantitative factors. The ultimate goal is to derive a comprehensive ‘Enterprise
Systems Benefits Measurement Instrument’ to (a) assess the goodness of the Enterprise System and (b) identify
where organizations should focus their ‘benefits realisation’ efforts and further investments. The study focuses
on the public sector and uses the Balanced Scorecard approach (Kaplan and Norton, 1992, 1996, 2000) to
capture both financial and non-financial aspects of ES measurement. The research questions are:
1.
What measures are appropriate for evaluating an ES?
2.
Is the Balanced Scorecard an appropriate approach for ES measurement?
3.
What perspectives should be considered when measuring the performance of an ES?

---

Proceedings of the Twelfth Australasian Conference on Information Systems
The study outcomes will significantly contribute to the ES and general MIS arenas by: (1) providing a better
understanding of the Balanced Scorecard in the public sector and for ES performance evaluation (a domain
previously not addressed with this approach), (2) identifying and justifying specific measures that link to co-
operate goals and strategies for successful ES performance measurement, and (3) ultimately resulting in a
comprehensive “ES Performance Measurement Instrument”.
The Queensland Government ES Benefit Realisation initiative and participating Government agencies are the
focus of core data collection on the study. This is mainly due to the close alignment of the Queensland
Government’s ES benefit realisation approach and our study objectives, and to our relatively good access to case
agencies and relevant information. The research design includes: (1) a comprehensive literature review of
various ES related themes and of the application and development of the Balanced Scorecard; (2) design of a
preliminary Balanced Scorecard for ES based on measures identified from the literature; (3) a preliminary case
study of the Queensland Government’s Benefits Realisation Network (QGBRN); (4) a pilot case study to
validate the initially identified Scorecard perspectives and measures; and (5) a multiple case study across several
Queensland Government agencies to further evolve and validate the final public sector ES Balanced Scorecard.
This paper presents early results from the study, summarizing findings from the literature review and describing
a high-level case study of the QGBRN.
REVIEW OF RELATED LITERATURE
The following section summarizes findings to date from the literature review. It first explores the study context,
demonstrating potential contributions of this study, and relating to past ES success and failure studies and those
that implicitly or explicitly state the importance of a structured ES performance measurement technique. It then
introduces the Balanced Scorecard approach and summarizes key studies that have proposed the Balanced
Scorecard for IT (and sometimes specifically for ES) performance measurement. Finally, this section concludes
with a discussion on the appropriateness of the Balanced Scorecard for Public Sector ES performance
measurement.
Research on ES success and failure
A study conducted by Boston Consulting Group (2000) using interviews of more than 100 executives who were
involved in Enterprise Systems implementations showed that only 33 % of ES projects could be classified as
successful. Another study by the Standish Group established that most ES implementations today result in cost
and schedule overruns. A comprehensive study of nearly 100 large companies (with revenue greater than US $ I
billion) was carried out by Deloitte consulting (2000) and benchmarking partners Hammer and Co. This study is
said to be one of the most successful studies ever conducted into the performance of ES (McCann and Lucas,
2000). Not surprisingly, this study also concluded with mixed outcomes regarding the ES pay-offs. The study
identified that approximately one third (37%) of the companies studied reported significant and quantifiable
benefits from their ES investments. However, 15% of the companies believed that they had received ‘nothing’
out of their ES. 25% of the organisations received modest results from ES, and they are not positive about
gaining further benefits. The remaining 23% were unable to report any specific quantifiable benefits. The overall
research figures show that as many as half of the companies have not realized sufficient benefit from their ES
investment.
A study of US manufacturing firms (Mabert et al., 2000), conducted with the objective of assessing the return on
investment from ES, has revealed the extent of dissatisfaction. This study found that more than 50% of the ES
initiatives studied have reported lower Return On Investment (ROI) than expected. A survey conducted by the
Stanford based META group Inc., of 63 companies with ES, revealed an average negative return of $1.5 million,
when quantifiable cost savings and revenue gains were balanced against hardware, software, consulting and
support (Stedman, 1999; Stein, 1999). However, the META group admits that ES software do help deliver more
important intangible benefits, such as better customer services.
There exists a common consensus that ES, given it is managed carefully, provide substantial intangible (non-
financial) results as well as tangible (financial) results. Bartholomew (1999) claims that in a typical business
environment, 80% of the organisation's value is in intangible assets. This intangibility of assets has made it
difficult for organisations to illustrate whether they have received the benefits of ES or not (e.g. Baer, 1999;
Bylinsky, 1999; Cameron, 2000; McCann and Lucas, 2000; Radding, 1999; Stein, 1999; Stedman, 1999;
Vickers, 2000; Wah, 2000). For a range of reasons, few organisations have sought to carefully measure ES
benefits, using both tangible and intangible measures, of course thereby making evidence of these benefits
elusive and vague.

---

Proceedings of the Twelfth Australasian Conference on Information Systems
Need of an ES Performance Measurement
Recent studies indicate that issues pertaining to IS / ES evaluation and management are of great importance to
managers all over the world (e.g. Brancheau et al., 1996; Burn et al., 1993; Galliers et al., 1994). The increased
role of and dependency on ES, and the ‘uncertainty’ of benefits, have in combination magnified the need to
monitor and measure ES performance. On the other hand, several studies have indicated that while the ES / IT
investment is correlated with organisational revenue; it is not clearly correlated with either productivity or
profitability (e.g. Ives, 1994; Pennar, 1988; Sethi et al., 1993; Strassmann, 1997a, 1997b). This situation is very
much true in the context of ES, where in many cases, intangible benefits are harder to quantify.
Many traditional measurement and evaluation methods (e.g. Return On Investment [ROI], Economic Value
Added [EVA], Net Present Value [NPV], Internal Rate of Return [IRR], Return On Capital Employed [ROCE]),
have failed to yield an appropriate estimate of the ‘pay back’ from these complex systems (e.g. Barua et al.,
1996; Martinsons, 1991; Mukhopadhyay et al., 1997; Sharba et al., 1988). Some claim these performance
indicators have a high reliance on financial perspectives and thus portrait only one facet of the organisation.
Unfortunately, as discussed earlier, the impact of an ES is so vast and diverse that it cannot be measured by
using only traditional financial measures (e.g. Deloitte Consulting, 2000; Mabert et al., 2000; Martinsons et al.,
1999).
A growing number of companies have begun searching for new ways to measure the return on investment from
their IT projects, in particular from ES and with emphasis on ES performance measures (Cameron, 2000, Thorp,
1999). According to Vickers (2000), companies should analyze their ES investment by: (1) justifying the initial
purchase with return on investment and (2) trying to find the ROI after implementation. Vickers further (2000)
states that cost justification in manufacturing is straight forward, where the tangible benefits are much larger than
the intangible benefits. However, cost justification in services and human resources is harder to demonstrate. The
timing of benefits measurement is important, due to the fact that most of the benefits of ES take at least 2-3 years
to surface (e.g. Baer, 1999; Davis, 1999; Levingston, 1999; Mabert et al., 2000; Radding, 1999; Ross and Vitale,
1999). Cameron (2000) illustrates the importance of measuring the intangible benefits of ES, stating that
evaluating a complex system by just looking at the tangible benefits is a huge mistake. Calculating the gain from
an ES investment in general requires a deeper understanding of the organisational goals, strategies and
expectations.
These challenges have generated new interests and research on approaches to performance measurement,
performance evaluation and business value creation from ES, with emphasis on a balance between financial
(quantitative and historic) measures and non-financial (qualitative and futuristic) measures (e.g. Bruggeman,
1999; Eicklemann, 1999; Thorp, 1999; Van Der Zee, 1999a, 1999b). Up to now, there have been no
comprehensive tools to help organisations measure the performance of an ES.
There are several advantages from applying a performance measurement system in ES implementation,
especially for top management. These reasons include: (1) ES expenses are significant - as ES expenditure grows
and becomes a significant portion of business costs, top management and stakeholders become increasingly
interested in being able to [a] compare the value of these expenditure with the obtained benefits and [b] compare
those benefits with how other organisations benefit from a similar ES; (2) To find out the benefit generating
areas - top management wants to find the benefit generating points (if any) so that they can further exploit these
and apply corrective measures where the benefits are small; (3) ES have a ‘holistic’ impact on every operation in
an organisation; (4) ES performance measures aid in a competitive thrust - performance measures help top
management to obtain and maintain Key Performance Indicators (KPI); and (5) As an aid for innovation – as a
consequence of the above, top managers can identify and exploit areas for deploying innovative business
strategies (e.g. United Communications Group, 1989; Sethi et al., 1993; Sedera et al. 2001; Parker et al., 1988).
Balanced Scorecard: as a performance measurement method
The Balanced Scorecard (BSC) is a technique developed by Kaplan and Norton (1992) that helps organisational
decision makers to navigate the organisation towards success. The technique enables organisations to translate
their mission and strategy into a comprehensive set of performance measures that provide the framework for a
strategic measurement and management system. Organisations have used the Balanced Scorecard to; (1) clarify
and translate vision and strategy, (2) communicate and link strategic objectives and measures, (3) plan, set
targets and align strategic initiatives and (4) enhance strategic feedback and learning, and succeed in realizing
both tangible and intangible benefits of their investments (Kaplan and Norton, 1992, 1996, 2000).
The Balanced Scorecard measures organisational performance, with emphasis on financial objectives. But, it
also includes the performance drivers of these financial objectives, and measures organisational performance

---

Proceedings of the Twelfth Australasian Conference on Information Systems
across four balanced perspectives; (1) financial, (2) customer, (3) internal business processes and (4) learning
and growth.
Creators of the Balanced Scorecard argue that, traditional financial measures “Tell the story of the past”
(Kaplan and Norton, 1992). They try to address this inadequacy by complementing past performance measures
(financial measures) with the drivers of future performance indicators (customers, suppliers, employees,
processes, technologies and innovation. The fundamental concept of the Balanced Scorecard is to derive the
objectives and measures from the overall corporate vision and strategy and to use the four perspectives as a “
balanced” framework to monitor and achieve these objectives.
A properly developed Balanced Scorecard
should:
(1) Represent financial and non-financial measures from all levels of the organisation (front line to executives)
(2) Maintain an equilibrium between:
External measures (developed for the stakeholders and customers) and Internal measures (developed for
the bushiness processes, innovation, learning and growth)
Outcome measures (results from the past) and measures that are for Future performance
Objective (easily quantifiable outcome measures) and Subjective (judgmental performance drivers)
outcome measures
(3) Include only measures that are elements in a chain of cause-and-effect relationships that communicate the
meaning of the organisation’s (or business unit’s) strategy.
It must be pointed out that the traditional Balanced Scorecard approach is used to measure the performance of
the overall organisation, while the focus of this research is on measuring ES performance and evaluating the ES.
The following section addresses the appropriateness of the Balanced Scorecard for IT / ES evaluation.
Balanced Scorecard for IT evaluation
Recently, several researchers (e.g. Martinsons et al., 1999; Martinsons, 1992; Rosemann and Wiese, 1999;
Bruggeman, 1999; Reo, 1999) have suggested that the Balanced Scorecard may also help to evaluate the
performance of an Information System (IS) and to evaluate IS investments in a holistic manner. Some studies
have proposed substantial modifications to the perspectives and measures applied by Kaplan and Norton, while
some have preferred to use the original (Financial, Customer, Process and Learning and Innovation) Balanced
Scorecard perspectives. Table 1 summarizes several studies (alphabetically ordered), indicating their context and
illustrating the perspectives they have used to evaluate the performance of an Information System.
The other two perspectives: learning and innovation and process perspectives are recognized and mentioned
explicitly or implicitly as suitable perspectives to measure ES/IT performance (Reo, 1999: Rosemann and Wiese,
1999: Van Der Zee, 1999b). Krauth (1999), Reo (1999) and Rosemann and Wiese (1999) have proposed more
than four perspectives to evaluate the performance of ES/IT. Krauth (1999) discusses the connection between the
European Foundation for Quality Management (EFQM) standard, ISO 9001 system and the Balanced Scorecard
and shows the impact of using these three evaluation methods at the same time. Reo (1999) supports the
integration of the EFQM model with the Balanced Scorecard and also mentions the importance of integrating
Total Quality Management (TQM) principles. Rosemann and Wiese (1999) discuss the importance of adding the
‘Project’ perspective in the context of ES management. The Queensland Government utilizes the four original
perspectives of the Balanced Scorecard: Financial, Customer, Learning & Innovation, Process. This approach is
further described in detail under case study description.

---

Proceedings of the Twelfth Australasian Conference on Information Systems
Study
Context
Original Balanced Scorecard
Perspectives
Other Perspectives suggested by the
FI
CU
LE & IN
PR
authors
Beeckmann and Gemini,
IT
X
X
X
X
N/A
1999
Brogli, 1999
IT / FI
N/A
X
X
X
BUSINESS CONTRIBUTION
Bruggeman, 1999
IT
X
X
X
X
N/A
OPERATIONS
Cartney, 2000
ES
X
X
N/A
N/A
HUMAN RESOURCE
Durrani et al., 2000
IT strategy
X
X
N/A
X
N/A
Eicklemann, 1999
IT
X
X
X
X
N/A
Business
Greasley, 2000
process
X
X
X
X
N/A
simulation
Gordon et al., 1997
IT
X
X
X
X
N/A
EFQM
Krauth, 1999
IT
X
X
X
X
ISO 9001
BUSINESS VALUE
Martinson et al., 1999
IT
N/A
X
N/A
X
FUTURE READINESS
EFQM
Reo, 1999
IT
X
X
X
X
TQM
Rosemann and Wiese,
ES
X
X
X
X
PROJECT
1999
INFORMATION TECHNOLOGY
ORGANISATION (STRUCTURE &
Thorp, 1999
IT
N/A
X
N/A
X
CULTURE)
BUSINESS STRATEGY
Van Der Zee, 1999a
IT
X
X
X
X
N/A
Van Listenburg, 1999
ES
X
X
X
X
N/A
Van Der Zee, 1999b
IT
X
X
X
X
N/A
IT MISSION
Walton, 1999
IT
N/A
X
N/A
X
TECHNOLOGY
PEOPLE/ORGANISATION
FI: Financial, CU: Customer, LE & IN: Learning & Innovation, PR: Process
Table 1: Summary of IT / ES Balanced Scorecard perspectives
Balanced Scorecard: as a Public sector ES evaluation method
A fundamental feature of the Balanced Scorecard, is that it requires that each measure should relate to the
corporate strategies and to each other in a cause and effect relationship. The individual measures at each instance
would be unique depending on corporate goals and strategies (e.g. Barua et al., 1996; Kaplan and Norton, 1996;
Lezta, 1996). Thus, identifying the corporate goals and strategies in relation to the core perspectives is a critical
preliminary step in a Balanced Scorecard approach.
This study looks at developing a Balanced Scorecard for ES performance measurement, in the public sector. The
proposed starting metrics are to be extracted from the mainstream ES management literature as well as from
emerging literature on Information Economics, Government directives, and ES success studies (e.g. Ballantine et
al., 1996; DeLone and McLean, 1992; Eicklemann, 1999; Saarinen, 1996).
The Balanced Scorecard is applied in Private and Public sectors from two different viewpoints. In the Private
sector, the main emphasis is on financial indicators for managing the organisation. The Private sector responds to
fluctuations in market share, share prices, dividend growth and other changes in the financial perspective. In the
Public sector however, entities must respond mainly to legislative acts and are responsible to higher Government
authorities. The most common difference between a Private sector Balanced Scorecard and a Public sector
Balanced Scorecard lies in the purpose of utilizing the Balanced Scorecard. Public sector focuses on cost
reduction and customer satisfaction, while Private sector is mainly focused on revenue generation and
profitability.
CASE STUDY: QUEENSLAND GOVERNMENT BENEFIT REALISATION
NETWORK
This section describes an initiative by the State Government of Queensland to realize the ‘full potential’ from
their implemented ES – SAP. In late 2000, the Queensland Government, through the initiative of the Office of
Financial Systems and Training (OFST), initiated a ‘Benefit Realization Network’, aimed at maximizing benefits

---

Proceedings of the Twelfth Australasian Conference on Information Systems
from the ES (SAP R/3) investment. The Queensland Government’s Benefit Realisation Network (QGBRN) is a
voluntary association of agency representatives, who seek to share SAP benefits realization experiences and
‘learn from each other’. We were invited to participate in the QGBRN and have been providing advice and
recommendations on data collection tools for sharing experiences (i.e. Case study protocol). The QGBRN has
recognized the importance of the Balanced Scorecard and has integrated this into a framework for deriving
increased benefits of ES. Following, the history of the Queensland Government’s ES implementation and
objectives of the ES initiative are described. A synopsis of the current ES Benefit realization situation is then
provided. This section concludes with a discussion of the Queensland Government ES Balanced Scorecard
(Queensland Treasury, 1998, 2000a, 2000b). During the last eight months, since our involvement with QGRBN,
we have been documenting Balanced Scorecard experiences and efforts in Queensland Governmental agencies.
One such example (in Departments of Primary Industries – DPI) is briefly discussed in this section.
Queensland Government – ES history
In 1983, the Queensland Government adopted the Management Services America (now Dunn and Bradstreet),
financial modules. This was the first statewide deployment of a financial management system in Australia
(QGFMS-Queensland Government Financial Management System). A decade later, QGFMS, initially broadly
considered a success, was in the minds of many inadequate to support the Government's ambitious plans for the
future. In 1994, Queensland Treasury sent a request for information (RFI) to key ES vendors of ES. In October
1994, offers were (RFO) sought from three short-listed ES vendors and in December 1994, Queensland Treasury
selected SAP R/3.
The main SAP modules installed were Financial Accounting, Controlling, Materials Management and later in
some agencies Human Resources. In 1995 the state government of Queensland commenced implementation of
SAP Financials across all state Government agencies (later followed by Controlling, Material Management and
Human Resources in some agencies). The Queensland Government approach was very much focused on using
the Enterprise System as a common reporting and financial management tool (Queensland Treasury, 1998,
2000a). The objectives of the QGFMS are to provide a financial management system to Queensland Government
agencies that will: (1) support the Managing for Outcomes (MFO) framework and financial management
improvement activities, (2) encourage best practice resource management across Queensland Government, (3)
facilitate the consolidation of Queensland Government financial information, (4) meet the business needs of
agencies and (5) achieve economies of scale in main operations (Queensland Treasury, 1998, 2000a).
Queensland Government Benefit Realisation Approach
The exploitation of intangible assets has become a decisive aspect of information age organisations. The
Balanced Scorecard approach to measuring intangible assets as well as tangible assets has undoubtedly helped
managers to make strategic decisions. Further, the approach provides employees with the necessary information
required to carry out their duties in support of the strategic decisions of a company. The Balanced Scorecard
approach is particularly appropriate in situations where there are no suitable financial indicators to measure
effectiveness: for example, in non-profit organisations and in the public sector. Unlike in Private sector, the
objective of the public sector is not merely to make or save money, but to offer increased and improved services
with resources available. The Queensland Government has chosen to utilize the Balanced Scorecard approach to
evaluate the performance of their SAP Enterprise System and to guide its agencies to maximize benefits through
strategic decisions. There are three main “tools” that the Queensland Government uses to maximize benefits and
thus achieve business objectives: (1) a clear vision of the strategic management plan, (2) a Balanced Scorecard,
and (3) a benefit realisation plan (Sedera et al. 2001, Queensland Treasury, 2000a).
Queensland Government Sample Balanced Scorecard
The Queensland Government’s Balanced Scorecard approach is tightly integrated to the “Managing for
Outcomes” (MFO) budgetary system. The expected benefits from the Queensland Government Balanced
Scorecard are to: (1) understand the management approach in a holistic manner, (2) relate strategy to
performance and action, (3) set performance targets, (4) focus, communicate and coordinate effort, (5) reduce /
eliminate blind spots, and (6) Improve management and performance of the organisation. The proposed
Balanced Scorecard will cover all quadrants (Financial, Customer, Internal Process and Learning and
Innovation) and a wide spectrum of information needs. The smaller business units (i.e. organisational entities)
need to link their Balanced Scorecards to the overall corporate Balanced Scorecard and lower level strategies
should be aligned with the Queensland Government mission and vision (Queenalsnd Treasury, 2000a, 2000b).
The Queensland government Benefits Realisation Plan consists of 6 main steps: (1) specifying the appropriate
business drivers, (2) identifying key stakeholders, (3) determining the Balanced Scorecard perspectives, (4)
identifying and applying method/s of measuring (or quantifying) the benefit, (5) identifying initiative to achieve
the recognised benefit and (6) deploying a risk management strategy including potential risks, constraints and
dependencies (Queensland Treasury, 2000a).

---

Proceedings of the Twelfth Australasian Conference on Information Systems
In recent years, the Queensland Government priorities, has increased pressure on agencies to see more holistic in
their approaches to portfolio issues (Department of Primary Industries, 2000). For instance, it is no longer
acceptable for one agency to concentrate solely on economic gain through industry, while another agency
concentrate solely on environmental protection. All agencies have to acknowledge their links with other agencies
to deliver the best service to the community.
Performance measurement in Department of Primary Industries (DPI) includes evaluation of the Department’s
outputs against statewide trends. This broad approach, based on the Balanced Scorecard, takes both internal and
external perspectives of DPI into consideration in measuring performance (see figure 1). In this way, the
department can link their actual performance with the expectations of stakeholders (Department of Primary
Industries, 1999, 2000). Like in the standard Balanced Scorecard, DPI uses the same four perspectives to
measure performance and seek improvements.
Figure 1: DPI’s Performance management framework (adopted from Department of Primary Industries, 2000)
Table 2 briefly depicts the internal strategies and example measures of performance used in the Department of
Primary Industries’ Balanced Scorecard.

---

Proceedings of the Twelfth Australasian Conference on Information Systems
Strategies
Methods of measuring performance
Regularly review DPI services in terms of satisfaction and relevance
Client satisfaction surveys
to clients and the community
Awareness polls of Queensland
ips
h
Broaden the interaction of managers and staff with clients and the
community perceptions of DPI
and
community to cover all aspects of food and fiber chains
Staff awareness assessments
nt
Ensure region-specific requirements are a prime consideration in
Consumer research
mmunity
lations
Clie
c
o
reviewing DPI client and community relationships and services
Re
Review the level of awareness of the DPI brand and the image of DPI
within the community
Develop organisational leadership skills
Culture survey
g


i
n
Develop skills and effective knowledge management relevant to
Workforce profile reports
h
future needs
Training and development reports
l
earn
rowt
Enhance the diversity and flexibility of our workforce
g

l
e
,
Recognise and reward appropriate achievements and behaviours
Support investment in innovative learning approaches
e
op
and
P
Ensure staff are aware of and behave in accordance with the Code of
Conduct
Continue to focus on improving overall performance management
Audit of investment processes
s
Implement rigorous and participative investment management
Internal and external compliance
al
s
s
processes
audits
ine
Evaluate targeted areas of performance
Performance evaluation
t
ern
cesse
In
bus
r
o
Use risk assessment to optimise business opportunities
Trend analysis
P
Develop and implement integrated systems support
Enhance information flows and encourage knowledge sharing
Develop DPI’s leadership role in financial management of resources
Trend analysis
Develop benchmarks for financial management
Internal comparative analysis
e
nt
rce
Implement an internal hiring system for depreciable assets
External comparative analysis
Link strategic planning with investment decisions and budgets
Audit of processes
Government Asset Management
Resou
Managem
System
Table 2: Internal Strategies and Measuring Performance (Adopted from Department of Primary Industries, 2000)
CONCLUSION AND OUTLOOK
The goal of this paper was to propose the Balanced Scorecard as an appropriate approach for measuring the
performance of ES employed in the public sector. It first introduced the design and objectives of the research,
detailing results from past ES success and failure studies, and justifying the need for a balanced ES performance
measurement. The Balanced Scorecard approach was then introduced, proposing its appropriateness for ES
performance evaluation in both private and public sectors.
Finally, the Queensland Government Balance scorecard approach was discussed describing the overall goals and
techniques of applying a balanced scorecard in the public sector.
A high-level case study of the overall QGBRN and an example of DPI’s Balanced Scorecard with a
comprehensive literature review has been completed to date. Further tasks planned include; a number of case
studies of ES benefits realisation projects in Queensland Government agencies to evaluate the effectiveness of
the Balanced Scorecard approach to measure Enterprise Systems performance. A pilot case study of the
Queensland Police followed by a multiple case study of other agencies practicing the QGBRN guidelines, will be
conducted, and the proposed Balanced Scorecard measurement instrument will be developed and justified using
pattern analysis techniques (Yin, 1994).
REFERENCES
Baer, T., (1999) "Finding Value in all the right places", Manufacturing Systems, November.
Ballantine, J., Bonner, M., Levy, M., Martin, A., (1996) “The 3D model of Information Systems Success: the
research for the dependent variable continues”, Information Resources Management Journal, 9:4, pp 5-14.
Bartholomew, D., (1999) "Process is back", Industry week, Cleveland, (248:20), p. 31-36.
Barua, A., Lee, S., C., H., Whinston, A., B., (1996) “The calculus of reengineering”, Information Systems
Research, 7:4, pp 408-428.

---

Proceedings of the Twelfth Australasian Conference on Information Systems
Bingi, P., Sharma, M., Godla, J., (1999) "Critical factors affecting an ERP implementation", Information
Systems Management, Summer.
Beeckmann, D., (1999) “IT Balanced Scorecards: Filling the missing link between IT-effectiveness and IT-
efficiency. Symposium on IT Balanced Scorecard, Antwerp, March.
Boston Consulting Group, (2000) “Getting Value from the enterprise initiatives, a survey of executives”,
available from http://www.bcg.com.
Brancheau, B., Janz, B., Wetherbe, J. (1996) “Key issues in Information Technology Management: 1994-95 SIM
Delphi results”, MIS Quarterly, 20:2, pp 225-242.
Bruggeman, W., (1999) “The Balanced Scorecard: Functional, Business Unit and Corporate Level Application”,
Symposium on IT Balanced Scorecard, Antwerp, March.
Brogli, M., (1999) “Using the Balanced Scorecard in the IT of a financial service company”, Symposium on IT
Balanced Scorecard, Antwerp, March.
Burn, J., M., Saxena, K., B., C., Ma, L., Cheung, H., K., (1993) “Critical issues of IS management in Hong
Kong: Cultural comparison”, Journal of Global Information Management, 1:4, pp 28-37.
Bylinsky, G., (1999) "The challengers move in on ERP", Fortune, November
Calogero, B., (2000) "Who is to blame for ERP failure?", Sunsaver, June.
Cameron, K., S., Whetten, D., A., (1983) "Some conclusions about organisational effectiveness", in
"Organisational effectiveness: A comparison of multiple models", New York: Academic Press, pp 261-
277.
Cameron, P., (2000) "Measuring Up", CMA Management, March.
Cartney, D., (2000) “How to effectively use management reports”, Managing for Outcomes Balanced Scorecard
& Financial Analysis, Corporate Advisory Service. PowerPoint Presentation.
Chung, S., Snyder, C., (1999) "ERP initiation - A historical perspective" in proceedings of the Americas
Conference of Information Systems (AMCIS ’99).
Davis, B., (1999) "ERP vendors offer integrated analysis", Informationweek, May.
Deloitte Consulting, (2000) "ERP second wave: Maximizing the value to Enterprise applications and process"
Delone, W., H., McLean, E., R., (1992) “Information Systems Success: the quest for the dependent variable”,
Information Systems Research, Vol 3, pp 60-95.
Department of Primary Industries (2000), “Corporate Plan 2000”, Department of Primary Industries
Department of Primary Industries (1999), “Operating Environment”, Department of Primary Industries
Durrani, T., S., Forbes, S., M., Carrie, A., S., (2000) “Extending Balanced Scorecard for technology
development”, IEEE.
Eicklemann, N. (1999) “A comparative analysis of BSC as applied in Government and Industry organisations”,
Symposium on IT Balanced Scorecard, Antwerp, March.
Gable, G., Scott, J. and Davenport, T. H. (1998) " Cooperative ERP life cycle knowledge management" in
proceedings of the 9th Australasian Conference on Information Systems, Sydney
Galliers, R., Merali, Y., Spearing, L. (1994)“Coping with IT?: How British executives perceive the IS
management issues in mid 1990s” Journal of Information Technology, 9, pp 223-238
Gordon, D., Kunov, H., Dolan, A., Carter, M., (1997) “The development of the Balanced Scorecard information
system”, IEEE.
Greasley, A., (2000) “Effective uses of Business process simulation”, in proceedings of the 2000 Winter
Simulation Conference.
Holland, C., Light, B., Gibson, N., (1998) “Global ERP implementation” in proceedings of the American
Conference of Information Systems (AMCIS ’98), Baltimore.
Ives, B., (1994) “Probing the productivity paradox”, MIS Quarterly, Vol 18, pp R21-R24.
Kaplan, R., Norton, D., P., (2000) "Having trouble with your strategy? Then map it", Harvard Business Review,
September-October.

---

Proceedings of the Twelfth Australasian Conference on Information Systems
Kaplan, R., S., Norton, D., P., (1992) "The Balanced Scorecard - Measures that drive performance", Harvard
Business Review, January-February.
Kaplan, R., S., Norton, D., P., (1996) "Translating Strategy in to action: The Balanced Scorecard", Harvard
Business School Press, Boston, Massachusetts.
Klaus, H., Rosemann, M., Gable, G., (2000) "What is ERP?”, Information Systems Frontiers, 2:2, pp 141-162.
Knowles, H., Fotos, S., Henry, N., (2000) "Q & A from the internet: Implementing SAP, The controllers update,
September.
Krauth, P., (1999) “Balanced Scorecard approach to strategy formation at a Hungarian IT company”,
Symposium on IT Balanced Scorecard, Antwerp, March.
Levingston, M., (1999) "Full ERP benefits take time, effort", Electronic Commerce News, July.
Lezta, S., R., (1996) “The design and implementation of the Balanced Business Scorecard: an analysis of three
companies in practice”, Business Process Reengineering and Management Journal, 2:3, pp 54-76.
Li, C., (1999) "ERP packages: what's next?", Information Systems Management, 16:3, Summer.
Mabert, A., M., Soni, A., Venkataraman, M., A., (2000) "Enterprise Resource Planning survey of US
manufacturing firms, Production and Inventory Management Journal, SQ.
Martinsons, M., G., (1991) “A domain selection and evaluation framework for the introduction of knowledge
based systems in smaller businesses”, Journal of Information Systems, Vol 1, pp 207-215.
Martinsons, M., G., (1992) “Strategic thinking about information management”, Keynote address to the 11th
annual conference of the International Association of Management Consultants, Toronto.
Martinsons, M., G., Davison, R., Dennis, T., (1999) “The Balanced Scorecard: a foundation for the strategic
management of Information Systems, Decision Support Systems, 25:1.
Matthys, N., Shorter, J., D., (2000) "Electronic resource planning solutions for business processes", The journal
of Computer Information Systems, Fall.
McCann, G., Lucas, T., (2000) "Making ERP spell ROI", Charter, March.
Mukhopadhyay, T., Lerch, F., J., Mangal, V., (1997) “Assessing the impact of labour productivity: a field
study”, Decision Support Systems, 19:2, pp 109-122.
Parker, M., Benson, R., Trainor, H., (1988) “Information Economics: Linking business performance to
Information Technology”, Prentice-Hall, Englewood, Cliffs, NJ.
Parr, A., N., Shanks, G., Darke, P., (1999) "Identification of necessary factors for successful implementation of
ERP systems" in 'New Information technologies and theoretical organisational processes: field studies
and theoretical reflections on the future of work.", IFIP publications.
Pennar, K., (1988) “The productivity paradox: why the payoff from automation is still illusive?”, Business
Week, 3055, pp 100-102.
Queensland Treasury, (1998) “QGFMS Lead Agency Strategic Plan 1988-2003”, The Office of Financial
Management Systems and Training.
Queensland Treasury, (2000a) “QGFMS Benefits Realisation Guidelines: Translating Strategy in to performance
benefits and initiatives", The Office of Financial Management Systems and Training.
Queensland Treasury, (2000b) “Understanding Managing For Outcomes”, The Office of Financial Management
Systems and Training.
Queensland Treasury, (2000c), “QGFMS Lead Agency Strategic Plan”, The Office of Financial Management
Systems and Training.
Radding, A., (1999), "Return on Investment: Measuring technology's dividends", Ent, November.
Reo, D., A., (1999) “The Balanced IT Scorecard for software intensive organisations: benefits and lessons learnt
through industry applications, Symposium on IT Balanced Scorecard, Antwerp, March.
Rosemann, M., Wiese, J., (1999) "Measuring the performance of ERP software: a Balanced scorecard approach"
in proceedings of the 10th Australasian Conference on Information System, Wellington.

---