ENQA Workshop Reports
Kauko Hämäläinen
Finnish Higher Education Evaluation Council (FINHEEC)
Kimmo Hämäläinen
European Network for Quality Assurance in Higher Education (ENQA)
Anette Dørge Jessen
Danish Evaluation Institute (EVA)
Minna Kaartinen-Koutaniemi
Diaconia Polytechnic, Finland (DIAK)
Dorte Kristoffersen
Danish Evaluation Institute (EVA)
Benchmarking in the Improvement
of Higher Education
ENQA Workshop Reports 2
European Network for Quality Assurance in Higher Education • Helsinki
1

---

ENQA Workshop Reports
The views expressed in this Workshop Report are those of the authors. Publication does not imply either
approval or endorsement by the European Network for Quality Assurance in Higher Education or any of
its members.
© European Network for Quality Assurance in Higher Education, 2002, Helsinki
This publication may be photocopied or otherwise reproduced without any specific permission
of the publisher.
Cover: Jussi Hirvi / Green Spot Media Farm
Layout: Pikseri Julkaisupalvelut
ISBN 951-98680-6-2 (paperbound)
ISBN 951-98680-7-0 (pdf)
ISSN 1458-106X
Printed by Monila
Helsinki, Finland 2003
2

---

ENQA Workshop Reports
Preface
Benchmarking as a higher education evaluation tool is used commonly but differently throughout Eu-
rope. Even the exact definition of the actual term benchmarking varies from country to country. A deter-
mination to map these various benchmarking concepts and to find common denominators to benchmarking
resulted in an active workshop organised by ENQA on 14–15 June 2002 in Finland. European evaluation
experts attacked these problems and were able to identify several good principles of benchmarking as
well as recommendations for future action. This report, the second ENQA Workshop Report, is a direct
result of the experts’ hard work.
The Steering Group of ENQA sees benchmarking as one of the increasingly relevant evaluation meth-
ods today. As a consequence the Steering Group decided to support this workshop in its annual action
plan. The feedback ENQA has received from the participants of the workshop indicates that discussions
and group work were definitely among the most active ones in recent ENQA workshops. This is gratify-
ing, because one of the cornerstones of our Network’s activities is to bring together European experts to
exchange views on supportable evaluation practices.
The number of participants in the workshops has intentionally been limited to about 30 persons per
workshop in order to ensure the kind of intensive discussion among the participants, which again leads to
joint practical results. This workshop showed once again that this method indeed is a practical one.
I would like to thank the organisers of the workshop who also have written this final report and I hope
the reader will find the result worthwhile.
Christian Thune
Chairman
ENQA Steering Group
3

---

ENQA Workshop Reports
Contents
1
Introduction ________________________________________________________________ 5
2
Benchmarking in brainstorming _____________________________________________ 7
Asko Karjalainen, University of Oulu, Finland
3
Reflections and recommendations _________________________________________ 11
Kimmo Hämäläinen, European Network for Quality Assurance in Higher Education;
Dorte Kristoffersen, Anette Dørge Jessen, Danish Evaluation Institute
3.1
Principles of good benchmarking in the development of higher education _________ 11
3.2
Concrete examples of benchmarking _________________________________________ 12
3.3
Recommendations _________________________________________________________ 13
4
Case studies on benchmarking ____________________________________________ 14
4.1
Creative benchmarking – designing sustainable international cooperation
in higher education (Soili Niinikoski, University of Oulu, Finland) __________________ 14
4.2
International comparative evaluation of BSc programmes of agricultural
science in four European countries (Anette Dørge Jessen, Danish
Evaluation Institute, Denmark) ________________________________________________ 19
4.3
Recent developments in benchmarking in the United Kingdom and an overall
look at three benchmarking subject statements as case studies (Fiona Crozier,
Quality Assurance Agency for Higher Education, United Kingdom) ________________ 23
4.4
Benchmarking in the perspective of a “learning institution” and as a means
to search for best practices (Bente Kristensen, Copenhagen Business School,
Denmark) __________________________________________________________________ 27
4.5
Benchmarking as a tool in higher education guidance (Mirja Toikka,
Kymi Polytechnic, Finland) ___________________________________________________ 33
4

---

ENQA Workshop Reports
1 Introduction
Purposes and target groups
Context and rationale of the
seminar
The Steering Group of the European Network for
Quality Assurance in Higher Education (ENQA)
ENQA commissioned and conducted this seminar
commissioned a seminar on benchmarking to:
due to an increasing interest in recent years to ap-
• Establish an understanding of the principles for
ply benchmarking as a tool for quality improve-
“good” (true) benchmarking in the development
ments within higher education in Europe.
of higher education;
The interest for benchmarking in higher educa-
• Provide concrete examples of various bench-
tion should be understood in the context of the Bo-
marking practices with the view to establish good
logna process, which emphasises the need for more
practices;
comparison, transparency and visibility of quality
• Discuss strengths and weaknesses related to
in higher education.
benchmarking in evaluation;
Benchmarking does not, however, only relate to
• To reach conclusions on perspectives for Euro-
the international political context. It should also be
pean benchmarking within higher education.
seen as a response to the growing competition
among educational institutions (nationally as well
The seminar took place in Helsinki, Finland on 14–
as internationally) and their search for best prac-
15 June 2002, and was targeted towards the staff of
tices and superior performance.
ENQA member agencies (currently 37), partner in-
Finally, the motivation of ENQA to initiate this
stitutions and external benchmarking experts. The
seminar should be viewed in the light of the grow-
methods used for the sessions included working
ing diversity in the definition and understanding of
groups and plenary sessions. Twenty-six partici-
the concept of benchmarking, which has raised a
pants from 13 countries took part, and it was at-
need to establish common principles for bench-
tended by a broad range of representatives, which
marking within the framework of the ENQA coop-
ensured an appropriate geographical balance of rep-
eration.
resentatives from northern, central and southern
Europe. Furthermore, ENQA had invited partici-
pants with extensive practical experience in
Structure
benchmarking as well as participants for whom
benchmarking is a new working field. In addition
The seminar was structured around three sessions.
to the ENQA member agencies, the participants
The first session was facilitated by an external ex-
included representatives from higher education in-
pert and focused on the introduction of the concept
stitutions, the European Commission and the newly
of benchmarking and the formulation of principles
established Center of Accreditation and Quality
for “true” benchmarking (as opposed to “false”
Assurance of the Swiss Universities.
benchmarking). The session was organised as a
mixture of presentations (by the external expert) and
intensive group work involving the participants in
5

---

ENQA Workshop Reports
the discussions on the concept and definition of
vided an example of an international benchmarking
benchmarking.
project initiated by the University of Oulu involv-
While the first session provided the participants
ing partnership with seven institutions (and seven
with a conceptual framework of benchmarking, the
programmes) in six different countries. The project,
second session was based on presentations of con-
which is characterised as ”cooperative bench-
crete examples of various benchmarking practices
marking”, concentrated on reaching agreements on
from United Kingdom (two cases), Finland (two
common standards for good practice through a
cases) and Denmark (two cases).
lengthy process of negotiation between the involved
The third and final session concentrated on reach-
partners.
ing conclusions for the future perspectives of
The first case from Denmark concerned an inter-
benchmarking within the European context and to
national comparative evaluation conducted by the
provide recommendations to the Steering Group of
Danish Evaluation Institute (EVA) involving BSc
ENQA for future actions. This publication is a sum-
programmes of agricultural science in four differ-
mary of the cases, recommendations and conclu-
ent countries. The evaluation developed and tested
sions that the participants made on the basis of the
a set of pre-defined criteria against which the four
cases.
programmes were assessed. From a methodologi-
cal perspective the outcome of the evaluation is an
improved methodological framework for interna-
Issues covered by the cases:
tional comparative assessments derived from the
United Kingdom, Finland,
methodological lessons obtained during the imple-
Denmark
mentation of the evaluation. From the perspective
of the institutions the main outcome is the range of
The first case from the United Kingdom provided
cases for best practices, which the institutions can
an overview of the recent developments with
use to improve their own practices. The second
benchmarking in higher education in the UK. These
Danish case included a presentation of an exten-
developments followed the recommendations of the
sive benchmarking strategy carried out by the Co-
report of the National Committee of Inquiry into
penhagen Business School (CBS) involving vari-
Higher Education (the Dearing report), published
ous types of quality improvement mechanisms. The
in July 1997, concerning a need to develop explicit
presentation focused on the learning culture of CBS
standards (benchmark statements) for programme
and the application of both internal and external
approval. As a continuation of this, another UK case
benchmarking to support the transformative learn-
gave concrete examples of the work of the Quality
ing at the institution.
Assurance Agency for Higher Education (QAA)
concerning the process of establishing subject
benchmark statements within three different sub-
The contents of the report
ject areas (Dance, Drama and Performance; Radi-
ography and Chemistry).
The report is divided into four chapters. Chapter 1
The two cases from Finland viewed the issue of
provides an overview of the overall conclusions
benchmarking from a different perspective, namely
derived from the discussions on the definition of
from the perspective of the education institutions.
the concept of benchmarking, including the estab-
In both cases the benchmarking projects were initi-
lished criteria for a good benchmarking project.
ated by the institutions themselves. The first case
Chapter 2 deals with the concept of benchmarking
concerned the work of three national polytechnic
itself. Chapter 3 summarises the conclusions and
institutions in the development of a benchmarking
recommendations made for the future perspectives
system applied for student guidance viewed from
for European benchmarking and Chapter 4 presents
the perspective of students. The second case pro-
the various cases.
6

---

ENQA Workshop Reports
2 Benchmarking in brainstorming
Asko Karjalainen, University of Oulu, Finland
“Our task is to try to redefine benchmarking by
using new concepts and surprising metaphors.“
During the last decade quality assessment phenom-
Definitions
enon called benchmarking has expanded very rap-
idly.1 Benchmarking stories are told all over the
In the literature benchmarking has many definitions.
organisations, also in the public sector. And some
I have divided these definitions to three categories:
stories they are! They are success stories about
practical definitions, existential definitions and
growing efficiency, great improvements, with an
metaphorical definitions.
undertone of enthusiasm and promising visions. In
Practical definitions tell, through prose, what
those narratives benchmarking is introduced as the
benchmarking is or what activities it includes:
modern management tool as well as the most effec-
Benchmarking “is the systematic study and com-
tive quality enhancement method. It seems that
parison of a company’s key performance indicators
benchmarking really works. Why is it so? There
with those of competitors and others considered
may be many practical reasons like “building on
best-in-class in a specific function.” (Dervitsiotis
the work of others makes sense”, “it can lead to
2000)
cooperation” or that the method is concrete and well
“…is a way of comparing a product or process
documented (Keehley & al 1997)
against others, with reference to specified stand-
It should, however, be self-evident that any
ards.” (Pepper, Webster & Jenkins 2001)
method adopted from business life is most likely to
fail in defensive organisations like the universities.2
Existential definitions try to connect benchmarking
Surprisingly enough, this seems not to be true with
with the experiences, emotions and basic processes
benchmarking. There is an increasing interest to
of the human existence. These definitions bring the
apply benchmarking in the higher education organi-
method closer to our ordinary living world. They
sations (Schofield 2000). The unexpected reaction
suggest that benchmarking is only a more formal-
compels us to set the philosophical question, what
ised dimension of our natural everyday interaction.
is the hidden nature of the benchmarking process?
“…it is, at bottom, a systematic way of learning
What is the secret of benchmarking? In the
from others and changing what you do.” (Epper
ENQA workshop, a group of benchmarking experts
1999)
approached the subject under the heading of
“benchmarking philosophy”. Below I will utilise
“It is actually a matter of imitating successful be-
some ideas of our “creative morning session”. To
haviour.” (Karlöf & Östblom 1993)
create something new, useful and far-reaching is the
“Benchmarking is a form of human being’s natural
task of the reader now. In this article I only give
curiosity with which s/he explores the possibilities
some incentives based on the workshop results.
of cooperation and friendship.” (Karjalainen,
Kuortti & Niinikoski 2002)
1 By using the expression “assessment phenomenon”, I will now
“Benchmarking is a learning process, which re-
underline the importance of reflection. Benchmarking must be tested
quires trust, understanding, selecting and adapt-
by creative and critical thinking.
2 Some further critiques of benchmarking, see for example Gooden
ing good practices in order to improve.” (One team
& McCeary 2001, Dervitsiotis 2000, Palfreyman 1999.
in ENQA workshop 2002)
7

---

ENQA Workshop Reports
So far there are no really strong metaphorical
– competitive
– benchmark-oriented
benchmarking definitions. This indicates that re-
– collaborative
– functional
searchers, consultants, managers and other
– outcome–oriented
– generic
benchmarking users merely see the method as a
– bureaucratic
– explorative
technical question. Metaphorical definitions, by
– qualitative
– co-operative
– quantitative
– dialogical
using metaphorical expressions, could provide new
– independent
– implicit
and astonishing perspectives. They could provide a
– experience-seeking
– explicit
surprising and a revelatory angle to the nature of
– process-oriented
– academic
benchmarking or give a sudden insight to the inner
– functional
– international
meanings of the method. “State of mind of an or-
– internal
– touristy
ganization” is an example of a weak metaphorical
– external
– horizontal
expression:
– spying
– vertical
– copying
– ranking-oriented
“…it is the state of mind of an organization which
– creative
– improvement oriented
encourages the continuous effort of comparing func-
– visionary
– diagnostic
tions and processes with those of best in class, wher-
ever they are to be found.” (Zairi & Leonard 1994)
A free combination of these attributes can produce
But why would we not develop stronger ones? What
a huge amount of variations for methodological
if we called benchmarking “the shortcut through
framework. At the ENQA workshop, teams tried to
the forest of the quality assessment”, “the flower of
define the distinction between “true” and “false”
the organisational curiosity” or “the envious energy
benchmarking. True benchmarking, some said, is
between the managers”? Each of these metaphors
always improvement-oriented. Negotiation, col-
implies a very different benchmarking concept and
laboration and developing a process for mutual un-
process. Metaphors are tools for creating self-aware-
derstanding are necessary parts of it. In true bench-
ness. When starting a benchmarking project, why
marking organisations and people learn from each
would you not search your thoughts and create your
other and there is dialogue. It has explicit and open
own metaphor? The metaphor matters in our post-
goals and the decision-making process is (as) clear
modern world of narratives.
(as possible). True benchmarking is always crea-
tive. Adapting best practices does not mean the same
as copying them.
True or false benchmarking?
False benchmarking is rank-oriented or merely
explorative without interest in improvement. It has
Benchmarking literature sometimes uses the con-
hidden purposes and it may even be spying. Nor is
cept “true benchmarking”. One challenge of our
touristy visiting true benchmarking. Fuzzy goals and
conceptual adventure now is to explore if there is
undefined processes are typical false benchmarking
such a phenomenon at all. It is clear that there are
constituents. Performance measurement by using
numerous types of different benchmarking meth-
some benchmarks moves into true benchmarking
ods and styles. Here we have some examples of at-
when it defines targets for improvement by identi-
tributes, which the benchmarking experts have
fying best practices and adapting them to achieve
given to the benchmarking method. Benchmarking
continuous improvement in one’s own organisation
can be:
(see also Fine & Snyder 1999, Palfreyman 1998).
8

---

ENQA Workshop Reports
Paths of benchmarking
The table is more of an artistic image than an
academic product. There is a lot of unexplained
But finally, do such qualities as “true” or “false”
semantics in it, which I leave to the critical reader
really exist in the benchmarking sphere? Should we
for creative interpretation. The table draws atten-
instead use the term “benchmarking-like activities”,
tion to the fact that benchmarking is not an unam-
which will flourish at the conceptual space of the
biguous method. It has many different sources and
benchmarking constituents? In that space there are
processes, and can lead to multiple results.
many philosophical paths through which the
The benchmarking process in the higher educa-
benchmarking activities flow. I introduce the pro-
tion context is not the same as in business life or in
totype of the benchmarking space in tabular form
the public sector overall. It may not even be the
(see Table 1).
same between universities in different continents.
The table shows that the benchmarking space
There is a great difference between American and
emanates between the sources and quality cultures.
European universities, for example. Higher educa-
The sources give the reasons and the agents for
tion in Europe is mostly arranged with close con-
benchmarking. Quality cultures construct the so-
tact to research, whereas in the United States there
cial context of the benchmarking sphere. It is also
are more teaching universities than research uni-
noteworthy that all the entities in grey area have
versities. The idea of the scientific community –
the dialectic characteristic to act as the elements of
the quality product of the continental philosophy –
both the sources and the cultures. The same entities
with its universal values of truth, collectivity, ob-
form the theoretical forces, which causes the ten-
jectivity and criticism gives mode to the academic
sion for the particular benchmarking method crea-
action in the European context. Participants of the
tion. The outcomes and the results of the
ENQA workshop were predictably quite critical of
benchmarking process will create changes on the
the ranking-oriented “benchmarking” activities.
quality cultures, which have impact on the sources,
Hard values of ranking may be incompatible with
and the “circle” goes round again.
the European spirit of the scientific community,
whereas the cooperation, discourse and tradition of
Table 1. Benchmarking space
S O U R C E S
OWNER
EXTERNAL
INTERNAL
SHARED
CUSTOMER
INTEREST
TO QUALIFY
TO COMPARE
TO IMPROVE
TO WIN
TO COOPERATE
ENERGY
SEEKING FOR
STANDARDS
BENCHMARKS BETTER
BEST
JOINT
TARGET
PRACTICES
PRACTICES
TOPICS
PROCESS
RANKING
ACCREDIT-
COMPARING
(BENCH)
STYLE
ATION
LEARNING
OUTCOME
QUALITY
CERTIFICATES
COOPERATION
ALLIANCE
RESULTS
PRICES
DEVELOPMENT
COMPETITION
Q U A L I T Y C U L T U R E S
9

---

ENQA Workshop Reports
knowledge are more convenient values for it.
Dervitsiotis, K. N. (2000). Benchmarking and
Business Paradigm Shifts. Total Quality
Perhaps there is a kind of social order to “true”
Management. Vol. 11: 641–646.
European benchmarking. Maybe there were some
Epper, R. (1999). Applying Benchmarking to Higher
elements of it present in the workshop. Was the great
Education. Change: 24–31.
metaphor very nearly reached? In Finland there is
Fine, T. & Snyder, L. (1999). What is the Difference
an old proverb, which says, that “It’s worth fishing
Between Performance Measurement and
for salmon, even if you do not catch any.” The same,
Benchmarking. Public Management 80:1.
I believe, is true of benchmarking: it is worth try-
Gooden, S. T & McCeary, S. M. (2001). That Old-
Time Religion: Efficiency, Benchmarking and
ing.
Productivity. Public Administration Review. Vol. 61:
116–120.
Grant, R.
Final comments
(2001). Improving Service Quality with
Benchmarks. Educause Review. Vol. 36: 12–13.
Hodginson, L. (2000). Benchmarking Key Skills
It was a very invigourating workshop. The most
Using National Standards: the Open University
inspiring moment was collaborating in teams with
Experience. Jackson, N. & Lund, H. (Eds.)
people from different universities and different
Benchmarking for Higher Education. Society for
Research into Higher Education & Open University
countries. A similar kind of inspiring stimulus may
Press.
be one of the reasons why stories about bench-
Jackson, N. & Lund, H. (Eds.) (2000).
marking are mostly enthusiastic. While evaluation
Benchmarking for Higher Education. Society for
processes are often very bureaucratic and paper-
Research into Higher Education & Open University
Press.
based, the benchmarking process can offer more
interaction and emotional stimulus, both of which
Karjalainen, A., Kuortti, K. & Niinikoski, S. (2002).
Creative Benchmarking. University of Oulu &
are the most important factors in a meaningful learn-
Finnish Higher Education Evaluation Council.
ing process. Getting the benchmarking passport to
University Press of Oulu.
a strange culture is always a fascinating adventure.
Karlöf, B. & Östblom, S. (1993). Benchmarking: a
Signpost to Excellence in Quality and Productivity.
Chihester Wiley.
Palfreyman, D. (1999) Benchmarking in Higher
Education: A Study conducted by the
Commonwealth Higher Education Management
References
Service. Book Review. Perspectives, Vol. 3: 71–72.
Keehley, P., Medlin, S,, MacBride, S & Longmire,
Wan Endut, W. J, Abdullah, M. & Husain, N.
L. (1997). Benchmarking for Best Practices in the
(2000). Benchmarking institutions of Higher
Public Sector. San Francisco CA: Jossey-Bass
Education. Total Quality Management. Vol 11: 796–
Publishers.
799.
Schofield, A. (2000). The Growth of Benchmarking
Zairi, M. & Leonard, P. (1994). Practical
in Higher Education. Life Long Learning in Europe.
Benchmarking. The Complete Guide. Chapman &
2.
Hall. United Kingdom 1994.
Bohte, J. & Meier, K. J. (2000). Goal Displacement:
Organizational Cheating. Public Administration
Review. Vol. 60: 173–182.
Coleman, R. & Viggars, L. (2000). Benchmarking
Student Recruitment: The UCAS Institutional
Planning Service. Jackson, N. & Lund, H. (Eds.)
Benchmarking for Higher Education. Society for
Research into Higher Education & Open University
Press.
10

---

Document Outline

• Contents

---