0(14/+0) %+'0%'
1.7/' F 1 E EMMM
A Groupware-based Peer Review Process:
An Exploratory Case Study
Bridget N. O'Connor
New York University
bridget.oconnor@nyu.edu

Abstract
This paper uses Lewin’s planned change theory and Rogers’ diffusion of innovation theory as lenses through which to describe what occurred in
an organizationally flat, team-based organization when it rolled out a groupware-based peer review process. The Information Systems Depart-
ment wanted to implement a high-level groupware product. The Human Resource Department wanted to enable team members to rate each other
on their teambuilding skills as well as provide a criterion to be used by the organization’s Policy Committee in awarding biannual bonuses. Lit-
erature related to corporate peer reviews as well as the use of groupware provided the basis for questions posed. Both nondirective and focused
interviews were conducted with key players and a sampling of actual evaluation data was collected. Analysis indicated that perceived manage-
ment (non)commitment to using peer review data and issues of confidentiality may have led many individuals to be skeptical about the value of a
peer review. As to the use of groupware to enable the evaluation, users were quite pleased; however, planners’ initial resistance to using this
technology was evident and early efforts to devise usable reports were troublesome.
Keywords: groupware, peer review, planned change, innovation
Background to the Problem
creasingly, however, organizations are experimenting with sys-
The ability to work effectively as a member of a team is a
tems whereby employees are rewarded for extraordinary efforts
vital skill in most organizations. Teams have been described
(Flynn, 1994; Work week: Evaluations, 1995). The resulting
as the basic business unit of the global economy and team-
problem is how to allocate reward dollars to individuals for team
work as “the final ingredient that makes all the other parts
outcomes.
add up to something greater than the sum of their parts”
As a potential response to this problem, some corporations are
(O’Hara-Devereaux & Johansen 1994, pp. 138-139).
adopting aspects of university faculty peer review systems. In
Teams have advantages over traditional hierarchical organ-
university merit/performance systems, the basic premise is that
izational structures as teams can be flexible and respond
members of an academic community are able to give honest,
creatively and quickly to changing economic and social
evaluative, and constructive feedback to each other. Peer review
forces. Effective teamwork mandates that individuals have
appears to work in academe, if no other reason than historically,
a wide range of communication skills and styles, including
“it’s the way we have always done it.”
the ability to give and take frank criticism, provide self-
examination of team effectiveness, and accept shared re-
Several distinctions complicate corporate peer evaluation efforts,
sponsibility for outcomes (Weisbord, 1987).
however. One primary problem has been identifying and measur-
ing specific performance outcomes for diverse job categories.
As teamwork becomes more and more the norm for the way
Effective work teams are often made up of individuals who have a
work is done, the issue of compensating individual team
wide range of content expertise and come from differing
members for group productivity becomes apparent. Com-
organizational levels; teams are often not truly “peer.” The
pensation is more than just salary--it tells us how well we
second problem has been behavioral, how to get the workforce
are doing; it tells us the value of our work within the larger
itself to accept changes in the way its compensation figures are
organization; and it tells us our value as compared to our fel-
determined.
low workers. In most organizations, compensation figures
are computed based on set salaries or hourly wages. In-
Additionally, since teams may exist for short periods of time or be
ongoing; how often should reviews be done? In a given group,
Material published as part of this journal, either on-line or in print, is
particularly a virtual group, an individual may have more--or
copyrighted by the publisher of Informing Science. Permission to make
less--responsibility than others; should criteria differ? Problems
digital or paper copy of part or all of these works for personal or class-
room use is granted without fee provided that the copies are not made
can also arise as the very concept of peer evaluation differs from
or distributed for profit or commercial advantage AND that copies 1)
most existing corporate evaluation norms and individuals= ex-
bear this notice in full and 2) give the full citation on the first page. It is
permissible to abstract these works so long as credit is given. To copy
periences with how evaluation should take place. These problems
in all other cases or to republish or to post on a server or to redistribute
compound when a flat organization rewards team efforts rather
to lists requires specific permission and payment of a fee. Contact Edi-
than individual efforts. Even when concrete team outcomes are
tor@gise.org to request redistribution permission.

---

41729#4'g$#5'& ''4 '8+'9 41%'55
measurable, the question is how to recognize and compen-
are mixed. In a survey investigating self-directed teams, Hitch-
sate a specific individual for his or her contribution to a final
cock (1996) found that linking peer reviews to merit pay was not
product.
well received. However, peer reviews were more accepted in
field studies where it was reported that management had clearly
Team peer evaluations also add new layers of administrative
identified team priorities, results, and appropriate measures (Tal-
issues for human resource professionals. Even if everyone
bott, 1994; Zigon, 1997). It is clear that organizations are
understands and buys into the peer evaluation process, it is
working to develop measures of team performance, citing the
difficult to ensure the compilation of anonymous data in a
need for the soft skills required for effective teamwork (Covey,
timely manner and in an appropriate format for those who
1996). Peer reviews have been described as a way for teams to
will use it. This is why groupware was considered to be a
support each other in improving both individual and organiza-
potential enabler of the peer review process, as it supports
tional performance (“The Power of Peer Review,” 1994).
group processes and productivity (as opposed to individual
productivity). Often called group support systems (GSS)
Studies related to the use of information technology have shown
or electronic meeting systems, groupware products are
that groupware can support data collection, processing, and dis-
evolving.
tribution. Groups using technology work longer (Steeb &
Johnston, 1981; Gallupe & McKeen, 1990; Caouette &
Level 1groupware products include screen sharing capabili-
O’Connor, 1998) and have higher quality outcomes on certain
ties and electronic mail. Level 2 groupware products are
tasks, such as idea generation (Valacich, in press). Longitudinal
virtual toolboxes of support for group processes such as
research suggests this can be heightened over time (Hackman,
group writing, idea organizing, and voting. Level 2 prod-
1991; Martz, Vogel, & Nunamaker, 1992). Other studies, how-
ucts can perform data manipulation and statistical analysis
ever, have shown that some users do not like using the computer,
as well as provide hardcopy printouts of exact textual dis-
depending upon their own expectations, previously existing or-
cussions (Vogel, 1990). Level 3 systems, currently under
ganizational norms, and/or their own experiences in using the
development, include filtering systems, agents, and tools
groupware previously (O=Connor & Bronner, 1995). As group-
such as Roberts Rules of Order (Hsu, 1993).
ware has been implemented, the tools have been adapted to the
An Exploratory Case Study
context and the priority of the task (Bikson & Eveland, 1996). In
This paper demonstrates how an information systems direc-
an investigation of a Level l system, team members using group-
tor, a human resource executive, and a corporate trainer
ware reported more confidence in their outcomes (McClernon &
worked together to design and implement a technology-
Swanson, 1995). Increased information handling also raises
supported peer review system and the results of that effort.
questions regarding who has access to information and additional
This is an exploratory case study showing how two innova-
questions related to problems of misinterpreting information
tions--a peer review process and a Level 2 groupware
(Smith & Vanecek, 1990).
product--were rolled out together as a means to enable the
This case research was guided by the following questions:
peer review process to happen easily, anonymously, and effi-
ciently. The specific groupware used was GroupSystems
What formal and informal activities related to the rollout of the
for Windows ™, which allowed peer groups to complete re-
technology-based peer review process took place?
views at their own time and in their own offices (different
What were individual reactions to the peer review process?
time/different place). The statistical reports generated
through the system were expected to provide useful perform-
What impact (if any) did the use of the technology have on the
ance data quickly.
overall value of the peer review process?
Individual team members rated and described each other's
The Research Method
contributions to a team’s performance, and these ratings
This corporation was a specialized financial-guarantee insurance
were to be used to provide feedback to individuals on their
company based in New York City where deals were the basis for
teambuilding skills as well as serve as one criterion for Pol-
a project director assigning the right mix of individuals to work
icy Committee members who were awarding biannual
together. Rarely did the exact same team work on a given deal
bonuses. While peer review had no history in this organiza-
and deals could take as little as two hours or as long as six
tion, a bonus system was already in place as an established
months to complete. The corporation also had a history of using
compensation method. Moreover, for most users, this was
computers in all phases of their day-to-day work and some indi-
their first experience using a Level 2 groupware product.
viduals had had some experiences using an earlier version of the
groupware, GroupSystems V. Because it was organizationally
Prior Research
flat, this medium-sized corporation provided a useful backdrop
Evidence exists that team-based peer review procedures are
for an investigation of peer review and groupware as teamwork
being implemented in organizations; however, the findings
12

---

4+&)'6 T s10014
was the primary form of decision making and all employees
views, which ranged in length from 20 minutes to 55 minutes,
were technologically adept.
were tape recorded and subsequently transcribed. Data were
analyzed by examining recurring themes and triangulating find-
The case study has been called the most famous (or infa-
ings to ensure reliability.
mous) methodology in business education (Gay & Diehl,
1992). As a research method, case study has a history of
being used successfully in investigating human resource de-
The Peer Review Process
velopment in small to medium sized companies (Rowden,
Results The introduction section of the peer review questionnaire
1995). A case study allows many variables to be examined
that I had drafted set out objectives and guidelines (see Appendix
at the same time in a real-world context. While insights
B). The peer review questions themselves were based on the ten
garnered from case study research lack generalizability, they
attributes of effective team work: commitment, acceptance, clari-
do provide insights as to what happened and why when in-
fication, belonging, involvement, support, achievement, pride,
vestigating a new phenomenon (Gay & Diehl, 1992). The
recognition, and satisfaction (Kormanski & Mozenter, 1987).
case discussed here can be labeled “partnership research.”
Specific attributes were described relative to the work being
In partnership research, the research problem is derived
done, and required respondents to rate each of his or her team
from practice, but both partners anticipate using the results
members on a five-point Likert scale and to offer a critical inci-
of the research for different purposes (McLean, 1995); the
dent as evidence of the rating. For example, an item measuring
organization to improve practice and in this instance, the re-
support read:
searcher to improve education.
He/she has good communication skills; motivates
the team; serves as a role model; takes time to
Data Collection and Analysis
mentor/coach; provides timely feedback; is
To determine the value of the peer review process from
approachable; demonstrates sensitivity; does not
management’s viewpoint, I conducted nondirective (unstruc-
favor one team member over another. Rating:
tured), face-to-face interviews with the chief executive
officer, the information systems director, the human resource
___________
executive, and the corporate trainer at all stages of the proc-
Critical Incident:
ess. The nondirective interview is flexible and because as
Much thought and discussion went into the final decision to have
the interviewer does not provide direction, respondents "can
raters electronically sign their evaluation reports. Because it
be encouraged to relate their experiences, describe whatever
was deemed important to have some level of accountability, raters
events seem real to them, provide their own definitions of
were asked to add their names to their evaluations. Reviewers
the situations, and to reveal their opinions and attitudes as
were promised that they would only be connected with a specific
they see fit" (Frankfort-Nachmias & Nachmias, 1996, p.
review should it be apparent that a review was not given seriously
235). Nondirective interviews were considered an appropri-
or was extraordinarily negative. Additionally, individuals were
ate data collection method because the topic of a technology-
repeatedly assured that no one other than the human resource ex-
supported peer review could be perceived as complex and
ecutive who was compiling the data could know specifically who
emotionally loaded (Merriam and Simpson, 1995). Nondi-
had said what about whom, and this information would not be
rective interview data can provide insights not only on what
readily available as the sign “sheet” was only linked to the re-
happened, but how and why and from various perspectives.
views and not automatically available.
To see first-hand how the peer review instrument was used, I
Team members had access to the software and the evaluation files
reviewed (anonymously) responses to the peer review in-
via a local area network that connected their office computers to a
strument. Additionally, focused (structured) telephone
central file server. Within the groupware, the information sys-
interviews were held with four participating individuals who
tems director had established an electronic folder for each of four
volunteered to discuss their experiences. While the focused
specific deals (projects), and the twenty individual reviewers had
interview is the least flexible of interviews, its use reduced
access to only those folders for which he or she was a deal par-
the risk that the order and the wording of questions would
ticipant. Signing on required users to click on the Agenda icon,
influence responses (Frankfort-Nachmias & Nachmias,
and then identify a specific deal. At that point, eligible reviewers
1996). The interview guide was divided into two parts, one
were automatically ushered into the vote tool, where they were
evaluating the peer review process itself and the other the
asked to “click” the name of the individual they were rating, then
use of the groupware (see Appendix A). All interviewees
rate the individual as described earlier. Changes could be made
were assured of complete confidentiality; I would be the
on evaluations up to the point when the ballot was cast.
only person to have access to the interview data or to know
Compiled results would be available to the human resource ex-
who had been interviewed. The focused telephone inter-
ecutive, who would compile ratings on individuals and forward

13

---

41729#4'g$#5'& ''4 '8+'9 41%'55
the ratings to the individual’s manager and the organiza-
were quite adept and at ease with the process; examples of appro-
tion’s 12-member Policy Committee. Managers could share
priate critical events follow:
results with the individual being rated and the Policy Com-
He has done a good job of assessing some of the underlying op-
mittee would have an additional criterion for the biannual
erational and credit issues and working through the details to
bonus discussion process.
make sure the transaction mechanics are properly structured.
Results
Communications in the form of email status updates are a key
The work of both Everett Rogers and Kurt Lewin provided
strength. He has also done an excellent job of alerting team
the lens for examining this case. Everett Rogers explained
members and management of potential problems and delays in ar-
that characteristics of an innovation influence its infusion.
ranging liquidity.
Rogers explained that the rate of adoption differs depending
on five characteristics: relative advantage, compatibility,
She has very good judgment skills and knows when to stand firm
complexity, trialability, and observability (Rogers, 1983).
and when to give in when negotiating this deal. She was able to
This study involved the infusion of two innovations simulta-
develop a model to present value the arbitrage amount so we
neously, and these terms can help us understand the
would know how much to take as an upfront fee. We were able to
innovations’ acceptance. As a means to put the case in per-
get an additional $500,000 in fees this year.
spective, Kurt Lewin’s planned change model calls for
However, much of the resultant data was often vague or incom-
continual progression through three stages: (1) unfreezing;
plete. The following critical incidents demonstrate these
(2) changing (or moving); and (3) refreezing. (Lewin,
problems:
1972) Unfreezing is the stage where individuals become
The best person to work with in this company!
aware of the innovation; changing involves actually using
the innovation; and refreezing, a determinant of an innova-
He tends to employ his knowledge of the industry as it pertains to
tion’s acceptance, is where the innovation becomes “the way
his area of expertise in a “take it or leave it” manner rather than
we do things around here.” These concepts will be used to
trying to work with the team to achieve results.
describe what happened in this case and why.
She was good at keeping the customer happy and addressing cus-
What formal and informal activities related to the rollout of
tomer concerns.
the technology-based peer review process took place?
He definitely counts on others to do his work. He will generally
Unfreezing Stage. Training played a crucial role in the un-
take on the easiest components of the transaction, not caring much
freezing stage. An informational meeting was held to
for the “details” such as legal documentation or other intricacies
describe the organization’s rationale for implementing a peer
(mechanics) of the transaction. He is very quick, however, to
review, as well as provide guidelines for how to provide
opine in front of senior management, so that they have the im-
constructive feedback. The procedures for the process were
pression he has done all the work.
outlined and an opportunity for questions and discussion was
What were individuals’ reactions to the peer review process?
provided. A week or so later, individuals were trained to use
the software.
Overall, while it had been anticipated that team members would
easily find words to describe each other’s performance, many had
Changing Stage. Upon the completion of each of four
difficulty in doing so. While it had been anticipated that complet-
deals, the twenty participants in those deals were asked to
ing reviews would take only a few minutes each, individuals
complete the peer review instrument. The human resource
reported spending up to an hour or two on a single review. While
executive, the information systems manager, and the corpo-
it had been anticipated that the initial training on understanding
rate trainer were all available for questions and support
the purpose of the reviews and completing reviews would be ade-
throughout. These planners reported that users reported
quate, it was not.
problems with the system and with a too-lengthy question-
naire. To address these concerns, procedures were changed
Consistent with the work of Talbott (1994) and Zigon (1997),
mid-course to allow individuals to select only two or three
priorities, results and proper measures were important as several
colleagues to rate per deal, and not require responses to
of those interviewed reported that they understood an official and
every question.
an unofficial reason for the peer reviews. Nearly all of the indi-
viduals interviewed reported they had never had any feedback: “I
Planners also found that numerical ratings were often not
never knew what the results were or where they [reviews] went or
consistent with the critical event described. Despite initial
how they were tabulated or what feedback was given.” This par-
training, describing critical events turned out to be a chal-
ticular interviewee related his lack of feedback to the unofficial
lenge for most respondents. Several individuals, however,
reason that data were needed to deal with two individuals known
to be non-performers and who were consequently moved to other
14

---

4+&)'6 T s10014
positions. The individual who did report getting feedback
more understandable proved to be a difficult (but eventually mas-
did not feel that data were used by top management as in-
tered) learning experience.
tended: “Eventually, yeah, I did get feedback. It [feedback]
To the contrary, individual users reported that the groupware was
was very good and you know, that is why I say that it is all a
an enabler: “Yes, I liked the fact that you could log on and log
political game here because despite the fact that I was
off, log back and then submit it….I was using it all the time.”
ranked high in the company, I was passed over for promo-
Another said, “It was very user friendly once you got into it, so
tion. … Yes, I think it [the peer review process] is
that was just fine.” Yet another user said, “I thought the system
important, and it also needs to be treated importantly as well
was pretty easy to use even though I missed the training. It
by senior management.” Later in the interview, she said:
seemed to follow through. And it made sense.” These techno-
“Where the peer review process was meant to work was for
logically-literate individuals were quite at ease in understanding
those people who were most at risk from suffering from it.”
and using the system, and when asked if they would like to use
The peer review instrument itself, likewise, met with mixed
the system in the future for such a process, responses were posi-
reactions; one comment was “[I] worked in a very small (at
tive: “Yes, I would, definitely. Because it was very user
that time) group of people so you really knew the players
friendly.”
pretty well which is why it [the instrument] was easy to use,
Consistent with earlier studies (Steeb & Johnston, 1981; Gallupe
and I felt comfortable using it.” Another interviewee re-
& McKeen, 1990), individuals believed the groupware was an
ported that “If someone gets all good [response], they are
appropriate fit for the peer review task. However, no one took the
good. If you start asking questions with 1 through 5, people
promise of confidentiality seriously: “Given the way I thought
start to tune out…fade out. Probably too many questions
this stuff was being essentially used, I would almost be more
and too many variations. Probably fewer questions and just
comfortable writing something on paper and handing it to some-
more casual: was the person weak, adequate, or strong?
body. …It may have been better if I knew exactly what was going
That is all that is really meaningful on a team.” The view
to happen with this data. How was it going to be used or not
that the questionnaire itself was too lengthy was explained,
used.” Another interviewee said, “I think people took it with a
also: “The first one [deal], I think five people answered
grain of salt in terms of confidentiality. I think it depends on peo-
every one [question]. …..it was too long a process. And
ple, personalities, to keep this confidential or not.” One
then I cut it down; I think I answered 6 or 7 which were the
interviewee said he told everyone he evaluated that he had given
most important. Some of them didn’t apply for what some
them high marks.
people were doing but that was my one feeling….that it was
a little long. But it was very good.”
Lessons Learned
Concordant with Hitchcock (1997), peer review was not
This paper described what happened when two innovations were
seen as an appropriate measure for bonuses. There was
introduced simultaneously. Results were mixed. Perhaps this is
general wariness in thinking that evaluation data were actu-
because while management was convinced of the value of peer
ally used in determining a monetary award: “The role of a
review, individuals were skeptical. While top management was
good manager is to figure out what motivates the people that
convinced of the value of groupware, implementers were not. To
he manages, and in some cases it is not necessarily mone-
succeed, these two innovations—peer review and groupware—
tary. So yeah, the bonuses were fine, but I know that the
needed to be seen by everyone as having an advantage over the
teams I worked on generated a lot of money for the com-
way bonuses were traditionally awarded if they were to refreeze,
pany, and I believe bonuses were based on revenue, not
to become part of the organizational culture.
personal qualities or anything.”
Lesson learned: No good idea succeeds on its own merits; con-
What impact (if any) did the use of the technology have on
tinual experimentation with peer review and the use of distributed
the overall value of the peer review process?
groupware systems is needed. Keep the process itself as simple
as possible.
Planners, including myself, initially considered that the rea-
son the peer review process was not as successful as
The dual goal of the peer review process, to provide descriptors of
anticipated was because users were finding the technology
how individuals were able to work as part of a team for bonus de-
difficult to use. However, this was not the case as only the
liberations as well as confidential feedback given to reviewers
information systems executive who set up the system and the
was not seen by those interviewed as complementary outcomes
executive who had been charged with compiling the review
and were not totally understood. In terms of value, individuals
data expressed negative reactions to its use. Reports were
needed more assurance that their reviews would be taken seri-
not generated as quickly or as easily as had been expected,
ously and used for their intended purposes. Only one of the
and learning to format the reports so that data would be
individuals interviewed reported that she had had review feed-
back; and she had (mis)understood that the high marks she

15

---

41729#4'g$#5'& ''4 '8+'9 41%'55
received from her colleagues would be used for promotion
Epilogue
purposes, as opposed to bonus deliberations. Others re-
Refreezing Stage. The purpose of this section is to complete
ported that a perceived unofficial purpose, to remove
Lewin’s paradigm—unfreezing, changing, and refreezing. Since
specific individuals, dominated the process.
the conclusion of this investigation, the organization studied has
Lesson learned: State intended peer review objectives
merged with a much larger organization. However, informal in-
clearly; ensure everyone understand objectives; and follow-
terviews with top management, confirmed by the focused
through on intentions.
interviews, suggest that the innovative culture of the smaller or-
ganization (the one studied) is taking precedence. Despite mixed
Moreover, promises of confidentiality were not accepted at
reviews on the peer review process, management is reportedly
face value. Electronically signing reviews, meant to ensure
committed to both peer review and continuing its use of group-
review accountability, also meant that someone could know
ware; they consider this investigation to be a pilot study or
who said what. While this perceived lack of confidentiality
learning experience. The human resource executive is currently
was not cited as a major issue by individuals participating in
working to revise the peer evaluation instrument and evaluate the
the process, it may be seen as a potential stumbling block,
fit of the specific groupware tools used in data collection as a
thus inhibiting openness.
means to ensure more useful reports. In addition, information
Lesson learned: Design the groupware system to ensure
and operational training will be revised and more concerted ef-
confidentiality.
forts will be made to assure individuals that data will be used as
planned.
Concluding Comments
In the short term, work will be done to ensure that the complexi-
The lessons learned here may initially seem obvious. How-
ties of the peer review process are ironed out and another pilot
ever, we learn from our experiences and it is noteworthy to
test will ensure a more usable instrument and reporting process.
consider that this rather “ideal” site—an organization where
In developing a blueprint for ensuring the success of both innova-
individuals routinely work in teams and are technologically
tions, training will continue to play an important role in all stages
adept—had growing pains in establishing a technology sup-
of change, eventually changing the way “we do things around
ported peer review process. Given that teams are the
here.”
building blocks of many of today’s organizations, continued
reports of strategies that assess individuals’ teambuilding
References
skills and reward good performance are valuable. Man-
Bikson, T. K. & Eveland, J. D. (1996). Groupware implementation: Re-
agement will increasingly be faced with issues of how to
invention in the sociotechnical frame. Proceedings of the ACM
reward a given individual for team outcomes, and we need to
Conference on Computer Supported Cooperative Work (pp. 428-
learn from what others have done.
437). New York City: ACM.
A peer review process would seem to be compatible with the
Caouette, M. J. & O’Connor, B. N. (1998). The impact of group support
systems on corporate teams’ stages of development. Journal of Or-
way work is done in a team-based, technology-driven or-
ganizational Computing and Electronic Commerce 8(1), 57-81.
ganization, and this investigation shows that even
innovations that appear to be a perfect fit require planning
Covey, S. (1996). Unstoppable teams. Executive Excellence, 13(7), 7-8.
and have their own learning curve. Rogers’ premise that
Flynn, G. (1994). More companies use bonuses in place of traditional
the characteristics of an innovation impact its infusion into
pay increases. Personnel Journal, 73(11), 16.
an organization is important here. Key to the activity is that
Frankfort-Nachmias, C. & Nachmias, D. (1996). Research Methods in the
the technology-supported peer review process be considered
Social Sciences (3rd ed). New York: St. Martin's Press.
valuable and an improvement over the way things are done--
Gallupe, R. B. & McKeen, J. (1990). Enhancing computer-mediated
in this case, a bonus deliberation. Perhaps more testing, or
communications: An experimental study into the use of a decision
trialability, and experience in actually doing reviews, or ob-
support system for face-to-face versus remote meetings. Information
servability, by both management and individual users will
and Management, 8, 1-13.
support the usefulness of these innovations. A shorter, more
Gay, L.R. & Diehl, P. L. (1992). Research methods for business and
direct and truly anonymous questionnaire could reduce the
management. New York: Macmillan.
complexity of the process, enhance validity, and provide
more useful data. In short, to refreeze the peer review proc-
Hackman, J. R. (1991). Groups that work (and those that don’t). San
Francisco: Jossey-Bass.
ess in this organization, the value, compatibility, complexity,
trialability, and observability of the two innovations investi-
Hitchcock, D. (1996). What are people doing around peer review? Jour-
gated here must be continually addressed.
nal for Quality and Participation, 19(7), 52-55.
Hsu, J. & Lookwood, T. (1993) Collaborative computing. Byte, 28(3)
112-120.
16

---

4+&)'6 T s10014
Kormanski, C., & Mozenter, A. (1987). A new model of team
Rowden, R. W. (1995). The role of human resource development in suc-
building: A technology for today and tomorrow. In J. W. Pfeif-
cessful small to mid-sized manufacturing businesses: A case study.
fer (editor), The 1987 Annual: Developing Human Resources
Human Resource Development Quarterly 6(4), 355-373.
(pp. 255-267). San Diego: University Associates.
Smith, J. & Vanecek, M. (1990). Dispersed group decision making using
Lewin, K. (1972). Quasi-stationary social equilibrium and the prob-
nonsimultaneous computer conferencing: A report of research. Jour-
lems of permanent change. In N. Margulies and A.P. Raia
nal of Management Science, 7(2), 71-92.
(editors) Organizational development: Values, process, and
Steeb, R. & Johnston, S. C. (1981). A computer-based interactive system
technology. New York: McGraw-Hill.
for group decision making. IEEE Transactions on Systems, Man,
Martz, W. B.; Vogel, D. R.; & Nunamaker, J. F., Jr. (1992). Elec-
and Cybernetics, 11(8), 544-552.
tronic meeting systems: Results from the field. Decision
Talbott, S. P. (1994). Peer review drives compensation at Johnsonville.
Support Systems, 8(2), 141-158.
Personnel Journal, 73(10) 126-132.
McClernon, T. & Swanson, R. (1995). Team building: An experi-
Valacich, J. S.; Paranka D.; George, J. F., & Nunamaker, J. F. (In press).
mental investigation of the effects of computer-based and
Communication concurrency and the new media: A new dimension
facilitator-based interventions on work groups. Human Re-
for media richness. Communication Research.
source Development Quarterly 6(1), 39-58.
Vogel, D. R. (1990). Research on electronic meeting systems, in Re-
McLean, G. N. (1995). Partnering in human resource develop-
search issues in information systems: An agenda for the 1990’s,
ment. Human Resource Development Quarterly 6(1), 1-4.
Jenkins, A. M.; Siegel, H. S.; Wojtkowski, W.; and Wojtkowski, G.,
Merriam, S. B. & Simpson, E. L. (1995). A guide to research for
eds. Dubuque, IA: Brown, p. 136.
educators and trainers of adults. Second edition. Malabar,
Weisbord, M. R. (1987). Productive workplaces: Organizing and man-
FL: Krieger Publishing Company.
aging for dignity, meaning, and community. San Francisco: Jossey-
O’Connor, B. N. & Bronner, M. (1995). Facilitating curriculum
Bass.
development: The role of electronic meeting systems. NABTE
Work week: Evaluations (1995, May 2). The Wall Street Journal, p.
Review, (22), 5-9.
A,1:5.
O’Hara-Devereaux, M. & Johansen, R. GlobalWork: Bridging dis-
Zigon, J. (1997). Team performance measurement: A process for creating
tance, culture & time. San Francisco: Jossey-Bass, 1994.
team performance standards. Compensation and Benefits Review,
The power of peer review (1994). Training and Development,
29(1), 38-47.
48(7), 38-41.
Rogers, E. M. (1983). Diffusion of innovations. Third edition.
New York: Macmillan.
Appendix A
Structured Telephone Interview Guide
Focused Telephone Interview Guide
Thank you for taking the time to talk with me today. As you know, we are discussing the peer review process you went through
last year. Your perspectives on both the peer review itself and your use of GroupSystems are needed. Using the following scale,
please tell me the number that describes the degree to which you agree or disagree with the following statements. You will have
an opportunity to explain your responses; the number is being used strictly as a way of getting you to begin discussing the state-
ment.
5 4
3 2
1
Totally
Somewhat Agree
Neutral Somewhat Totally
Agree
Disagree
Disagree
Part I. Peer Review Process
1. Approximately how many reviews did you complete? _______
2. I understand why and how the reviews took place.
1 2 3 4 5
Comments:
3. The questionnaire was an appropriate and easy to use evaluation instrument.
1 2 3 4 5
Comments:
4. I was comfortable in making informed ratings of my team members.
1 2 3 4 5
Comments:

17

---

Groupware-based Peer Review Process
5. I was comfortable having team members rate my own performance.
1 2 3 4 5
Comments:
6. I have no doubts that data from reviews will be kept confidential.
1 2 3 4 5
Comments:
7. I personally believe that a peer review process is important.
1 2 3 4 5
Comments:
Is there anything else you'd like to talk about with regard to the peer review process?
Part II. Use of GroupSystems Technology
Using the following scale, please tell me the number that describes the degree to which you agree or disagree with the following
statements. Again, you will have an opportunity to explain your responses; the number is being used strictly as a way of helping
you begin discussing the statement.
5 4
3 2
1
Totally
Totally
Somewhat Agree
Neutral Somewhat
Agree
Disagree
Disagree
1. Approximately how much time did you spend completing each review? ___
_ minutes
2. I routinely use computers in my daily work.
1 2 3 4 5
Comments:
3. I have used GroupSystems previously for other tasks.
1 2 3 4 5
Comments:
4. The GroupSystems training enabled me to competently use the system to complete the reviews.
1 2 3 4 5
Comments:
5. The system was an appropriate “fit” for the peer review process.
1 2 3 4 5
Comments:
6. The relative anonymity of the system allowed me to say things I
might not have said otherwise.
1 2 3 4 5
Comments:
7. I am looking forward to using GroupSystems again.
1 2 3 4 5
Comments:
Is there anything else you'd like to talk about with regard to the peer review process?
Appendix B

Peer Review Introduction
The objectives in implementing this peer review:
Self-development: feedback should be used as a means to behavioral change.
Measure teamwork effectiveness: feedback from one’s peers on how a team leader or team member is performing, and the
resulting modified behavior, can lead to increased productivity.
Development of a “partnership attitude.”
Provide the Policy Committee with another tool to be used in evaluating an employee.
Guidelines:
Reviews must be signed. Any review without a signature will not be used.
Please rate each peer (team leader or team member) on the 5 point scale and add the critical event that supports your rank-
ing. Any ranking without the corresponding critical event will not be used. A critical event is a specific situation where the
skill/trait was observed by you.
Criticism should be constructive.
Review must be done on groupware.
Review on: _________________ (name of person being reviewed)
Your name: _____________________________
Rating scale
5
4
3
2
1
Exceptional
Superior
Consistently
Could
Improvement

All facets
Most facets
Capable
Improve
Required
18

---

Document Outline

• Abstract
• B
• Background to the Problem
• An Exploratory Case Study
• Prior Research
• The Research Method
  • Data Collection and Analysis
  • The Peer Review Process
• Results
• Lessons Learned
• Concluding Comments
• Epilogue
• References
• Appendix A
• Structured Telephone Interview Guide
  • Focused Telephone Interview Guide
  • Part I. Peer Review Process
  • Part II. Use of GroupSystems Technology
• Appendix B

---