The application of Dempster-Shafer theory demonstrated with justification provided by legal evidence

Judgment and Decision Making, Vol. 2, No. 5, October 2007, pp. 257–276
The application of Dempster-Shafer theory demonstrated with
justi?cation provided by legal evidence
Shawn P. Curley?
Department of Information & Decision Sciences
University of Minnesota
Abstract
In forecasting and decision making, people can and often do represent a degree of belief in some proposition. At least
two separate constructs capture such degrees of belief: likelihoods capturing evidential balance and support capturing
evidential weight. This paper explores the weight or justi?cation that evidence affords propositions, with subjects
communicating using a belief function in hypothetical legal situations, where justi?cation is a relevant goal. Subjects
evaluated the impact of sets of 1–3 pieces of evidence, varying in complexity, within a hypothetical legal situation.
The study demonstrates the potential usefulness of this evidential weight measure as an alternative or complement to
the more-studied probability measure. Subjects’ responses indicated that weight and likelihood were distinguished;
that subjects’ evidential weight tended toward single elements in a targeted fashion; and, that there were identi?able
individual differences in reactions to con?icting evidence. Speci?cally, most subjects reacted to con?icting evidence
that supported disjoint sets of suspects with continued support in the implicated sets, although an identi?able minority
reacted by pulling back their support, expressing indecisiveness. Such individuals would likely require a greater amount
of evidence than the others to counteract this tendency in support. Thus, the study identi?es the value of understanding
evidential weight as distinct from likelihood, informs our understanding of the psychology of individuals’ judgments of
evidential weight, and furthers the application and meaningfulness of belief functions as a communication language.
Keywords: belief functions, evidential weight, likelihood, Dempster-Shafer theory, legal evidence.
1 Introduction
i?cations of beliefs that fall short of knowledge. The
analysis highlights two separate criteria along which such
Probabilities are useful when acting in the absence of
beliefs may be quali?ed: truth and justi?cation. This the-
complete knowledge, e.g., in forecasting or decision mak-
oretical distinction forms the basis of a long-standing dif-
ing. Such probabilities are interpreted as measures of de-
ferentiation between Pascalian probability based on like-
grees of likelihood and are assessed against a criterion of
lihood relative to a criterion of truth and Baconian prob-
truth (von Winterfeldt & Edwards, 1986). Scoring rules,
ability based on support relative to a criterion of justi?-
as assessments of the quality of probability judgments,
cation. (See Shafer, 1978, for an excellent historical dis-
operate from this perspective, comparing likelihood as-
cussion). The distinction is also the basis of a common
sessments to actual outcomes, in an application of the
differentiation between the weight and the balance of ev-
truth criterion (see, e.g., Yates, 1990, chap. 3).
idence that can be traced to Keynes (1921) and which has
However, from the very origins of probability theory,
played a major role in motivating the study of ambiguity
scholars recognized that truth is not the only criterion
in decision-making beginning with Ellsberg (1961).
of potential interest for interpreting probabilities. Smith,
Benson and Curley (1991) tied this recognition to a philo-
In short, likelihoods are intended to capture the bal-
sophical analysis of knowledge as “justi?ed true belief”
ance of evidence and are connected with the criterion of
(e.g., Shope, 1983) and to the use of probabilities as qual-
truth. If A is true, not-A is false. To the degree that the
evidence favors A, the balance of evidence moves toward
?The author wishes to thank James I. Golden for his assistance with
A and away from not-A in equal measure. The weight of
the experiment, and the editor and reviewers who provided such helpful
evidence is connected with the criterion of justi?cation.
feedback on an earlier version of the manuscript. This research was sup-
Weight depends upon the quantity and credibility of the
ported by the Decision, Risk, and Management Science program of the
evidence: How much good evidence is there? How well
National Science Foundation. Address: Shawn P. Curley, Department
of Information & Decision Sciences, University of Minnesota, 321 19th
does the evidence afford any differentiation of possibili-
Avenue South, Minneapolis, MN 55455 USA.Email: curley@umn.edu.
ties?
257

---

Judgment and Decision Making, Vol. 2, No. 5, October 2007
Evidential weight
258
Unlike evidential balance, evidential weight does not
ment for applying Dempster-Shafer theory is based on
imply complementarity. In probability theory, when the
correspondence between aspects of evidential weight and
judgment of one hypothesis increases, the sum of the
unique features of the theory, e.g., its noncomplementar-
judgments for the remaining hypotheses must decrease
ity and the natural representation of ignorance, i.e. the
by the same amount. In truth, one and only one of a mu-
case where no information is present (Curley & Golden,
tually exclusive set of events can occur, thus likelihoods
1994).
should exhibit complementarity, and probabilities capture
In terms of previous work using Dempster-Shafer the-
this feature.
ory, most prior research with this system has been theo-
In contrast, evidential weight as a construct, grounded
retical, for example, in pursuing the use of belief func-
in the criterion of justi?cation, is not expected to exhibit
tions for propagating uncertainty in AI/expert systems in
this property. Increased support for one possibility does
addition or instead of using probabilities (e.g., Barnett,
not necessarily impinge on the support for other possi-
1981; Cohen & Shoshany, 2005; Gillett & Srivastava,
bilities. The belief functions of Dempster-Shafer theory
2000; Henkind & Harrison, 1988; Yang, Liu, Wang, Sii
are discussed in this paper as justi?cation-based measures
& Wang, 2006).
that do not incorporate complementarity as a necessary
Although sparse, there is some suggestive empirical
axiom.
work. The cited work of Grif?n and Tversky (1992), di-
One source of the confusion between the constructs of
rectly, and the extensive work on the effects of ambiguity
likelihood and weight, and of the measures attached to
in decision making (e.g., Camerer & Weber, 1992; Cur-
them, is that these constructs and measures generally cor-
ley, Yates & Abrams, 1986; Einhorn & Hogarth, 1986;
relate. A useful analogy can be drawn here with height
Hogarth & Kunreuther, 1989), indirectly, testify to the
and weight as two aspects of size. Though these mea-
relevance of evidential weight to decision behavior. In ad-
sures correlate, they capture distinct size constructs. Sim-
dition, responses in hypothetical legal contexts that em-
ilarly, probabilities as measures of likelihood and belief
phasize justi?cation exhibit noncomplementarity of de-
functions as measures of justi?cation may correlate, but
grees of belief in a manner consistent with the tenets of
they capture different degree-of-belief constructs. Grif-
Dempster-Shafer theory (Curley & Golden, 1994; Schum
?n and Tversky (1992) provided a demonstration of the
& Martin, 1987; van Wallendael & Hastie, 1990). Briggs
usefulness of the distinction, showing how the inclusion
and Krantz (1992) adopted a measurement perspective
of considerations of weight, in addition to the balance of
and demonstrated that judgments of evidential strength
evidence, can serve to explain various empirical charac-
are separable. That is, subjects “showed clear separation
teristics of con?dence judgments.
of relevant from irrelevant evidence and of designated
There are a number of situations in which justi?cation
from surrounding relevant evidence” (p. 77). In sum, the
is of primary interest to the decision maker, or of interest
results support the value and viability of measuring ev-
in addition to truth. For example, justi?cation is of inter-
idential weight as distinct from the more commonly as-
est in legal settings (where the goal is to remove doubt),
sessed construct of likelihood.
in stock analysis (for which the emphasis is upon justi-
Since likelihood judgments have received more atten-
fying recommendations to clients), in diagnostic tasks in
tion than weight judgments and are often confused with
which the truth is not feasibly determinable (e.g., within
them, particular emphasis must be placed on this dis-
public policy debates), and in scienti?c inference (cf. Ray
tinction. Speci?cally, important distinctions from dis-
& Krantz, 1996).
cussions in the literature need to be drawn: separating
Despite this history and their potential usefulness,
justi?cation-based measures such as in the present appli-
measures of justi?cation have been little studied empir-
cation of Dempster-Shafer theory from weak theories of
ically or been confounded with measures of likelihood.
likelihood and from the theory of subjective probability
The research has probably been somewhat hampered by
called Support Theory.
the respective and different natures of truth and justi?-
cation. Probability theory as capturing likelihoods bene-
1.1 Important Distinctions
?ts from the ultimate realization of the truth in many in-
stances for which it is applied and because of the under-
1.1.1 Weak Measures of Likelihood
pinnings of randomization and relative frequency from
which it historically derives (Curley, in press; Hacking,
With accumulating evidence that Expected Utility (EU)
1975). The application of a system used for capturing
theory does not provide an adequate descriptive theory
justi?cation, and the use of Dempster-Shafer theory for
of choice, one of the research directions has been to in-
this purpose, is more equivocal about the underlying the-
vestigate weaker theories of choice while maintaining the
oretical mechanisms supporting such judgments (cf. Ray
expectation framework. Often this approach involves re-
& Krantz, 1996; Shafer, 1976; 1981). Here the argu-
laxing or omitting one or more of the axioms that un-

---

Judgment and Decision Making, Vol. 2, No. 5, October 2007
Evidential weight
259
derlie EU theory (e.g., as expressed by von Neumann
use “probability” and “relative support” to describe the
& Morgenstern, 1947, or by Luce & Suppes, 1965), or
likelihood-based idea of balance, and use “strength of ev-
its close cousin Subjective Expected Utility (SEU) the-
idence” to describe the justi?cation-based idea of weight.
ory (Savage, 1954). The use of weighting functions, like
Unfortunately, such differences in terminology pervade
those in Prospect Theory (Kahneman & Tversky, 1979;
the literature. Respectively, Shafer (1976) distinguishes
cf. Karmarkar, 1978) or in Cumulative Prospect Theory
“chance” and “support,” Shafer and Tversky (1985) dis-
(Tversky & Kahneman, 1992), exemplify an approach in
tinguish “likelihood-based” or “Bayesian designs” with
which a likelihood-based function in choice is modi?ed
“Jeffrey” or “belief-function designs.” In this paper, I will
to accommodate subjects’ behaviors that are incompati-
generally use the terms of likelihood (truth, balance) and
ble with EU and SEU.
evidential weight (justi?cation, support). But, amidst the
It is important to recognize that belief functions are not
terminology, the main point should not be lost. Support
being used in this way. Although probability theory can
Theory is likelihood-driven, de?ned relative to a criterion
be expressed mathematically as a special case of belief
of truth. Justi?cation is a distinct criterion and measures
functions (Shafer, 1976), conceptually the two are dis-
of it have distinct characteristics.
tinct. Of interest are subjects’ expressions of justi?cation,
As also noted by Tversky and colleagues, justi?cation-
not of likelihood. These are accepted as separate con-
based weights, in contrast to likelihood assessments, have
structs. Belief functions are not applied as a weaker mea-
been little studied. The current paper, and the research
sure of the same likelihood construct that is captured by
stream within which it ?ts, serves to ?ll this void. To
probability judgments. Belief functions measure a sepa-
operationalize the evidential weight construct, I employ
rate construct with distinguishing features, e.g., noncom-
Dempster-Shafer theory, the best formulated system with
plementarity.
features appropriate for capturing evidential support in
situations emphasizing justi?cation (Shafer, 1976; cf.
1.1.2 Support Theory
Dempster, 1968).
The current study demonstrates an assessment ap-
Support Theory has recently used a construct labeled
proach grounded in Dempster-Shafer theory as a basis
“support” as a building block of subjective probability
for developing hypotheses. Curley and Golden (1994),
(Rottenstreich & Tversky, 1997; Tversky & Koehler,
using similar though cruder methods, found that nearly
1994). This can easily lead to confusion since the term
half of the subjects were able to consistently express be-
has also been used to describe the construct of evidential
liefs that qualitatively matched hypothesized expectations
weight (notably by Shafer, 1976). However, as it has been
based on the evidential content. Even subjects whose re-
operationalized, Support Theory and the tasks to which
sponses did not match the expected pattern showed con-
it is applied are clearly likelihood-driven. When Tver-
sistency in their use of the language, supporting the co-
sky and Koehler directly assess “support” they do so by
herence of people’s use of belief functions. Subjects also
having subjects rate “the [basketball] team you believe is
consistently responded in ways differing from the pre-
strongest” (Study 3) and “the suspiciousness of a given
scriptions of probability theory, ?nding aspects of the be-
suspect” to a hypothetical crime (Study 4). These rat-
lief function language useful in expressing their beliefs.
ings of cues that serve as the basis for subject’s likeli-
Golden (1993/4) followed by examining the reliability
hood judgments, as evidenced in their experiments, are
(using a test-retest procedure) and validity of subjects’
not directly equivalent to the support assessments within
responses in two studies. With improvements in training,
Dempster-Shafer theory that are the subject of this paper.
subjects showed even better qualitative consistency than
These authors explicitly acknowledge this distinction
was observed by Curley and Golden. Numerically, sub-
of justi?cation and likelihood, and appropriately so, for
jects also were able to respond reliably, though with room
example:
for improvement, and the validity was high relative to the
Judgments of strength of evidence, we sug-
reliability. Thus, the studies complemented the analy-
gest, re?ect the degree to which a speci?c body
sis of Briggs and Krantz (1992) in supporting the via-
of evidence con?rms a particular hypothesis,
bility of belief functions from a measurement standpoint.
whereas judgments of probability express the
Dempster-Shafer theory potentially offers a meaningful
relative support for the competing hypotheses
response measure, particularly in a qualitative sense, that
based on the judge’s general knowledge and
is distinct from that of probability theory for use in inves-
prior belief. The two types of judgments, there-
tigating subjects’ degrees of evidential weight.
fore, are expected to follow different rules.
This paper demonstrates a technique using a measure
(Rottenstreich & Tversky, 1997, p. 413)
from Dempster-Shafer theory, applying it to sets of ev-
idence that are systematically constructed to get a fuller
There is a difference in the terminology: These authors
understanding of evidential weight judgments from a psy-

---

Judgment and Decision Making, Vol. 2, No. 5, October 2007
Evidential weight
260
chological standpoint. The next section provides a brief
No Evidential
Evidence 1:
Evidence 2:
overview of Dempster-Shafer theory that serves as the
Support
Target Set {a,d}
Target Set {a,b,c}
study’s theoretical basis. Completing the paper are four
sections describing key features of evidential weight to
be investigated empirically, the methodology, the results,
and a general discussion, respectively.
1.2 Dempster-Shafer theory
Following is a brief description of elements of Dempster-
Shafer theory as it is applied here empirically. The theory
Figure 1: Movement of belief under Dempster’s Rule for
is a system for qualifying one’s beliefs using numerical
two pieces of evidence without con?ict (K=0).
expressions of degrees of support. Shafer (1976) provides
a fuller theoretical treatment for the interested reader.
Shafer described several, inter-related measures, con-
is a non-negative, real-valued function, m, on the power
veying slightly different messages about evidential
set of ? such that:
weight, and the transformation functions connecting
(a) m(?) = 0, where ? is de?ned as the null or impossi-
them. One of these, Bel is termed a belief function and
ble event; and,
is a commonly employed measure from the system. For
(b) ? m(A) = 1, A ? ?.
example, this is the measure used by Briggs and Krantz
The term reserve is borrowed from the idea of a contin-
(1992). Here, a different measure is elicited, the basic
gency reserve in budgeting, in which money is assigned
probability assignment, or what I shall call the reserve
to a category without specifying how to divide it among
function. Both measures capture a degree of belief. The
the subcategories. Assigning m% of one’s belief to a sub-
two measures have a 1–1 correspondence and are mathe-
set in the power set of ? can be interpreted as: “Based
matically inter-transformable, so the selection for assess-
on the evidence, I believe with m% of my belief that the
ment is a matter of experimenter preference. The reserve
hypotheses in this set are supported; however, I cannot
function measure is chosen here as being most concep-
distinguish between the elements in the set individually.”
tually like probabilities. Both probabilities and reserve
Although this interpretation also holds for m = 0, we use
functions can be characterized as dividing the whole of
the term assign belief to signify that a positive number is
one’s belief (1.0) into smaller elements. Consequently,
attached to a set. Also note that this function is connected
the measure is believed to be an intuitive one for individ-
to, but distinct from, the belief function Bel de?ned by
uals to assess. As noted, Briggs and Krantz provide an
Shafer:
empirical example using Bel, instead. Which of the two
Bel(B) = ? m(A), for all A ? B.
measures might be better for assessment is an open em-
pirical question that is not addressed here. I do argue that
the assessments obtained in this study are meaningful and
Singleton: A subset of the power set of ? that contains
informative.
only one element, e.g., {a}.
For brevity of exposition, hereafter belief is used in-
terchangeably with“degree of belief.” Other terminology
Simple support function: A reserve function that as-
from the theory that is used in this paper includes:
signs a positive number to two and only two subsets of
the power set, where one of the subsets is ?. All ev-
Frame of discernment ?: A ?nite set of possible val-
idence used in the study was designed to elicit simple
ues for a variable X, such that one, and only one, element
support. The non-? subset for which the evidence was
of the set is true. These elements are the possible states
designed to elicit positive belief is called the target set of
of nature or hypotheses. In general, the items within the
the evidence.
frame of discernment develop as evidence accumulates,
Multiple evidence requires an assessment of the joint
i.e., one can assign belief to ? without specifying what
impact of the evidence. In a formal theory, like prob-
elements might be contained within it. However, in this
ability theory, this is accomplished by a combination
study for experimental control, the elements in the frame
rule, like Bayes’s Theorem. In Dempster-Shafer theory,
are given to subjects, ? = {a, b, c, d, e, f, g}.
Dempster’s Rule is posited as a basis for assessing the
joint impact of multiple evidence.
Reserve Function: This is the name given by Ray and
Krantz (1996, denoted as r in their paper) to Shafer’s “ba-
Dempster’s Rule: A method for combining two inde-
sic probability assignment,” and it is adopted here. This
pendent functions, m1 and m2, into a new function, m:

---

Judgment and Decision Making, Vol. 2, No. 5, October 2007
Evidential weight
261
(a) Con?ict
the weight from {a, d} to the intersection of the two sets:
Evidence 1:
Evidence 2:
{a}. In this way, as evidence accumulates, support be-
Target set {d}, Strong (.7)
Target set {g}, Moderate (.4)
comes differentiated into ?ner subsets capturing the jus-
ti?cation for the possible evidential conclusions.
1.3 Key Aspects of Evidential Weight
We form beliefs in response to evidence; we assign de-
grees of belief to the extent that the evidence is not de?ni-
tive. The present study demonstrates psychological as-
pects of subjects’ judgments of evidential weight using
systematically created sets of evidence. Key character-
istics for analysis in the assessment of justi?cation are
identi?ed in this section.
(b) Dempster’s Rule
(c) Con?ict-to-? Rule
1.3.1 Inconclusiveness
As noted earlier, an important difference between likeli-
hood and support is the noncomplementarity of eviden-
tial weight. If the evidence does not justify A, this does
not necessarily imply justi?cation for not-A. Instead, the
evidence may be silent with respect to either. For exam-
ple, if evidence of questionable reliability supports A, we
Figure 2: Movement of belief where evidence creates
would qualify the justi?cation it provides for A based on
con?ict (K>0) under (b) Dempster’s Rule and (c) the
the evidence’s unreliability. We would not, however, then
Con?ict-to-? Rule.
transfer the remainder of its justi?cation to not-A. To the
extent the evidence is unreliable, it does not implicate ei-
ther A or not-A.
m(A) = (1 - K)?1
m1(Ai)m2(Aj),
Relatedly, the representation of ignorance has been a
for all Ai ? ?, Aj ? ?
controversial topic in the use of probability theory (e.g.,
where Ai ? Aj = A; and
DeGroot, 1970; Winkler, 1972). Having a natural means
K =
m
of expressing ignorance, by assigning belief to the super-
1(Ai)m2(Aj),
for all A
set ? may prove to be an attractive, intuitive feature of
i ? Aj = ?.
belief functions. The degree of belief m(?) represents
The parameter K is a measure of con?ict in the evidence.
one’s undifferentiated belief that is withheld in complete
The idea behind the combination rule is that initially
reserve, expressing nonsupport for any subset of possibil-
your belief is undifferentiated and allocated to ?. As ev-
ities, e.g., due to evidential unreliability.
idence becomes available, you partition your belief into
If individuals view inconclusiveness as a meaningful
smaller subsets. This idea is illustrated by Figure 1 for
aspect of support, then a sensible and persistent use of
two pieces of evidence. Although shown successively,
m(?) should be observed. First, since the evidence in
Dempster’s Rule is commutative, the order of evidence is
the study is inconclusive, subjects should generally as-
irrelevant. (See Golden, 1993/4, for evidence of commu-
sign belief to ? to communicate this. Second, as evidence
tativity in subjects’ assessments of evidential weight and
accumulates and becomes more conclusive, m(?) should
of a discussion of other properties of Dempster’s Rule and
decrease. Not all theories of support have this latter prop-
other proposed combination rules.)
erty. For example, Dubois and Prade (1986) proposed an
Initially, there is no evidence and all support (1.0) is in
averaging rule in which the m(?) of combined evidence
the undifferentiated set ?. As shown, the ?rst piece of
is proposed to be intermediate to the m(?) of the individ-
evidence implicates a and d, not differentiating between
ual pieces of evidence. This rule corresponds to a leveling
them. The function m1 moves a portion of the weight of
of support rather than a focusing of support as the under-
evidence into the set {a, d} to convey this, leaving the re-
lying psychology in accumulating support.
mainder of the weight in the set ?. How much weight is
moved depends on the reliability, credibility and strength
1.3.2 Con?ict
of the evidence. The second piece of evidence implicates
a, b and c. The function m2 moves a portion of the weight
Another key issue in understanding justi?cation is the re-
from ? into {a, b, c} and moves the same proportion of
action to evidential con?ict. Consider Figure 2, showing

---

Judgment and Decision Making, Vol. 2, No. 5, October 2007
Evidential weight
262
an example of support with con?icting evidence. Evi-
sis to investigate how individual subjects react to con?ict.
dence 1 provides simple support for {d}. It is fairly strong
evidence, though inconclusive with m1(d) = .70.1 Evi-
1.3.3 Simpli?cation
dence 2 is of more moderate strength and implicates {g}
with m
Evaluating evidence becomes increasingly complex as
2(g) = .40. Clearly, the evidence is con?icting, im-
plicating disjoint sets of possibilities. Applying Demp-
evidence accumulates, even more so in assessing justi-
ster’s Rule for independent pieces of evidence, the result-
?cation than likelihood. For belief functions the possi-
ing weight of con?ict is K = .28 (Figure 2a).
ble number of assessments increases exponentially with
Two general possible reactions are that one can re-
the number of distinct alternatives. For example, if ?
act to con?ict with continued con?dence in the evidence
contains seven separable alternatives, then no more than
or by pulling back support. Formal rules corresponding
seven probability assessments are needed, but as many as
to each of these psychological reactions have been pro-
(27 - 1) = 127 positive reserve numbers (values of m) may
posed. The rules receiving best support in a preliminary
be applied. Thus, the number of values can quickly ex-
study (Golden 1993/4) are highlighted here, and represent
ceed the capacity of an individual to maintain information
these two divergent psychological approaches to con?ict.
in working memory (Miller’s, 1956, 7 ± 2).
The rules are not claimed as descriptive in the sense that
Given the limitation, subjects likely will simplify their
individuals are presumed to perform the calculations of
reserve functions with the accumulation of evidence of
the combination rules. However, as capturing different
differing implications. However, subjects should do so
approaches to con?ict, the rules provide useful standards
in a reasoned, not haphazard, manner, maintaining main
of comparison for contrasting the underlying psycholog-
lines of implication while truncating others. Of interest is
ical theories.
this purposiveness as it exists: What strategies do individ-
Dempster’s Rule exempli?es a rule capturing contin-
uals employ to simplify the lines of justi?cation without
ued con?dence in the evidence by distributing con?ict
sacri?cing important information?
proportionally into already implicated sets (Figure 2b).
In sum, the study demonstrates the use of belief func-
Following this normalization, m(?) = .25, less than both
tions, using the reserve function form, for communicating
m
evidential weight, while addressing three psychological
1(?) = .30 and m2(?) = .60. It is easily shown that
these strict inequalities will hold whenever the evidence
concerns:
is inconclusive, mi(?) > 0 for i = 1, 2. The attitude is
• How do subjects communicate inconclusiveness?
one of: “I know there is con?ict, but my beliefs are still
sound, just not focused yet.”
• How do subjects react to con?ict, particularly do
Since Dempster’s Rule does embody implicit claims
they tend to show continued con?dence in the evi-
about how people assign evidential weight, other re-
dence or pull back support?
searchers have questioned these claims and proposed
alternative combination schemes. One alternative is a
• How do subjects simplify complex evidential
Con?ict-to-? Rule (Yager, 1987). The rule operates like
weight?
Dempster’s Rule, except when there is con?ict, K > 0.
Instead of normalizing, the rule assigns all of K to ?, as
The study addresses these questions using an established
shown by Figure 2c. Thus, the rule captures indecisive-
task and systematically varied sets of evidence. The more
ness as the psychological reaction to con?ict. The atti-
extensive evidence sets also allow individual-level analy-
tude is one of: “The con?ict indicates that I do not know
ses, affording the possibility of identifying individual dif-
what is happening. It re?ects indeterminacy and my ig-
ferences in behavior with respect to these questions.
norance. Thus, I should pull my belief back into ?.” For
this rule, in the example, the combined m(?) = .46 after
2 Method
adjustment. In this case, the value is greater than m1(?)
= .30. Although, this does not necessarily happen, we see
2.1 Subjects
here that the rule allows the possibility of greater indeci-
siveness with increasing evidence, in marked contrast to
Sixty-six non-law graduate students at the University of
the attitude embodied in Dempster’s Rule.
Minnesota voluntarily participated in the study. The sub-
Golden (1993/4) reported evidence that subjects in ag-
jects engaged in a juror-type task, evaluating evidence in
gregate behaved midway between Dempster’s Rule and
a hypothetical legal setting and requiring no special law
the Con?ict-to-? Rule, with no support for other tested
experience. They were paid a ?xed fee for participating
rules. The present study allows an individual-level analy-
in a single session lasting less than two hours. Each sub-
ject was in the session individually with a single experi-
1For simplicity the notation m({d}) is shortened to m(d).
menter.

---

Judgment and Decision Making, Vol. 2, No. 5, October 2007
Evidential weight
263
Table 1: Hypothetical situation to which subjects responded
Bensten Murder Case
Your Task: You have been asked to help the county attorney assess evidence gathered by police in a murder case.
The county attorney would like you to evaluate the evidence and state how you believe the evidence implicates
the seven suspects. The county attorney may or may not have more evidence, but at this time the county attorney
is only interested in examining the effects of the following pieces of evidence. The county attorney asks that
the evaluation be done for the pieces of evidence individually, as well as collectively, because the county attor-
ney is unsure which suspect will be charged and which evidence will be used in court. Your analysis will be
used to guide the on-going police investigation and to help the county attorney in the pre-trial preparation of a case.
At this time the police are sure of a couple of things: 1) the murderer acted alone, and 2) the list of suspects is complete.
The Crime: On Monday, the 20th of April, Thomas Bensten was found murdered in his 3rd ?oor Edina of?ce suite. Mr.
Bensten is a 45 year old single executive for a company named PSV Enterprises. Mr. Bensten’s body was discovered
at approximately 6 a.m. by one of the building’s janitors. The janitor was unlocking the building’s doors as part of
his job. The police arrived shortly after 7 a.m. and concluded that Mr. Bensten had been murdered with a 44 caliber
handgun. The gun had been shot into Mr. Bensten’s left shoulder at close range after what appears to have been a
signi?cant struggle. The time of death was set between 7 p.m. Friday, April 17th and 11 a.m. Saturday, April 18th.
2.2 Procedure
Table 2: Sample response area.
The experimental procedure began with a training session
Evidence #1
that provided subjects with instruction in the language of
Sets
Strength
reserve functions. The Appendix contains the full train-
ing materials. These materials were similar to those em-
ployed by Golden (1993/4); and, the training case was
similar to the experimental case. It involved the same
task as described in Table 1, but for a different crime de-
scription — an auto theft. Aside from familiarizing the
subjects with the task, the training instructed subjects in
the vocabulary of the belief function language. That is,
given a belief, how could a subject express this belief in
the theory’s language? And conversely, given a reserve
function, what does it communicate? The order and con-
Total
tent of training involved instructions about:
(must add to
1)
• The task (Table 1)
• What it means to assign belief to a set of suspects,
e.g., selecting the set {B, D} “represents your be-
• Practice case, Part 2: The subject responded to four
lief that either Suspect B or Suspect D is guilty, but
pairs of evidence for the auto theft case.
based on the evidence, you cannot differentiate this
belief between the two suspects.”
The subjects were not schooled in any particular form
of reserve function, and were informed that there was
• The response form (Table 2), demonstrated for sin-
no right or wrong belief given the evidence. They also
gle pieces of evidence
were not instructed in any particular way of combining
evidence when responding to more than one piece of
• Seven examples pairing text descriptions of beliefs
evidence. Golden (1993/4) used a post-training quiz to
with the belief functions that communicate these de-
check subjects’ understanding after training. All subjects
scriptions.
(total N = 64) achieved suf?cient mastery, so the quiz was
not employed in the current study.
• Practice case, Part 1: The subject responded to four
Following the training, subjects read a page describ-
individual pieces of evidence for an auto theft case
ing the hypothetical situation to which the evidence re-
similar to the upcoming experimental case.
lated and the role that they were to take in responding

---

Judgment and Decision Making, Vol. 2, No. 5, October 2007
Evidential weight
264
to the evidence (Table 1). Each subject then responded
in succession to 18 single pieces of evidence, 4 evidence
(a) Identical
(b) Embedded
pairs and 17 evidence triples. The pairs and triples were
'$
#
'$

constructed using items from the 18 single pieces of evi-
dence. The stimuli are described below.
d
a
d

Subjects ?rst received a stimulus booklet containing
&%
"!
&%
the 18 single pieces of evidence, each on a separate page.
They responded to each piece of evidence in turn and in
(c) Intersecting
(d) Disjoint
isolation, separately from all preceding evidence. Each
'$
'$ '$
'$
single evidence was numbered consecutively, and the re-
sponses were recorded on a separate response booklet. In
e
a
f
d
g
providing their responses, subjects were advised during
&%
&% &%
&%
the training to ?rst identify which sets should be assigned
belief and then to assign the numerical beliefs to these
Figure 3: Possible structures for pairs of evidence.
sets. Thus, the qualitative assessment of identifying the
implicated sets preceded and was separate from the quan-
titative assessment.
the employees had inadequate quality of work.
After completing the ?rst stimulus booklet, the sub-
The ?rings cost each of the men approximately
jects received a second stimulus booklet with the four ev-
$40,000.
idence pairs (which appeared ?rst) and the 17 evidence
Suspects
Recently Fired by Mr. Bensten
triples . For the pairs and triples, stimuli and response
forms were in the same booklet. Evidence used in the
Suspect A
Yes
stimuli were numbered to match the numbering used in
Suspect B
Yes
the single-evidence booklet. Subjects could refer back to
Suspect C
No
their single-evidence response booklet to check their re-
sponses while going through the pairs and triples booklet.
Suspect D
No
This capability was described during the training.
Suspect E
Yes
After completing the second booklet, subjects were de-
Suspect F
No
briefed and paid.
Suspect G
No
2.3 Stimuli
Each subject saw each of the 18 pieces of evidence in
Table 3 and saw evidence implicating 18 target sets. The
Subjects responded to single pieces, pairs, and triples of
pairing of target sets to evidence content was randomized,
evidence. The structure by which the evidence sets were
and the order of evidence presentation was also random-
constructed is now described.
ized.
The single pieces of evidence were selectively com-
2.3.1 Single evidence
bined into sets of evidence containing two or three pieces
of evidence.
The materials were adapted from those tested by Curley
and Golden (1994) and Golden (1993/4). Table 1 de-
scribes the experimental situation and the subjects’ as-
2.3.2 Evidence Pairs
signed role. Subjects saw 18 single pieces of evidence.
For two pieces of evidence, there are four possible com-
Table 3 contains brief descriptions of the content of the 18
bination forms associated with target sets, disregarding
pieces of evidence. Subjects saw paragraph descriptions
the order of evidence and speci?c evidence content. The
of each. For each piece of evidence, subjects received in-
target sets for the two pieces of evidence may be either
formation for each of the seven possible suspects. They
identical, embedded, intersecting, or disjoint. Examples
responded using a response table like that in Table 2. The
of each are given and illustrated in Figure 3:
information was constructed to provide simple support,
implicating a single target set of suspects. For example,
(a) Identical: {d} {d}
the following piece of evidence (Fired from Job) provides
(b) Embedded: {a, d} {d}
support for target set {a, b, e}:
(c) Intersecting: {a, e} {a, f}
(d) Disjoint: {d} {g}
Some of the suspects had been recently ?red
The particular target sets were also selected so that
from PSV Enterprises by Mr. Bensten. The rea-
each of the regions in Figure 3 contains a single suspect.
son Mr. Bensten gave for the ?rings was that
Each subject responded to one of each of these structures,

---

Judgment and Decision Making, Vol. 2, No. 5, October 2007
Evidential weight
265
Table 3: Brief descriptions of the contents of the individual pieces of evidence used in the study along with the mean
(standard deviation) belief attached to the target set for that evidence.
Motive Evidence
.54 (.25) Suspect being Blackmailed by victim
.41 (.25) Suspect recently Fired from Job by victim
.38 (.26) Suspect felt Cheated in a Business Venture with victim
.35 (.26) Suspect in victim’s Will
.30 (.22) Recent Argument with victim
.30 (.23) Violent Personality indicated by psychological testing
Opportunity Evidence
.52 (.29) Suspect had Pass Key to the building
.52 (.32) No Alibi from another for time of crime
.40 (.31) Suspect with previously registered .44 caliber Handgun (gun unavailable)
Physical Evidence
.83 (.19) Blood type match
.64 (.28) Fingerprints (partial); possible match with suspect
.61 (.28) Left-Handed Stab Wound; left-handed suspect
.55 (.30) Foot Print (partial); possible match with suspect
.46 (.33) Aspirin Bottle at scene; suspect not under doctor’s orders to avoid aspirin
.43 (.30) Cigarette Ashes; suspect smoked cigarettes
.34 (.29) Nonprescription Sunglasses at scene; suspect does not have either prescription or contact lenses
.24 (.27) Glasses of untouched Scotch at scene; suspect drinks alcohol
.21 (.23) Valuable Baseball missing (otherwise no valuables taken); suspect is baseball fan
Note: N = 66 subjects for each mean and standard deviation.
involving eight separate single pieces of evidence from
triple structures was assessed by 33 subjects.3 (The use of
among those seen earlier.2 Recall that the content of the
66 subjects was motivated by this randomization design.)
evidence items was randomly varied across subjects. Pre-
Examples of four of the evidence triple structures are
sentation order of the pair structures was also random-
given and illustrated in Figure 4. Again, note that the par-
ized.
ticular target sets were selected so that each of the regions
in Figure 4 (and for all other triple structures) contains a
2.3.3 Evidence Triples
single suspect.
For three pieces of evidence, there are 34 possible com-
bination forms, disregarding evidence order and speci?c
3 Results
content. Testing all of these combinations requires 18
separate pieces of evidence (which led to the selection
3.1 Validity
of the 18 used as single pieces of evidence). Again, ev-
idence content and presentation order were randomized.
All single pieces of evidence seen by subjects were de-
Also, to stay within a reasonable time frame for a single
signed to provide simple support. The degree to which
session, each subject saw 17 evidence triples. Selection
subjects perceived the evidence in terms of simple sup-
of 17 of 34 triples was randomized so that each of the 34
port can be used as a qualitative measure of structural
2One subject mistakenly received one wrong pair structure. When
3Three subjects mistakenly received a wrong triple structure. When
analyzing the responses by pair structure, this response is excluded from
analyzing the responses by triple structure, these three responses are
the analysis.
excluded from the analysis.

---

Judgment and Decision Making, Vol. 2, No. 5, October 2007
Evidential weight
266
Figure 4. Sample of (4 of 34) possible structures for triples of evidence.
(a)
(b)
functions assigned belief to ?. For the evidence triples,
77% of the responses assigned positive belief to ?. Over-
a
all, mean m(?) declined as evidence accumulated from
b d
d
.53 for singles (n = 1188, s = .32) to .32 for pairs (n = 264,
e
s = .27), and .20 for triples (n = 1122, s = .23). Subjects
commonly used this ability to communicate that the evi-
dence as a whole was not conclusive. Inconclusiveness is
(c)
(d)
an evidential weight concept, not available in probability
assessment, that was meaningful to the subjects.
a
As a cleaner view of this tendency, consider responses
d
e
g
d
to accumulating evidence in favor of a single target ele-
e
f
ment: for a single piece of evidence, the evidence pair
in Figure 3a, and the evidence triple in Figure 4b. For
these responses, the mean m(?) declined with accumu-
Figure 4: Sample of (4 of 34) possible structures for
lating evidence from 0.53 to 0.31 to 0.21. Individually,
triples of evidence.
of the 33 subjects responding to the triple in Figure 4b,
22 (67%) decreased m(?) from single to pair and from
validity, i.e., a meaningfulness measure which does not
pair to triple. This tendency is consistent with the claim
rely on the numerical responses but simply on the sets
of theories like Dempster’s Rule and the Con?ict-to-?
that are assigned belief — the qualitative structure of the
Rule. In contrast, only 1/33 (3%) responded consistently
responses. Overall, 95% (1123/1188, 18 pieces of evi-
with an averaging approach both to the evidence pair and
dence × 66 subjects) of responses to single pieces of evi-
to the evidence triple, giving a cumulative response that
dence were structurally valid, assigning belief only to the
was intermediate to the component single evidence re-
target set and/or ?. This exceeds the 59% found by Cur-
sponses in both cases.
ley and Golden (1994) and is comparable to the 92% and
96% found in the two experiments reported by Golden
(1993/4). From a methodological perspective, the com-
3.3 Con?ict
parisons indicate: (a) the improvement in training mate-
In considering methods of combining the support pro-
rials after the ?rst empirical effort by Curley and Golden,
vided by multiple evidence, we distinguish situations that
and (b) the success in streamlining the training materi-
involve con?ict from those that do not. One pair of evi-
als after Golden’s study without loss of meaning. Un-
dence (66 responses) and fourteen triples (458 responses
like Curley and Golden, there were no consistent non-
= 14 × 33 subjects/triple, with 4 missing4) involved struc-
structurally valid response patterns. In particular, sub-
tural con?ict. To compare subjects’ responses, focus is on
jects did not respond with consistent use of the comple-
the weight attached to ?. It is in the assignment to ? that
ment of the target set or consistent use of non-target sin-
the rules most markedly differ in a way that is informative
gletons.
of how the respondents reacted to that con?ict (Figure 2).
Thus, subjects saw evidence as effecting a movement
of support into the implicated set. The subjects did not
Overall, in response to con?icting evidence, subject’s
overextend this support into smaller subsets than was
mean m(?) = .27. From applying Dempster’s Rule to
warranted by the evidence or into sets that were not di-
the responses for the single pieces of evidence, the ex-
rectly implicated. They also did not reply in a way mim-
pected mean m(?) = .25. From applying the Con?ict-to-
icking probability assessment (though such a function
? Rule to the responses for the single pieces of evidence,
was seen during training, see Example #5 in the Ap-
the expected mean m(?) = .44. These means suggest a
pendix).
closer correspondence of aggregate behavior with Demp-
Table 3 shows the mean beliefs assigned to the target
ster’s Rule. Of the 524 total m(?) responses, 297 were
set for the content of each of the individual pieces of ev-
closer to the value predicted by Dempster’s Rule whereas
idence. As to numerical validity, the orderings of the
121 were closer to the value predicted by the Con?ict-
means are reasonable relative to the content of the evi-
to-? Rule (the remainder were equidistant). Thus, the
dence. There are no obvious misalignments in the data.
single best descriptive model in the aggregate is provided
by Dempster’s Rule.
3.2 Inconclusiveness
4One response was missing due to an error in the administration of
the study as described by Footnote 3. Three responses were missing
There was a high use of ? across all subjects. For single
because the responses to individual evidence led to K = 1, making ap-
pieces of evidence and evidence pairs, 91% of the reserve
plication of Dempster’s Rule impossible.

---