Investigating intuitive and deliberate processes statistically: The multiple-measure maximum likelihood strategy classification method

Judgment and Decision Making, Vol. 4, No. 3, April 2009, pp. 186–199
Investigating intuitive and deliberate processes statistically: The
multiple-measure maximum likelihood strategy classi?cation
method
Andreas Glöckner?
Max Planck Institute for Research on Collective Goods
Abstract
One of the core challenges of decision research is to identify individuals’ decision strategies without in?uencing
decision behavior by the method used. Bröder and Schiffer (2003) suggested a method to classify decision strategies
based on a maximum likelihood estimation, comparing the probability of individuals’ choices given the application of
a certain strategy and a constant error rate. Although this method was shown to be unbiased and practically useful, it
obviously does not allow differentiating between models that make the same predictions concerning choices but different
predictions for the underlying process, which is often the case when comparing complex to simple models or when
comparing intuitive and deliberate strategies. An extended method is suggested that additionally includes decision times
and con?dence judgments in a simultaneous Multiple-Measure Maximum Likelihood estimation. In simulations, it is
shown that the method is unbiased and sensitive to differentiate between strategies if the effects on times and con?dence
are suf?ciently large.
Keywords: strategy classi?cation, judgment, decision making, maximum likelihood estimation, intuition.
1 Methods for strategy classi?ca- 1.1 Structural modeling
tion
Structural modeling uses a multiple regression approach
to identify how cues or attributes are utilized in making
In different situations, people might use different strate-
judgments (Brehmer, 1994; Doherty & Brehmer, 1997;
gies to decide. These strategies might sometimes be com-
Doherty & Kurz, 1996). Speci?cally, a set of judgments
pletely based on conscious processes, such as compar-
(criterion) is predicted by cue values (predictors). Re-
ing the available options on the most important attribute
gression weights can be interpreted as indicators for in-
and choosing the option that is better on this attribute
dividuals’ usage of cues in their judgments. Structural
(e.g., Beach & Mitchell, 1978; Fishburn, 1974; Payne,
modeling usually does not aim to analyze processes (as in
Bettman, & Johnson, 1988), or people might rely more
the paramorphic approach; see Hoffman, 1960) but input-
or less on automatic processes that integrate informa-
output relations between cues and judgments only (i.e.,
tion unconsciously (e.g., Busemeyer & Townsend, 1993;
as-if models; but see Bröder, 2000, Exp. 1). Although the
Dougherty, Gettys, & Ogden, 1999; Glöckner & Betsch,
method was tremendously useful in showing that people
2008b). Decision researchers often are interested in the
integrate information in a weighted compensatory man-
question which strategy was (more likely) used by each
ner when making more or less intuitive judgments (Do-
person. Several methods have been suggested to identify
herty & Brehmer, 1997; Hammond, Hamm, Grassia, &
decision strategies. The three predominant approaches
Pearson, 1987), its focus on outcomes limits its applica-
are structural modeling, process tracing, and compara-
bility for differentiating among process models.
tive model ?tting (for overviews see Bröder & Schiffer,
2003a; Glöckner & Witteman, in press; Harte & Koele,
2001).
1.2 Process tracing
?I am grateful to Joseph G. Johnson, Christoph Engel, Arndt Bröder,
Jonathan Baron, Benjamin Hilbig and Nina Horstmann for insightful
Process tracing methods record and analyze parameters
comments on earlier manuscript drafts. I thank Philipp Weinschenk
of information search before judgments or decisions and
and Andreas Nicklisch for their help with the math and the equations.
aim to infer decision strategies from the amount, distri-
Parts of this article were realized during a working stay organized
bution and order of information search. For instance,
by Edoardo Leva. Address: Andreas Glöckner, Max Planck Institute
for Research on Collective Goods, Kurt Schumacher Str. 10, D-53113
information boards are often used in which information
Bonn,Germany. Email: gloeckner@coll.mpg.de.
is provided behind hidden information cards, which are
186

---

Judgment and Decision Making, Vol. 4, No. 3, April 2009
Multiple-measure strategy classi?cation
187
opened on request or by mouse-click (e.g., Payne et al.,
we would classify this person as consistent with EQW.
1988; Rieskamp & Hoffrage, 1999).
However, this simple counting method leads to biased re-
This method allows differentiating between decision
sults if strategies predict random choices for some of the
strategies because some of them differ in their predictions
tasks (Bröder & Schiffer, 2003a; Bröder, in press).
concerning information search. A simple take-the-best
Maximum likelihood estimation provides a more ele-
strategy (TTB, Gigerenzer & Goldstein, 1996; cf. Fish-
gant means of performing strategy classi?cation that is
burn, 1974), for example, assumes that persons ?rst look
not prone to this source of bias. The basic idea is sim-
up the predictions of the most predictive (valid) cue for
ple: comparative model ?tting determines the strategy
all options. The option with the best cue value is selected.
that would most likely have produced the observed choice
If options are tied, the second most valid cue is consid-
pattern under the assumption of a constant error rate in
ered, and so on. In contrast, according to an equal weight
applying the strategy. For the example above, the best es-
strategy (EQW, Payne et al., 1988), individuals look up
timates for the error rates in strategy application would
all cue information for the ?rst option and sum them up.
be .40 (i.e., 4/10 “errors” in applying a TTB strategy)
Then they do the same for the second option and so on
and .20, respectively. According to the binomial equation
and select the option with the highest sum of cue val-
(see also equation 1, below), the likelihood that exactly
ues. Hence, a cue-wise information search (i.e., search
the observed number of strategy conforming choices (6
along cues) and a strong focus on the most valid cue are
out of 10 correct under the assumption of an error rate of
used as indicators for the usage of TTB (or similar non-
.40) was produced by TTB is .25 whereas the respective
compensatory strategies) and an option-wise information
likelihood for EQW is .30. Hence, it is more likely that
search and an equal inspection of all cues indicate the us-
choices were produced by application of EQW than by
age of EQW (or other compensatory strategies).
TTB.
Despite being highly useful for investigating deliber-
In contrast to classic process-tracing methods, the
ate strategies, standard process-tracing methods such as
comparative model ?tting approach avoids in?uencing
Mouselab (Payne et al., 1988) in?uence strategy selec-
decision behavior by the measuring method and never-
tion and hinder people from applying intuitive-automatic
theless allows process models to be tested. However, the
decision strategies (Glöckner & Betsch, 2008c; see also
method is applicable only when strategies make different
Norman & Schulte-Mecklenbeck, in press). One rea-
choice predictions. If strategies make the same predic-
son for this is that classic process tracing methods in-
tions for choices, the likelihoods for the strategies will
duce a serial information search and prevent quick com-
obviously always be equal.
parisons between options and the formation of holistic
Unfortunately, strategies often make exactly the same
impressions. One might argue that intuitive-automatic
choice predictions. This is due to the fact that the prin-
processes cannot be captured by the analysis of informa-
cipally investigated decision strategies (e.g., TTB and
tion search at all. This conclusion, however, seems to
EQW) are special cases of a weighted additive strategy
be too strong considering the successful use of less in-
(WADD). According to a WADD strategy, for each op-
trusive methods such as eye-tracking technology to in-
tion the cue information is weighted by its importance
vestigate intuitive processes (e.g., Glöckner & Herbold,
(or validity) and added up. The option with the highest
2008). Eye-tracking methods even provide further depen-
weighted sum is chosen (Payne et al., 1988). Although
dent measures (e.g., eye-?xation duration and physiolog-
this strategy sounds quite different from TTB and EQW,
ical arousal; Horstmann, Ahlgrimm, & Glöckner, under
it can be easily shown that WADD predicts the same
review), which could be included in strategy classi?ca-
choices as TTB. This is always the case if the validity of
tion.
each cue is higher than the sum of the validity of all less
valid cues (Bröder, 2000; Lee & Cummins, 2004). Sim-
1.3 Comparative model ?tting
ilarly, WADD predicts the same choices as EQW if the
validity of all cues is similar or equal. Hence, in a strict
The more recently-developed comparative-model-?tting
sense, classi?cation for EQW and TTB based on choices
approach uses a maximum likelihood method to compare
only never rules out that a more complex WADD strat-
choices with the predictions of a set of decision strategies
egy was used. A person could have used WADD with
(Bröder & Schiffer, 2003a; Bröder, in press; Wasserman,
a speci?c cue weighting scheme instead. Therefore, in
2000). For instance, assume one observes choices in 10
all studies relying on the choice-based strategy classi?-
decisions from which 6 are in line with the predictions of
cation method (e.g., Bröder & Schiffer, 2003b; Bröder &
TTB and 8 are in line with the prediction of EQW. An ob-
Gaissmaier, 2007) the estimated proportions of TTB and
vious scheme would be to classify persons according to
EQW users are upper limits for the usage of these sim-
the amount of strategy compatible choices — with 8/10
ple strategies whereas the usage of WADD is likely to be
choices in line with EQW, and only 6/10 in line with TTB,
underestimated.

---

Judgment and Decision Making, Vol. 4, No. 3, April 2009
Multiple-measure strategy classi?cation
188
The problem of similar choice predictions becomes
each item type was presented 10 times (in individually
even more severe if one considers that people might
randomized order), and choices, decision times and con-
also use intuitive decision strategies. Intuitive decision
?dence were recorded.
strategies often also predict choices that follow weighted
additive information integration without the assumption
that individuals calculate weighted sums (Busemeyer &
Johnson, 2004; Glöckner & Betsch, 2008b; Hammond,
2.1 Speci?cation of WADD
Hamm, Grassia, & Pearson, 1987). Therefore, based on
the analysis of choices only, they cannot be distinguished
In order to derive predictions from WADD, it has to
from WADD nor, in a strict sense, from TTB and EQW
be determined how cue validities are used in calculat-
either.
ing weighted sums. In this paper it is assumed that per-
In this article I aim to show that the problem can be
sons correct their weights for the fact that binary cues
solved in that the method for comparative model ?tting
with a validity of .50 have no predictive power (w = v
based on maximum likelihood estimation of choices sug-
- .50; cf. Glöckner & Betsch, 2008c). Although some-
gested by Bröder and Schiffer (2003a) is extended by in-
times stated otherwise, choice predictions (as well as time
cluding additional dependent measures such as decision
and con?dence predictions) of WADD are not invariant
time and con?dence (for earlier approaches, see Berg-
to this transformation. In the following I use the label
ert & Nosofsky, 2007; Glöckner, 2006). A Multiple-
WADDcorrected when referring to the predictions of such a
Measure Maximum Likelihood (MM-ML) strategy clas-
WADD strategy with corrected weights.
si?cation method is suggested that allows identifying de-
cision strategies even if they make the same choice pre-
dictions and different predictions concerning only one of
the other dependent variables (i.e., decision time, con?-
2.2 Decision Time Predictions
dence). Further advantages of the inclusion of additional
dependent measures will be discussed.
Decision-time predictions for the deliberate strategies
TTB, EQW, and WADD are determined according to the
2 Examples for strategy classi?ca- number of elementary information processes necessary to
apply the strategy (Payne et al., 1988). For instance, for
tion
item types 1 to 5 (see Table 1), according to TTB, only
one cue has to be considered, whereas for item 6, two
To apply a strategy classi?cation method, it is necessary
cues have to be considered, which necessitates applying
to select a set of strategies that allows for deriving predic-
more elementary information processes. For statistical
tions concerning choices, decision time, and con?dence.
reasons, decision time predictions are transformed to con-
Furthermore, item types have to be selected for which
trast weights which add up to zero and have a range of 1.
the strategies make different predictions on as many de-
For WADD and EQW, no differences in decision times
pendent variables as possible. These types have to be
are predicted and all contrast weights are set to zero. For
repeatedly presented (e.g., 10 times; Bröder & Schiffer,
PCS decision time predictions were derived from a sim-
2003a). In this analysis I focus on choices in proba-
ulation of the underlying network model (i.e., based on
bilistic inference tasks, in which persons select the better
the iteration the PCS algorithm needs to ?nd a consistent
of two goods based on recommendations of four advi-
solution; Glöckner & Betsch, 2008b).
sors (cues) with different reliability of recommendations
(i.e., cue validity). The considered strategies are WADD,
TTB, EQW, a random choice strategy (RAND), and an
intuitive parallel constraint satisfaction strategy (PCS;
2.3 Con?dence Predictions
Glöckner & Betsch, 2008b). The choice predictions of
PCS and WADD are essentially equal (considering differ-
Con?dence predictions of TTB were derived based on the
ent cue-validity transformation functions) and hence the
validity of the differentiating cue (Gigerenzer, Hoffrage,
strategies cannot be differentiated based on choices only.
& Kleinbölting, 1991). For WADD and EQW the dif-
The steps to derive the strategies’ predictions concerning
ference between the weighted (unweighted) sums of cue
choices, decision times and con?dence are explained in
values for the two options was calculated and used as pre-
detail elsewhere (Glöckner, in press) and the most im-
diction for con?dence. For PCS the predictions were de-
portant aspects will be summarized below. The resulting
rived from model simulations (i.e., based on the differ-
predictions for six types of items are shown in Table 1.
ence between the activation of the options after the con-
It is assumed that an experiment was conducted in which
sistent solution was found).

---

Judgment and Decision Making, Vol. 4, No. 3, April 2009
Multiple-measure strategy classi?cation
189
Table 1: Item types and predictions of strategies.
Types of decision tasks
1
2
3
4
5
6
A B
A B
A B
A B
A B
A B
Cue 1 (v = .80)
+ –
+ –
+ –
+ –
+ –
– –
Cue 2 (v = .70)
+ –
+ –
– +
– –
– +
– –
Cue 3 (v = .60)
+ –
– +
– +
– –
+ –
+ –
Cue 4 (v = .55)
– +
– +
– +
– +
– +
– +
Choice Predictions
TTB
A
A
A
A
A
A
EQW
A
A:B
B
A:B
A:B
A:B
WADDcorrected
A
A
B
A
A
A
PCS
A
A
B
A
A
A
Time Predictions (contrasts)
TTB
?0.167
?0.167
?0.167
?0.167
?0.167
0.833
EQW
0
0
0
0
0
0
WADDcorrected
0
0
0
0
0
0
PCS
?0.400
?0.310
0.600
?0.120
0.110
0.130
Con?dence Predictions (contrasts)
TTB
0.167
0.167
0.167
0.167
0.167
?0.833
EQW
0.667
?0.330
0.667
?0.330
?0.330
?0.330
WADDcorrected
0.630
0.230
?0.370
0.030
?0.170
?0.370
PCS
0.620
0.280
?0.320
?0.010
?0.190
?0.380
Note. Items types and predictions of decision strategies. In the upper part of the table, the item types are
presented. The cue validities v are provided beside each cue. Below the predictions concerning choices are
shown. A and B stand for the predicted option. “A:B” indicates random choices between A and B. The lower
part of the table shows predictions for decision times and con?dences expressed in contrast weights that add
up to zero and have a range of 1. Contrast values represent relative weights comparing different cue patterns
for one strategy.
3 The maximum likelihood strategy type j that are presented and let njk be the number of cor-
classi?cation method for choices
rect predictions of strategy k. The likelihood of observing
a certain number of correct predictions njk given a con-
stant error rate follows a binomial distribution. Hence,
The maximum likelihood strategy classi?cation method
the likelihood of observing a set of choices given a strat-
for choices calculates the conditional likelihood of an ob-
egy k and a constant error rate ?k can be calculated by:
served set of choices for different types of tasks j given
the application of a certain decision strategy k and a con-
stant error rate ?
J
k. The likelihood values of the different
nj
(nj?njk)
strategies are compared and individuals are classi?ed as
Lk(C) = p(njk|k, k) =
(1?
.
n
k)njk k
users of the strategy that most likely produced the ob-
j=1
jk
served choices. For each of the choices and each strategy,
(1)
it is determined if the choice was in line with the predic-
The single free parameter ?k can be estimated using stan-
tion of the strategy or not. Let nj be the number tasks of
dard statistical software packages such as STATA or, in

---

Judgment and Decision Making, Vol. 4, No. 3, April 2009
Multiple-measure strategy classi?cation
190
this simple case, by:
single maximum likelihood measure for choices, decision
?
?
?
?
times and con?dence.
J
J
ˆ
?
?
?
?
k =
(nj ? njk) ÷
nj .
(2)
j=1
j=1
4 Multiple-Measure
Maximum
Individuals are classi?ed as users of the strategy with the
highest likelihood value Lk(C). If a strategy does not dif-
Likelihood (MM-ML) strategy
ferentiate between options for a speci?c type of items,
classi?cation
individuals are assumed to guess and ?k is assumed to be
.5 for this type. Bröder and Schiffer (2003a) showed in
simulations that up to an error rate of 25% the method
Maximum likelihood estimation is, of course, not limited
differentiates well between strategies which make dif-
to dichotomous outcomes (i.e., choices) but can also be
ferent predictions concerning choices (i.e., classi?cation
applied to continuous variables such as decision times.
error below 20%).1 In decision research, the method
However, estimation of the likelihood of a set of obser-
has been successfully applied to judgments and choices
vations necessitates assumptions about the distribution
based on probabilistic inference (e.g., Bröder, 2003; for
underlying the data generation process for the variable.
an overview see Bröder, in press; Bröder & Gaissmaier,
One standard assumption is that log-transformed deci-
2007; Bröder & Schiffer, 2003b, 2006; Glöckner, 2006,
sion times are normally distributed (Bergert & Nosofsky,
2007) and decisions under risk (Glöckner & Betsch,
2007, Appendix C). Under this assumption, the likeli-
2008a).
hood value of observing a log-transformed decision time
An earlier publication (Glöckner, 2006) highlighted
x given N[µ, ?] can be derived from the density function
the limitations of this method and made a ?rst attempt
of the normal distribution:
to use decision times in individual strategy classi?ca-
tion. To differentiate between intuitive and deliberate de-
1
p(x|µ, ?) = ?
e? (x?µ)2
2?2
,
(3)
cision strategies with equal choice predictions, paired t-
2??2
tests were used to compare individuals’ decision times
in choices for different item types, for which one strat-
and for a set of i independent observations x drawn from
egy predicts no difference and the other does. A sim-
the same distribution by:
ilar method was used in a recent work by Bergert and
Nosofsky (2007). This method can be criticized in dif-
I
ferent respects: (a) it does not take into account the to-
1
L
?
e? (xi?µ)2
2?2
.
(4)
tal ?t of decision times to the total set of predictions of
k(T ) = p(x|µ, ?) =
2??2
i=1
the strategies but is based on pair-wise comparisons of
two types of items only, (b) it gives a certain strategy
the advantage of the null hypothesis without controlling
The density function of the normal distribution (equation
for the beta-error,2 and (c) the results of the choice-based
3) contains two parameters. The mean is represented by
strategy classi?cation (i.e., L
µ, the standard deviation is indicated by ? (? and e are
k(C)) and the t-test(s) (i.e., t
and p value) for choices cannot easily be integrated into
of course constants). The variable x indicates the value
one single measure of ?t for the strategy. While the ?rst
for which the likelihood value should be determined. Ac-
two problems might be circumvented using correlation
cording to the properties of a normal distribution, the like-
measures and estimating the beta-error based on the ex-
lihood value of x decreases with increasing distance from
pected effect size and number of observations, the third
µ (because the exponent of e becomes a higher negative
problem is harder to tackle (Glöckner, in press). The
number) and it also decreases with decreasing ?. The
Multiple-Measure Maximum Likelihood strategy classi-
total likelihood of events is the product of the single like-
?cation method which is introduced next solves the ?rst
lihoods of these events. Therefore in equation 4 the to-
and the last and reduces the second problem by using one
tal likelihood for all observed decision times results from
multiplying the likelihood of all single events (as indi-
1Note, however, that the exact estimations of classi?cation errors
cated by the product sign).
depend crucially on the number of item types and items per type (nj).
2The strategy that predicts no difference in decision time is often
Under the assumption that choices and decision times
given the advantage of being the null-hypothesis. Only if a signi?cant
are independent (for a more detailed discussion of the is-
difference in the direction predicted by the other strategy is found the
sue of independence see below), the likelihood of observ-
null-hypothesis is rejected and a classi?cation for the alternative strat-
ing a set of choices and decision times given the applica-
egy is done. As discussed elsewhere (Glöckner, in press), with small n
this leads to over-classi?cation for strategies that predict no differences
tion of a strategy k, a constant error rate for choices ?k,
because the beta-error is bigger than conventional alpha levels.
and decision times that are drawn from a unique normal

---

Judgment and Decision Making, Vol. 4, No. 3, April 2009
Multiple-measure strategy classi?cation
191
distribution N[µ, ?] is:
spectively, results in equation 8:
Ltotal =
Lk = Lk(T)Lk(C) = p(njk, x|k, k, µ, ?) =
p(njk, xT , xC|k, k, µT , ?T , RT , µC, ?C, RC) =
J
nj
(1 ?
(nj?njk)×
n
k)njk k
J
j=1
jk
nj
(1 ?
(nj?njk)
k)njk
×
I
k
1
njk
?
e? (xi?µ)2
j=1
2?2
. (5)
2??2
I
(xT ?(µT +tT RT ))2
i=1
1
?
i
i
e
2?2
T
×
2??2
i=1
T
Equation 5 should obviously be applied only for decision
I
1
(x
?(µ
R
?
Ci
C +tCi C ))2
2?2
strategies such as WADD and EQW, which predict equal
e
C
. (8)
2??2
decision times for all considered types of tasks. Strategies
i=1
C
TTB and PCS make (interval-scaled) predictions. Let us
This equation contains seven free parameters. For deci-
denote these predictions ti and assume that they are scaled
sion strategies that make different predictions for decision
as contrast weights which add up to 0 and have a range of
times and con?dence for the considered item types (i.e.,
1. Let us further assume that decision times for the item i
PCS, TTB) all seven parameters will be estimated. For
are drawn from different normal distributions with means
strategies that predict equal decision times, the parame-
ter RT is not necessary (i.e., EQW, WADD) and hence
only six parameters have to be estimated. Similarly, R
µ
C
i = µ + tiR,
(6)
can be omitted if a strategy makes all equal predictions
for con?dences. For a RAND strategy, RT and RC can be
omitted as well as the error parameter ?
in which R represents a (non-negative and to be esti-
k which is set to
be .50 (indicating random choices). Hence, for a RAND
mated) scaling parameter. The likelihood value for ob-
strategy only 4 parameters are estimated.
serving a set of choices and decision times drawn from
Likelihood values L
different normal distributions (with equal ?)3 can then be
k should be corrected for the differ-
ent numbers of free parameters N
calculated by inserting equation 6 in equation 5:
p using the Bayesian In-
formation Criterion (BIC) which also takes into account
the number of observations Nobs (Schwarz, 1978):
Lk = p(njk, x|k, k, µ, ?, R) =
BIC = ?2 ln(L) + ln(Nobs)Np.
(9)
J
nj
(1 ?
(nj?njk)×
Individuals should be classi?ed as users of the strategy
n
k)njk k
which has the lowest BIC value. The number of inde-
j=1
jk
pendent observations N
I
obs, which is used to calculate the
1
?
e? (xi?(µ+tiR))2
BIC, is not always equal to the number of total obser-
2?2
. (7)
2??2
vations. According to STATA 10.0 Online Manual, the
i=1
number of independent categories (i.e., types of tasks)
should be used if it can be assumed that the instances of
Furthermore, assuming that con?dence judgments are in-
these categories are highly correlated. This is the case
dependent of choices and decision times and normally
for our data because responses to the repeated presenta-
distributed, con?dence estimation can be added to equa-
tions of one type of items should be similar. I compared
tion 7 in the same manner as decision time.4 From ex-
results using the total number of observations per person
tending equation 7 and adding subscript T and C for pa-
(Nobs = 60 [tasks]* 3[choice, decision time, con?dence]
rameters referring to decision time and con?dence, re-
= 180) and the number of independent categories (i.e.,
types) (Nobs = 6 * 3 = 18) in the simulation reported be-
low and indeed found that the usage of the latter formula
3Alternatively, it could be assumed that ? differs between item types
seems to be preferable.
and increases with increasing ti. Although this relation might also be
The simulations investigated whether choices, decision
modeled in ML calculation, for simplicity a constant ? should be as-
times and con?dence data generated by different strate-
sumed.
gies with certain error rates for choices and different ef-
4The assumption that con?dence judgments are normally distributed
is rather common (e.g., Merkle, Sieck, & Van Zandt, 2008). For a dis-
fect sizes for decision time and con?dence are correctly
cussion of the independence assumption, see below.
classi?ed using the MM-ML method. I expected that this

---

Judgment and Decision Making, Vol. 4, No. 3, April 2009
Multiple-measure strategy classi?cation
192
method a) is capable of identifying the decision strategy
TTB
PCS
EQW
that generated the data, b) is not biased in favor of one or
1
the other strategy, c) differentiates appropriately between
.5
strategies which make identical choice predictions (if the
effect size for time and con?dence is suf?ciently large),
0
1.25
1
0.75
0.5
and d) leads to less misclassi?cations than the usage of
WADD
RAND
the choice-based strategy classi?cation.
1
.5
Proportion Classifications
5 Simulation
0
1.25
1
0.75
0.5 1.25
1
0.75
0.5
Size of Effect Decision Time / Confidence
5.1 Method
TTB class.
EQW class.
WADD class.
PCS class.
The simulation used the 5 decision strategies and 6 types
RAND class.
Graphs by Data Generating Strategy
of tasks from Table 1. I assumed that these tasks were
presented 10 times each, resulting in a total of 60 choices.
In the simulation, the choices, decision times and con-
Figure 1: Strategy classi?cation results by data generat-
?dence were generated by the 5 strategies TTB, EQW,
ing strategy for 60 observations.
WADD, PCS, and RAND. The error rate for choices var-
ied from 5% to 25% in steps of 5%. I also manipulated
the size of the differences between decision times and
constant amount of misclassi?cation in favor of PCS. Re-
con?dences for different types of items in relation to the
member that WADD and PCS make equal choice predic-
standard deviation. To do this, I drew data from normal
tions. Hence, the method generated very few misclas-
distributions N(µ = contrast weight, ? = sd) in which the
si?cations in favor of the more complex strategy (with
mean was the contrast weight de?ned in Table 1 and the
one additional parameter). On the other hand, the accu-
standard deviation sd was varied on the levels 0.8, 1, 1.33,
racy of the classi?cation of data produced by PCS de-
and 2. Remember that the contrast weights are scaled to
pended crucially on the effect size. As one would expect,
a range of 1. Hence, sd = 1 means that for comparing the
with decreasing effect size the number of misclassi?ca-
fastest with the slowest item type, the effect size is 1. The
tion in favor of the strategy not predicting a difference
maximum effect sizes produced by our manipulation of
(i.e., WADD) increased. For our small number of obser-
sd are consequently 1.25, 1, 0.75 and 0.5. For simplicity,
vations, the maximal effect size (i.e., measured between
sd was manipulated jointly for decision times and con-
the most extreme items only) of 1.25 and 1 led to accept-
?dences. For each combination from each strategy, 100
able results. Below that, misclassi?cations prevailed. Fi-
data sets were generated and the MM-ML strategy clas-
nally, data produced by a RAND strategy are to a certain
si?cation was applied. Hence, in the simulation I used a
degree misclassi?ed as being produced by EQW. Note
5 (data-generating decision strategy) x 5 (error rate) x 4
that this misclassi?cation was likely to be due to the fact
(standard deviation) x 100 (repetitions) design. Simula-
that, for the selected item types, EQW predicts random
tions were run using a BIC correction with N
choice for 4 out of 6 considered types. These misclassi?-
obs = 18 and
N
cations could be reduced by including a limit error rate of
obs = 180. The results for Nobs = 18 are reported only
because they were consistently better (i.e., less biased in
.30 for all systematic strategies and not classifying partic-
favor of strategies with less parameters).5
ipants with higher error rates (see discussion below).
The manipulation of the error rate in strategy applica-
tion had only a minor in?uence on strategy classi?cation
5.2 Results
results for all levels of sd. The results concerning the in-
Figure 1 shows the classi?cation results by data gener-
?uence of error rate on strategy classi?cation are shown
ating strategy and maximal effect size (i.e., inverse of
in Figure 2. The left part shows the result aggregated for
sd) aggregated over the manipulation of error rate. The
strong effects (sd ? 1) and the right side for weaker ef-
classi?cation for data generated by TTB and EQW were
fects (sd > 1). As one could have expected, for strong
almost perfect. The classi?cation of data generated by
effects, strategy classi?cation also worked quite well be-
WADD was very good as well, although there was a small
tween PCS and WADD. The classi?cation between these
strategies with equal choice predictions was considerably
5Following Schwarz (1978) we used BIC instead of the alternative
worse if the effect was weaker. There was a strong ten-
Akaike information criterion AIC = -2*ln(Likelihood) + 2*Np. Note,
dency towards misclassi?cation in favor of the strategy
that using AIC in the simulations led to results similar to using Nobs = 18
except for the fact that strategies with more parameters were classi?ed
that does not predict differences in response times (i.e.,
somewhat more often (because 2 < ln(18)).
WADD) as compared to the strategy that predicted differ-

---

Judgment and Decision Making, Vol. 4, No. 3, April 2009
Multiple-measure strategy classi?cation
193









































































































 














 







 
 





 

 


!"# !
!"# !
Figure 2: Strategy classi?cation results by error rate in strategy application for strong effects (sd?1, left) and weaker
effects (sd>1, right).
ences (i.e., PCS). Hence with weaker effects the method
TTB
EQW
is biased in favor of the strategy that predicts no differ-
1
ence (i.e., null hypothesis). In the high effect-size condi-
tions, the increasing error rate had almost no increasing
.5
effect for misclassi?cations (Figure 2, left) whereas in-
0
creasing error rate led to increasing misclassi?cations in
WADD/PCS
RAND
1
the lower effect-size conditions (Figure 2, right).
To investigate whether the MM-ML method leads to
.5
Proportion Classifications
fewer strategy misclassi?cations than the classic choice
0
based strategy classi?cation by Bröder and Schiffer
0.05
0.10
0.15
0.20
0.25 0.05
0.10
0.15
0.20
0.25
(2003a), I rerun the same analysis using the choice
Error Rate in Strategy Application
based strategy classi?cation method and excluding PCS
TTB class.
EQW class.
WADD class.
RAND class.
(because it obviously could not be differentiated from
Graphs by Data Generating Strategy
WADD based on choices only). In line with ?ndings by
Bröder and Schiffer, the analysis worked very well, but
revealed an increasing misclassi?cation rate with an in-
Figure 3: Strategy classi?cation based on choices only by
creasing error rate (Figure 3).
data generating strategy and error rate in strategy appli-
cation.
A considerable number of choices that were produced
by RAND were wrongly classi?ed as being produced by
EQW. This bias was stronger as compared to the one ob-
and RAND were close to perfect. With the higher num-
served for the MM-ML method (see Figure 1). In Table
ber of observations, the classi?cation of PCS was also
2 the classi?cation results for both methods are directly
satisfactory for a lower maximum effect size (i.e., 0.75)
compared for ? = 0.25. It can be seen that over all strate-
but for the lowest maximum effect size (i.e., 0.5) there
gies the MM-ML method leads to a higher level of correct
were still a considerable number of misclassi?cations in
classi?cations as compared to the choice based strategy
favor of WADD.
classi?cation (cf. bold numbers in the diagonals of Table
2).
5.3 Discussion
A ?nal simulation investigated the in?uence of the
number of observations on the quality of the strategy
The simulations revealed that the inclusion of decision
classi?cation with the MM-ML method. Therefore the
times and con?dences in the analysis generally improves
number of observations used in the analysis was raised
strategy classi?cation. This is particularly the case if the
from 60 to 120 (i.e., by using 20 instead of 10 decisions
effects for both variables are strong. If the effects are
per item type). Doubling the number of observation in-
strong, the method also allows differentiating reliably be-
creased the quality of classi?cation (Figure 4). With 120
tween strategies which make the same choice predictions.
observations the classi?cations of TTB, EQW, WADD,
In cases with weaker effects, the method is increasingly

---

Judgment and Decision Making, Vol. 4, No. 3, April 2009
Multiple-measure strategy classi?cation
194
Table 2: Comparison of strategy classi?cation methods.
TTB
PCS
EQW
1
Strategy classi?cation
.5
Data
0
WADD/
1.25
1
0.75
0.5
generating
TTB
EQW
RAND
PCS
WADD
RAND
strategy
1
.5
Choice-based strategy classi?cation
Proportion Classifications
0
TTB
0.93
0.01
0.01
0.00
1.25
1
0.75
0.5 1.25
1
0.75
0.5
Size of Effect Decision Time / Confidence
EQW
0.01
0.79
0.11
0.07
TTB class.
EQW class.
WADD/PCS
0.02
0.06
0.88
0.01
WADD class.
PCS class.
RAND class.
RAND
0.06
0.30
0.04
0.61
Graphs by Data Generating Strategy
Overall
1.01
1.14
1.03
0.68
MM-ML strategy classi?cation
Figure 4: Strategy classi?cation results by data generat-
ing strategy for 120 observations.
TTB
0.96
0.00
0.03
0.01
EQW
0.01
0.94
0.03
0.03
WADD/PCS
0.01
0.02
0.96
0.01
These ?ndings indicate that biased classi?cation due to
weak effect size might not be too much of a problem.
RAND
0.00
0.15
0.02
0.83
However, researchers should check the size of the effect
Overall
0.98
1.10
1.04
0.88
in pre-tests, or they should at least calculate it before in-
Note. Numbers represent percentages of strategy clas-
terpreting their results.
si?cation for the respective strategy (in columns) for
an error rate of ? = .25 only. In the MM-ML method,
WADD and PCS are combined concerning data gener-
6 Applying the MM-ML method in
ating strategy and strategy classi?cation for compara-
research practice
tive reasons.
Applying the MM-ML method obviously necessitates the
use of a statistical package that allows for calculating
biased in favor of the strategies which predict no differ-
complex maximum likelihood estimations. I have pro-
ence concerning decision times and choices (i.e., which
grammed the necessary estimation routines in STATA.
has less free parameters). This problem obviously occurs
The estimation programs are described in the supplemen-
with any statistical test because the latter strategies have
tary material (http://journal.sjdm.org/vol4.3.html). Ap-
the advantage of being the null hypothesis.
plying the method mainly requires bringing data into a
The simplest way to circumvent the problem is to in-
speci?c format and de?ning predictions. The overall es-
clude items for which particularly strong differences in
timation program (which can be applied to any number
con?dence and decision time are expected. Additionally,
of item types, choices per item, participants, and strate-
more items could be used to increase power by increas-
gies) provides per-individual estimates of the parameters
ing within-subjects sample-size. An increase from 10 to
for each strategy (Figure 5, top), as well as an aggregated
20 choices per item type reduces the bias in favor of the
matrix with the total likelihood (i.e., BIC score) that the
null hypothesis considerably. Finally, one might consider
data for each individual (in rows) were produced by a par-
using a different correction of the likelihood than the BIC
ticular strategy (in columns) (Figure 5, bottom).
correction (similar to setting a compromise alpha level).
However, to the best of my knowledge there is no sim-
The STATA output will be explained in more detail.
ple method to ?nd the correct adjustment (although it
The individual output (Figure 5, top) shows the results
could, of course, be derived from simulations). Hence
for comparing the data of subject 1 with the predictions
including items with expected large differences and in-
of strategy 4 (see last line of output). The total num-
creasing the number of items seem to be preferable. Note
ber of observations is 126 (6 frequencies for choices in
that previous studies found strong effects for con?dence
task types, 60 decision times, 60 con?dence ratings).
and time in probabilistic inference tasks (Glöckner, 2006;
The resulting parameter estimates are listed as constant
Glöckner & Betsch, 2008c), as well as in gambling tasks
coef?cients. In the example, the choice error rate (ep-
(Glöckner & Betsch, 2008a; Glöckner & Herbold, 2008).
silon) was .167, the log-transformed (and for order ef-

---

Judgment and Decision Making, Vol. 4, No. 3, April 2009
Multiple-measure strategy classi?cation
195
 *&)&*+&*,-
 *&)&*+&*,-
23 -







/&,*.01-
()&*+&*,-! ",*.-


  !"# #$% &'











 















 



  














 














 



























 




















 











































 



























 















 












  







 




 


 








 









   
























  




  

 

 
 

 



   
Figure 5: Example output of the STATA implementation of the Multiple-Measure Maximum Likelihood strategy
classi?cation method for parameters per individual (top) and for the overall estimation (bottom). The individual
output contains estimates for all coef?cients and the overall ?t of the individual data to the prediction of the considered
strategy. The overall estimation shows BIC scores for each individual (rows) and each of the ?ve considered strategies
(columns). Lower scores indicate a better ?t.
fects corrected) mean decision time was 8.57 (mu_Time),
mally distributed decision times and con?dence ratings,
the mean con?dence (mu_Conf ) was 53.98. The provided
and independence between observations.
signi?cance tests (which test if the estimated constant co-
The lower part of Figure 5 shows the output for the
ef?cient is different from zero) are mainly informative for
results of all individual comparisons. It presents the re-
the rescaling factors R. In this example, RT (R_Time) and
sulting BIC scores for each subject (in rows) and consid-
RC (R_Conf ) were both signi?cantly different from zero.
ered strategy (in columns; i.e., TTB, EQW, WADD, PCS,
This indicates that the speci?c predictions for time and
RAND). Lower values indicate a better ?t. The exam-
con?dence (re?ected in contrast weights) signi?cantly
ple result of subject 1 and strategy 4 is consequently pre-
contribute to explain the data. The tests for R produce
sented in row 1, column 4 of the matrix. The strategy that
results similar to correlations between data and contrast
explains a subjects’ data best can be easily determined by
weights (i.e., the correlations of observed decision times
identifying the lowest number in the persons’ row (e.g.,
with the contrast weights for the respective task types).6
the data of subject 1 are most likely generated by strategy
The BIC score (last line) indicates the overall ?t of data
1).
and strategy predictions for the speci?c participant. More
precisely, it gives the corrected log-transformed likeli-
The MM-ML method has been successfully applied to
hood for the data given the application of a certain strat-
empirical data (i.e., Figure 5 is based on real data) and it
egy under the assumptions of a constant error rate ?, nor-
appears that for the types of items considered here (us-
ing 60 observations only) the method is well applicable.
6Differences result from the fact that the parameter is estimated
Additional practical suggestions on the application of the
jointly with the other parameters in the MM-ML method.
method are given in Glöckner (in press).

---