Getting Real About Prediction in Marketing Research

Getting Real About Prediction in Marketing Research









By Jan Hofmeyr, International Director of Innovation
Synovate Brand and Communications Practice, Cape Town



Copyright © Synovate 2006

1

---

Getting Real About Prediction in Marketing Research







Abstract


I begin this paper with a meta-analysis of the predictive performance of customer satisfaction, purchase
intention, and advertising recall. I show that it is poor. Developments in cognitive neuroscience and
evolutionary psychology help us to understand why. I suggest two routes to improvement: first, through
the development of better advertising diagnostics using a new measure of emotional engagement
coupled to text mining tools; and second, through the development of a new measure of attitudinal equity
based on the Zipf distribution. I finish the paper with a brief discussion about the conservatism of our
industry and suggest the need for some changes in orientation.






























Copyright © Synovate 2007

2

---

Introduction


For all its complexity, a lot of attitudinal marketing research can be summarised very simply: find the
number that needs to go up!

• Find the brand attribute that will make the biggest difference to sales.

• Find the advertising execution that will communicate the desired brand attributes most
effectively.

• Find the media mix that will make the biggest difference to advertising impact.

Whatever the diagnostic variables – brand attributes, touch-point experiences, advertising recall,
perceived media exposure, barrier effects – this ‘find the number’ logic requires a dependent variable
against which to model... which leads to the question: how well do typical numbers do?

The answer is discouraging. In a meta-analysis of the customer satisfaction literature, Szymanski and
Henard (2001) found an average correlation between customer satisfaction and claimed loyalty
(measured using purchase intention) of R = 0.52; i.e., R2 = 0.27. The results relating customer
satisfaction to ‘word of mouth’ were not much better; i.e., R = 0.57, R2 = 0.32. And in what is still the
largest reported test of the performance of multiple advertising executions (Lodish et al, 1995), no
relationship was found between advertising recall and sales.

In this paper I ask the question: can we do better? And if so, why do we continue to use marketing
metrics that are clearly flawed?

The paper begins with an analysis of three of the most widely used measures in marketing research. All
perform poorly. I discuss the reasons and suggest remedies based on what are now only relatively new
learnings about how the world works - non-linear dynamics, small world networks, cognitive
neuroscience. I test two new measures; and close with a discussion of the conservatism of our industry
and what must be done to fix it.



A reminder: The validity of key marketing measures


One reason why it’s difficult to develop valid attitudinal marketing measures, is because we seldom have
data which compares our measures to real behaviour in controlled environments. Research based on real
behaviour is so sparse that Szymanski and Henard don’t even reference it in their meta-analysis. Yet, it
can be found. Perhaps we just prefer to ignore what’s there?


i). A critical review of ‘purchase intention’

Along with customer or brand satisfaction, there is probably no more widely used measure in marketing
research than purchase intention. It’s central to most loyalty measurement systems. It’s used as a
measure of ‘persuasion’ in many ad-testing systems. And it’s at the core of most concept and product
tests systems. So - how well does it relate to what people actually do?

Table 1 is based on a survey of published research on the relationship between purchase intention and
real behaviour that dates back to 1966.







Copyright © Synovate 2007

3

---

Table 1

Results from Longitudinal research linking Purchase Intention to subsequent Behaviour


Author(s) Product
Nature of Study

Results
Categories
Juster, F.T. (1966) Motor
cars,
Measure purchase intention
R = .43 (motor cars), .24
appliances
using 11-point probability
(appliances).
scale. Correlate with
R2 = .18 and .06
whether or not product
respectively.
subsequently bought.
Bonfield, E.H.
Brands
of
grape Measure purchase intention
Results
are
significant.
(1974)
juice
using 7-point likelihood
Average correlation: .40. R2
scale. Correlate with
= .16
whether or not the brand
subsequently bought.
Sewall, M.A.

Women’s apparel Mall intercept. Measure
Results are significant but
(1981)
purchase intentions using 5-
poor.
point scale. Compare with
R = .27; R2 = .07
subsequent purchases.
LaBarbera, P.A.
Margarine,
Diary study: Purchase
Purchase
intentiont

and D. Mazursky
coffee, toilet
intention measured every
Purchase t+1: R = .24
(1983)
paper, paper
two weeks for 20 weeks.
Average R2 = .06
towels, macaroni
Correlated with subsequent
brands bought.
Morwitz, V.G., E,

Motor cars, PC’s
Measure intention to buy in
No correlations reported. On
Johnson and D.
next six months, every six
average, 29% of those who
Schmittlein (1993)
months. Longitudinal
say they will buy, do; which
research measures actual
means that 71% don’t.
behaviour.
Bemmoar, A.C.
Multiple
durable A meta-analysis of
No correlations. 64% of
(1995)
categories
published studies in which
those who say they will
PI was measured and
definitely buy, don’t. Most
subsequent behaviour was
purchases come from those
observed.
who say they will not buy.
Chandon, P., V.G.
On-line
grocers, Measure purchase intention.
Correlations:
.44
(grocer),
Morwitz and W.J
motor cars, PC’s
Observe next purchase or
.12 (motor cars), .16 (PC’s).
Reinartz (2005)
purchase within six months
Average correlation: .24
(R2: .06)
Seiders, K., G.
High
end
Measure purchase intention.
Purchase
intention No of
Voss, D. Grewal,
clothing and
Correlate with 52 weeks of
visits: R = .11, R2 = .01; and
and A. Godfrey
home furnishings
behaviour in a data-base.
Amount spent: R = .10, R2 =
(2005)
.01
Perkins-Munn, T.,
Trucks,
Measure
respondent
Intentiont
Repurchaset+1:
l.Aksoy, T.L.
Pharmaceuticals
attitudes. Record subsequent
R = .44 and .65
Keiningham, D.
behaviour over a 15
respectively. Average R2 =
Estrin (2005)
month period.
.31
Intentiont
SoWt+1: R =
.47, .45 respectively.
Average R2 = .21


These results can only be described as ‘bad’: the average correlation between what people say they
intend to do and what they actually do, is R = 0.30, R2 = 0.09. In other words, 91% of the variance is not
captured by purchase intent. Two of the cited papers report no correlations. Both report that more than
half of the people who say they will ‘definitely buy’ something - don’t.




Copyright © Synovate 2007

4

---

ii). Customer satisfaction: just how predictive is it?

In this case we go back to 1983.

Table 2

Results from Longitudinal research linking Customer Satisfaction to Retention and Share of Wallet


Author(s)
Product
Categories
Nature
of
Study

Results
LaBarbera, P.A.
Margarine,
coffee,
Diary study: Satisfaction
Satisfactiont Purchase
and D. Mazursky
toilet paper, paper
measured every two weeks
t+1: R = .20
(1983)
towels, macaroni
for 20 weeks. Correlated
Average R2 = .04
with subsequent brands
bought.
Jones, T.O. and
Manufacturer
of
Longitudinal
study:
Extremely
satisfied
W.E. Sasser
industrial supplies
Measures customer
customers six times less
(1995)
satisfaction and compares
likely to defect – but
with subsequent retention-
doesn’t report overall
defection.
retention-defection rates.
Bolton, R. N.
Telecommunications Longitudinal
survey
of
Satisfaction
accounts
for
(1998)
customers: Two waves.
most of the variance
Models satisfaction and
explained (42%). But
other inputs against length
Bolton fails to report
of customer duration.
percent variance explained!
Mittal, V. and
Motor cars
Customer satisfaction in
Repurchase rate of
W.A. Kamakura
33rd month of car ownership
dissatisfied customers =
(2001)
– compared with whether
48%; repurchase rate of
brand switched or not when
satisfied customers = 72%.
new car bought
Verhoef, P. C.
Financial services
Measure attitudes inc.
Regression
model
(2003)
satisfaction. Modelled
acceptable: R = .43; R2 =
against subsequent
.18. But satisfaction fails to
retention-defection.
make the model.
Capraro, A.J., S.
University
health
Measure attitudes inc.
Regression model including
Broniarczyk, and
plans
satisfaction, one month
satisfaction is significant
R.K. Srivastava
before decision. Re-contact
but R2 is only .08.
(2003)
after decision.
Keiningham, T.L.,
Financial services
Measure attitudinal
A
dichotomized
satisfaction
T. Perkins-Munn,
satisfaction. Obtain
scale (1-8; 9-10) lifts SoW
and H. Evans
customer share of wallet
from about 10% to 15%.
(2003)
from 3rd party sources. Fuse
Average R = .27; R2 = .07
the data and analyze.
Bowman, D. and
Processed metals
Measures attitudes inc.
Satisfaction
correlates
D. Narayandas
Satisfaction. Compares with
poorly with SoW; and does
(2004)
subsequent data-base
not correlate with
information.
profitability.
Gustaffson, A.
Telecommunications Measure
attitudinal
Satisfaction Churn: R =
M.D. Johnson, and
satisfaction. Correlate with
.13, R2 = .17
I. Roos (2005)
churn defined as ‘time spent
as a customer’
Seiders, K., G.
High end clothing
Measure
attitudinal
Satisfaction No of vists:
Voss, D. Grewal,
and home
satisfaction. Correlate with
R = .07, R2 = .00; and
and A. Godfrey
furnishings
52 weeks of behaviour in a
Amount spent: R = .07, R2
(2005)
data-base.
= .00
Perkins-Munn, T.,
Trucks,
Measure
respondent
Satisfactiont
l.Aksoy, T.L.
Pharmaceuticals
attitudes. Record
Repurchaset+1: R = .24 and
Keiningham, D.
subsequent behaviour over
.22 respectively. Average
Estrin (2005)
a 15 month period.
R2 = .05

Copyright © Synovate 2007

5

---

These results are so bad one wonders what we have been doing over the years. The average correlation
between customer or brand satisfaction and behaviour is R = 0.13, R2 = 0.02. Notice how many authors
avoid reporting the percent variance explained. I once corresponded with one of these authors and noted
that they hadn’t reported the percent variance explained. They wrote back that they’d lost the results!

While on this subject, let’s note an attempt to link the American Customer Satisfaction Index (ACSI) to
shareholder value (Anderson et al., 2004). The article was published in 2004, yet it only covered the
relationship between the ACSI and shareholder value in the years 1994 - 1997. Anyone who knows the
history of securities will know that markets were volatile from 1998 (Russian crisis) to 2002 (dot-com
bubble). I wondered why the authors’ analysis ended just before the volatility? Is it because the volatility
broke the relationship?

The problem with articles like these, is that publication in a highly respected journal tends to lead to
uncritical citing - and so it lays the foundation for the uncritical acceptance that the ACSI is linked to
shareholder value (e.g., Cooil et al, 2007).


iii). Advertising testing: recall and persuasion

It is probably true to say that measures of recall and persuasion dominate communications measurement
- still. Yet one of the most thorough analysis of advertising effects ever conducted (Lodish et al, 1995)
showed that there was no relationship between either recall or ‘persuasion’ and subsequent sales (See
figure 1).


Figure 1: Advertising recall and subsequent sales
(from Beth Lubetkin presentation: Marketplace Advertising Research Workshop, 1991)





iv). To summarise

Our meta-analysis of satisfaction, purchase intention, and ad-testing and tracking metrics (recall,
persuasion) suggests that the links between these measures and what people really do are tenuous at
best. Given the extent to which they still dominate marketing measurement, it is worth repeating the
validation results here:

• Purchase intention:


R = 0.30; R2 = 0.09

• Customer/brand satisfaction:

R = 0.13; R2 = 0.02

• Advertising recall/persuasion:
No relation to sales, 389 split cable tests
Copyright © Synovate 2007

6

---

We ought to be embarrassed. Some may find consolation in the fact that our industry is not alone in
having poor models. Economists come to mind. Still, as I will show, we can do better. The question then
becomes: why don’t we?


Identifying the causes of our poor models

i). The root of our problems: Fishbein and Ajzen

I blame some of our industry’s addiction to poor measures to the lasting influence of the following
equation due to Fishbein and Ajzen (1975):


Behaviour 
 
 


Eq.1

The equation is based on the view that there should be a strong relationship between what people intend
to do and what they actually do, circumstances permitting. From the results of our meta-analysis,
however, we know that behavioural intention is a poor guide to behaviour in markets.

But it’s not only the left-hand side of the equation that is problematic. Consider the right-hand side: the
strength of a behavioural intention is seen to be a function of two factors. The first is an evaluation of the
outcomes of the behaviour (for example, a product, service, or brand choice). The second is an estimate
as to whether or not relevant others would approve of the behaviour (a cognate of loyalty metrics like
recommendation, positive/negative word-of-mouth). The ‘ ω ’ capture the idea that the lists of outcomes
and ‘relevant others’ should be weighted according to their importance to a person.

The entire approach is what cognitive scientists call ‘computationally intensive’. It requires the brain to
behave like a very exact calculating machine. Let’s take the first term. Getting to a ‘number’ involves the
following steps: list all the attributes that are relevant, attach a brand performance level to each attribute,
assign a level of desirability (or importance) to each attribute, weight each performance rating by the level
of desirability - and finally, add the results to produce a summary number. Store in memory.

Now turn to the second term. List all the people whose opinions you value. Estimate what you think they
would say about the choice. Assign an importance to the opinion of each person. Weight what you
believe their opinion would be by how much you value their opinion. Add the numbers to produce a
second summary number.

Add the two numbers. The higher the result, the stronger the intention should be. The stronger the
intention, the more likely the behaviour should be.

This computationally intensive but logically simple algorithm is at the heart of almost all equity, loyalty,
and driver research in marketing. It’s known in the cognitive decisions literature as the ‘weighted additive’
model of brand choice (See Bettman et al, 1998). But is it how the brain works? Increasingly we’re
realising that the answer is ‘no’; and that if it were, we probably would not have survived into the 21st
century.

Human beings are ‘ecologically rational’ (See Gigerenzer and Todd, 1999). Our brains have evolved to be
maximally efficient under constraints of energy, information, and time. This efficiency involves the use of
‘decision short cuts’ which work because they’re matched to the way the world is. In the brand domain,
two great example of this kind of short cut, are the choice of the most visible brand, or the choice of the
market leading brand. Why waste time testing every brand for yourself when you can benefit from the
implicit experience and endorsement of everyone else by choosing what most of them choose?

The evolutionary view of human choice coincides with what we’re learning about the brain when we watch
it under fMRI scanners. To quote the neuroscientist, Read Montague (the first person to look at how the
brain behaves when tasting Coke and Pepsi labelled or blind): ‘[The brain is] … almost perfect: it’s slow,
Copyright © Synovate 2007

7

---

noisy, and imprecise’. What he means is that we have evolved to do as little computation as possible
when making a choice; and that we do so with as little information as possible and only ‘just in time’. In
this way we make choices and pursue goals effectively in what, for most of our evolutionary history has
been (and still is) a resource- and time-poor environment.

In sum: we are heuristic decision makers. We may sometimes feel that we’re agonising over a decision.
But mostly we decide fast and effectively; with minimal use of energy and little reference to multiple
attributes (Deep Blue might have beaten Gary Kasparov in their chess match-up, but Kasparov was by far
the most efficient computer). We decide holistically and on the basis of summary evaluations. We do not
create long lists (not even subconsciously) and then laboriously plough through them to a summative
evaluation. The implications for modelling and research design are clear:

• Attribute intensive methods do not reflect the way decisions are made.

• Questions should be designed to capture the mind’s summary decision heuristics.

• Focus only on brands that are relevant to a respondent.


ii). The problem of habituation

If the weighted additive model of brand choice is one of the ways in which marketing research and real
people part ways, a second comes from what we now know about reinforcement learning.

Learning is a ‘trial and error’ process guided by evolutionary acquired brain mechanisms of which one of
the most important is looking at what other people do. The sign in the mind that a particular choice has
worked - whatever the choice might be, be it product, service, or brand - is pleasure; which goes with the
release of ‘reward’ substances like dopamine. But initial success doesn’t stop the learning process. What
the brain does next, is to shift the reward away from the choice itself, to precursors of the choice. In other
words, we start to gain more pleasure from working out how to achieve what we want, than from actually
getting there. This process of reinforcing ‘right’ choices; and then withdrawing the reward as soon as
we’ve learnt what works, is an irrevocable part of our natures - and here’s the point: it fundamentally
undermines product, service, or brand satisfaction because it constantly displaces the reward that is first
felt when the right brand is chosen. In psychological terms, it’s the process of habituation.

Every marketer knows that people become habituated to what’s being done for them. Every marketer
knows that there is literally no end to what people will come to expect. And yet, most marketers continue
to focus on ‘satisfaction’ (or even worse, ‘delight’) as the key to achieving loyalty.

Fortunately, as providers of brands and services we do not have to delight people all the time. For
another aspect of life which is obvious but which we overlook when we put on our marketing ‘hats’, is that
people form long-term commitments. The fact that a choice has become routine - boring even - does not
mean that we will stop making the choice. If it works; and if it allows us to move on and devote time and
energy to other things that we really care about - then there is every chance that the original choice will
survive. It becomes an automatic part of the way in which we get through daily life.

Our new understanding of the neural mechanisms behind learning, help us to understand why the narrow
focus on satisfaction has led to relatively poor returns. The real challenge when it comes to selling, is two
fold: first, how to be the first choice; and second, how to maintain loyalty once initial satisfaction has worn
off. Paradoxically, there are two primary routes. The first is - to make it an unthinking decision precisely
because it’s not important; and the second is - to make it so important that it survives moments of
disappointment (See Hofmeyr and Rice, 2000).


From the marketing and measurement point of view, the implications are:

• Loyalty can be had without involvement when a decision becomes a habit.

• Loyalty can also be had when high involvement leads to commitment.

• Establishing the level of involvement is therefore key to fixing loyalty metrics.
Copyright © Synovate 2007

8

---

iii). We are comparative animals and regularly experience ambivalence

It ought to be obvious that what matters is not how highly a product, service, or brand is rated per se, but
how highly it rates relative to others. It ought to be obvious therefore, that it is better for a brand to score
low, but win; than for it to score high, but lose. Yet comparative modelling appears to be the exception
rather than the rule. Most models are based on what I call ‘straight line’ thinking. In other words, they
model directly from a brand’s score, to its use. Even when multiple brands are included in a survey, each
brand still tends to be modelled against its scores alone.

When methods include comparative metrics (See Hofmeyr and Rice, 2000; Bowman and Narayandas,
2004), it’s usually the comparative metrics that account for the greatest variance.

If ‘straight line’ thinking is one of the aspects of our modelling approach that gets us into trouble, then a
second is failing to allow people to be ambivalent. Although we are comparative animals and prefer to be
able to make a choice based on a winner, the psychology of the comparative process doesn’t guarantee
that there will always be a winner (See Cacioppo and Berntson, 1994). Ties happen - and they should
therefore be allowed in measurement.

These considerations lead to two modelling and research design principles:

• Modelling needs to be done at an individual level. It is the ‘within respondent’ comparison

of brand scores that best predicts what people will do.

• Modelling needs to allow for ambivalence and indecision. It is a feature of people, that

they may rate two or more brands equally well.


iv). A mistaken focus on retention

Most validation in marketing has focused on retention or recruitment. Reichheld’s work in the mid- 1990’s
(Reichheld, 1993) helped to reinforce this and narrow the focus to retention alone. Increasingly,
marketers are realising that it’s unhelpful. Most changes in a brand’s fortunes, both up and down, come
not from people defecting from (or initiating use of) a brand, but from people ‘sliding’ - in other words,
from increases or decreases in the level of use, no matter what that level is (Coyles and Cokey, 2002).
Sheer use is an artificial threshold. Whether a brand be a packaged good whose use is captured in a
series of purchase transactions, or a subscription service which may have to fight for a person’s business
with other simultaneously owned services, what matters is share of wallet.

These considerations lead to the following principle when developing marketing models:

• Sheer churn is a poor dependent variable. Point-in-time purchase probabilities; or over

time ‘share of wallet’ are what matter as outcomes.

• Validation as ‘retention/acquisition’ is therefore unimpressive. The true test of a measures

of brand strength, is its correlation with share of wallet.


v). To summarise: the reasons many marketing models are poor

Let’s summarise the ways in which many of the practices and doctrines of marketing research fail to
reflect what we now know about people:

• People are not exact mechanical calculators. The weighted additive model of choice,

which underpins a lot of quantitative motivational analysis, does not reflect the way

people make choices.

• People are comparative animals. A poorly rated ‘winner’, still wins. A highly rated ‘loser’,

doesn’t. Modelling therefore ought to be based on how brands rate relative to each other.
‘Straight
line’
algorithms cannot but be weaker.



Copyright © Synovate 2007

9

---

• The process of habituation dooms customer satisfaction as the basis for loyalty models.

Loyalty has two foundations, namely, convenient, unthinking habit, or involvement driven

commitment.

• It’s not about retention or acquisition, it’s about purchase propensity and share of wallet. Data-
base research shows that it is when people change how much they are using a service or brand,
that businesses gain or lose.

• Ambivalence is a fact of customer/consumer life. The comparative decision process doesn’t
always produce clear winners. Sometimes two or more brands may be rated equally. Modelling
should accommodate that.

• It is the individual decision maker that needs to be modelled. Attribute lists should not be endless;
and brand lists should zero in on the unique set of brands that are relevant to each respondent.

Let’s turn now to the development of new tools. I will focus on two areas: advertising performance and
attitudinal brand equity.



Better Measurement of Advertising Performance

i). Ad-testing metrics: emotional engagement

Figure 2 is a list of typical diagnostics used in advertising research. The first three fall into conventional
groupings: first, there are statements about the advert itself - e.g., boring, irritating, liking, etc.; second,
there are statements about whether or not the advert might create ‘buzz’ - e.g., sticks in the mind, will get
people talking, etc.; and third, there are statements about what the advert says about the brand - e.g.,
conveys new information, makes the brand more interesting, etc. But the group that performs best is a
group seldom seen in advertising testing - it’s the fourth which has to do with the extent to which the ad
successfully embeds the brand in relevant personal experience.


Figure 2: A list of typical advertising diagnostic questions

Attributes of the advert

It is believable

It is understandable
It is boring
Average correlation with brand impact: 0.30

It is irritating

It is likeable

The advert leaves its mark and creates ‘buzz’

It sticks in the mind

It advert is impressive to watch
Average correlation with brand impact: 0.31

It will get people talking

I would talk about it

The advert evokes the brand

It makes me more interested in the brand
It tells me something new about the
Average correlation with brand impact: 0.36
Brand

The advert evokes relevant personal experience
It brings to mind things I really care about
Average correlation with brand impact: 0.42
It brings to mind things I really enjoy



Copyright © Synovate 2007

10

---