The Effect of Brand Awareness on the Evaluation of Search Engine Results

CHI 2006 · Work-in-Progress
CHI 2007 • Work-in-Progress
April 28-May 3, 2007 • San Jose, CA, USA

The Effect of Brand Awareness on the
Evaluation of Search Engine Results

Bernard J. Jansen

Abstract
College of Information Sciences
In this paper we investigate the effect of search engine
and Technology
brand (i.e., identifying name that distinguishes a
The Pennsylvania State University
product from its competitors) on evaluation of system
University Park, PA 16802
performance. Our research is motivated by the large
jjansen@acm.org
amount of search traffic directed to less than a handful

of Web search engines, even though many are of equal
Mimi Zhang
technical quality with similar interfaces. We conducted
College of Information Sciences
a laboratory experiment with 32 participants measuring
and Technology
the effect of four search engine brands while controlling
The Pennsylvania State University
for the quality of search engine results. Based on
University Park, PA 16802
average relevance ratings, there was a 25% difference
mzhang@ist.psu.edu
between the most highly rated search engine and the

lowest, even though search engine results were
Ying Zhang
identical in both content and presentation. We discuss
Department of Industrial and
implications for search engine marketing and the
Mechanical Engineering
design of empirical studies measuring search engine
College of Engineering
quality.
The Pennsylvania State University
University Park, PA 16802
Keywords
yzz114@psu.edu
User intent, Web queries, Web searching, search
engines
ACM Classification Keywords
H.3.3 [1] Information Search and Retrieval – Search
Copyright is held by the author/owner(s).
process: Measurement, Experimentation, Human
CHI 2007, April 28–May 3, 2007, San Jose, California, USA.
Factors
ACM 978-1-59593-642-4/07/0004.
2471

---

CHI 2006 · Work-in-Progress
CHI 2007 • Work-in-Progress
April 28-May 3, 2007 • San Jose, CA, USA


Introduction
sign, name, or mark that distinguishes an organization
There has been a rapid growth in the search engine
or a product from its competitors. Therefore, good
market since its inception. Search engines continue to
branding can results in loyal customers.
attract large number of Web searchers and consistently
rank as some of the heavily visited sites in the market
However, the effect of branding on technology design
in terms of the number of visitors. There are
has not been well acknowledged, a CHI 2001 panel on
approximately 4,000 search engines on the Web;
branding being an exception [5]. Park, Harada, and
however, only a handful dominate in terms of usage.
Igarashi [6] report that the users’ perceptions of a
product’s brand affect their perceptions of mental
From a technological point, this clustering is interesting
demand. While there may be some recognition that
because studies report that the performance of most
branding is important in the marketing of product,
major search engines is practically the same [c.f., 1].
there has been little research in to the brand effect on
Performance is typically defined as returning relevance
the evaluation of system performance.
results. Performance is measured by precision, which is
the ratio of relevant documents to the total number
In this research, we measure the effect of brand
returned at some point in the results listing.
perception on user perception of the performance of
Web search engines.
The interfaces of most search engines are also similar,
namely a text box, some verticals (i.e., tabs for
Research Objectives
searching the Web, Images, Audio, etc.). In studies of
Our research objective is: How does branding affect
search engine interface usability, the results among
overall user evaluation of results retrieved by Web
search engines has been similar [c.f., 7].
searching systems?
Given the similarity in terms of technology and
To address this research question, we designed a study
interface design, why do only a small number of search
that altered the brand of search engines for a set of
engines dominant Web traffic? Do other elements affect
queries while controlling for the quality and display of
the evaluation of a search engine’s performance?
the results. We report the specifics of our design in the
Seeking the answers to these questions motivate our
following section.
research.
Research Design
Review of Literature
Data Preparation
Searching engine interfaces contain branding elements,
To investigate our research questions, we first
such as symbols, logos, and names. A brand is the
extracted a set of e-commerce queries from an
intangible sum of an organization’s attributes, which
approximately one and half million queries Web search
can include an organization’s name, history, reputation,
engine transaction log using a modified snowball
and advertisement. A brand is also identifying symbol,
technique. From these queries, we selected four queries
2472

---

CHI 2006 · Work-in-Progress
CHI 2007 • Work-in-Progress
April 28-May 3, 2007 • San Jose, CA, USA


Procedures During Study
representing four searching domains: medical,
Study Procedure
For each participant, a moderator
entertainment, travel, and ecommerce. We developed
We recruited 32 participants from a major US
read the participant a short
searching scenarios around each of the four queries.
university. The age range was 18 to 25 years. There
introduction, explained to each
The four queries used are: camping mexico, laser
were 8 females and 24 males. Prior to the search tasks,
participant that they would be
removal, manufactured home, and techo music.
the participants completed a demographic
conducting some searches using
questionnaire and answered questions about his/her
Web search engines, and reminded
We then submitted these four queries to a major U.S.
Web searching include the search engine(s) most
the participant to think aloud. We
search engine (i.e., Google) using a software
frequently used.
used an unrelated practice task to
application that not only submitted the queries but also
explain the think aloud protocol.
retrieved the first search engine results page (SERP) for
We presented each participant with all four queries, one
each query exactly as it would be presented to a
at a time. Each participant completed one query before
We then read the participant one
human user. The total time from submission to
moving to the next. The moderator would read the
of the four searching scenarios,
completion of result retrieval took approximately 30
applicable scenario before moving to the next query.
informed him or her that the query
seconds. We then removed all identifying logos, text,
We counterbalanced the order of search engines and
already had been entered into the
URLs, and HTML code from the Google result pages. We
the order of the searching scenarios to eliminate
search engine and results
removed the redirects in the results, so the URLs
ordering effects.
returned, and asked the
pointed directly to the targeted Web site. This left us
participant to continue the search.
with four cleaned results pages.
While the participant was searching, the moderator
The participant would then
annotated utterances and user actions using an
continue the search as if he or she
We then got screen captures of SERPs from Google,
application that the researchers designed for
had entered the query. The
MSN Live Search, and Yahoo!, all major and well-known
quantitative and qualitative data capture during Web
session for that query would end
Web search engines, for each of the four queries.
searching studies such as this one. After the participant
when the participant took some
Additionally, we developed an in-house search engine,
had completed all four query sessions, the moderator
action that would remove them
AI2RS, and got screen captures of the AI2RS results
returned the participant to the first query, and the
from the presented results page
pages for each of the queries.
participant visited all Web pages for each query that
without returning (i.e., submit a
the participant had not visited during the session. The
new query, go to a new results
Using the cleaned Google results and the images from
participant evaluated the Web document and presented
page, go to a different search
the AI2RS, Google, MSN Live Search, and Yahoo!, we
a basis for the evaluation. The moderator collected
engine, etc.). The moderator
developed four experimental SERPs for each of the four
these Web document evaluations again using the data
instructed the participants to
queries. At the end of this process, we had sixteen
collection application. Approximately one hour was
describe the screen content they
experimental SERPs, four from each search engine for
required to complete the sequence for each participant.
were viewing , evaluate its
each of the four queries. However, regardless of the
relevance to the task, and explain
search engine branding elements, the results were
Results
why they moved from one item to
identical across all search engines for each query.
We now return to our research question (How does
the next.
Figure 1 shows the building of an experimental SERP.
branding affect overall user evaluation of results
retrieved by Web searching systems?) with results
shown in Tables 1 and 2.
2473

---

CHI 2006 · Work-in-Progress
CHI 2007 • Work-in-Progress
April 28-May 3, 2007 • San Jose, CA, USA


We cropped each SERP image
using only the branding elements
at the top of the SERP (i.e., logo,
search box and button) and
bottom (i.e., results page
hyperlinks) of each image. We
then built a hyperlink page
structure to hold the top and
bottom images. For the search
engine results, we used the
cleaned Google results.
Our goal in this process was to
be able to isolate the effect of
the branding variable while
controlling for the number of
results, result presentation,
and quality of research. We
used only the first SERP for each
query because most searchers
only view the first results page [3,
4].
We decided to use one style of
results formatting because prior
work has noted that minute
differences in the presentation of
search engine results can affect
how users interact with those
results [2]. There have been other
figure 1. Example of Experimental Search Engine Results Page.
studies of search engine

performance, but we wanted to
Discussion
Howe
o
ver, there were dramatic differe
r nces in how
o
control for variation in the quality
In this experiment studying the effect of branding on
participants rated th
participants
e perfo
rated th
r
e perfo mance of
manc
eac
e of
h
eac sear
sea ch
ch
of results.
engine
gi
usi
us n
i g relev
l a
ev n
a ce
c of retrieval res
e
u
trieval res lts.
lts Performance
Performa
evaluation of system performance, regardless of which
evaluations v
evaluatio
a
ns v ried
a
b
ried y
b mo
y
r
mo e
r than 25
e
% b
than 25
e
% b twee
e
n t
twee
he
n t
top
he
-
top
search engine a participant used for a particular
icular
most rate
most rat d se
d s arch
e
engin
arch
e
engin an
a d the bottom
n
.
d the bottom Again, this
Again, this
domain, th
domain, t e result
e
s fo
result
r ea
s fo
c
r ea h
c domain we
domain w re design
re desig ed to
ed to
di
d ffer
i
e
ffer nc
n e wa
c
s not
e wa
e
s not d ev
d
en
ev
thoug
en
h all
thoug
th
h all
e
th re
e
sults
re
we
sults
re
we
re
be the same.
identical.
identical

2474

---

CHI 2006 · Work-in-Progress
CHI 2007 • Work-in-Progress
April 28-May 3, 2007 • San Jose, CA, USA



Concerning what search engines
participants reportedly used
Queries
Average
Google was mentioned by 31
Search Engines
camping mexico
laser removal manufactured
home techo
music

participants, Yahoo! by 10,
AI2RS 0.35
0.31
0.26
0.37
0.32
Dogpile by 2, and AltaVista,
Naver, and MSN by one
Google 0.26
0.25 0.69 0.27
0.36
participant each. Participants
MSN 0.44
0.29
0.30
0.34
0.34
would list more than one search
Yahoo 0.39
0.29
0.55
0.44
0.42
engine, which is why the total is
Average 0.36
0.28
0.45
0.35
0.36
more than 32.
table 1. Comparison of Average Precision Scores by Query and by Search Engine.

Search Engines
Queries
Difference
from Average
We see from Table 1 that the
camping
laser
average precision rating for

mexico
removal
manufactured home
techo music

the search engines across all
four domains was 0.36,
AI2RS
-2.0% 10.9%
-42.9%
5.7% -10.3%
meaning that about 36% of the
Google -28.5%
-12.6% 52.2%
-24.5%
0.7%
results were judged relevant to the
MSN 21.9%
0.8%
-32.3%
-5.1%
-5.7%
query.
Yahoo 8.6%
0.8%
23.0% 24.0%
15.3%

Average 0.0%
0.0% 0.0%
0.0%

In Table 2, we present the
table 2. Comparison of Differences of Average Precision Scores by Query and by Search Engine.
difference in average precision
It certainly appears that lack of a brand was a
positive branding awareness. This may help explain
ratings for each search engine.
detrimental factor for the AI2RS search engine, who’s
why Yahoo! has endured and prospered in a
AI2RS, the unknown brand
average precision was 10% below average. Google,
competitive marketplace where so many other search
fared the worst – with an
used most often by the study participants, after
engines (c.f., Excite, Northern Light, and Infosearch)
average precision rating of
analyzing the demographic questionnaires, had an
have come and gone.
10% under the average.
average precision just above the norm. However,
Yahoo! had the highest rating
Google was below average in three of the four
Conclusion
at 15% above average.
domains. Yahoo! performed the best with above
In this research, we investigated the effect of branding
average precision ratings across all four domains.
on the evaluation of the system performance of Web
Surprising, given the stated
search engines. Study findings show that branding as a
preference by the participants,
It appears that even though Google is the most
perception of product has a dramatic effect on user’s
Google’s rating were only
commonly used engine for searching. Yahoo! has a
evaluation of system results.
slightly better than average.
2475

---

CHI 2006 · Work-in-Progress
CHI 2007 • Work-in-Progress
April 28-May 3, 2007 • San Jose, CA, USA


0.80
0.69
0.70
n
i
o
0.60
0.55
at
u
al
v
0.50
0.45
The overall average
Of all the search engines, Yahoo! had
0.44
0.44
for all search engines
the overall best average precision of
ce E
0.39
0.37
0.40
0.35
0.36
over all queries was
0.34
0.35
0.42. This was 15.3% better than the
van
0.31
0.29
0.28
e
0.30
approximately 0.36.
average of all four search engines.
el
0.26
0.29
0.27
0.25
0.30
0.26
e R
g
r
a
0.20
ve
A
0.10
0.00
camping mexico
laser removal
manuf actured home
techo music
Type of Que rie s
Implications
AI2RS
Google
MSN
Yahoo!
Average
The implications of these research
findings give empirical weight to the
figure 2. Graphical Comparison of Average Precision Scores by Query and by Search Engine.
notion that affective and cognitive
user perceptions affect user
Acknowledgements
[4] Jansen, B. J., A. Spink, and T. Saracevic, "Real Life,
interaction with systems and
We thank the study participants for their time. The Air
Real Users, and Real Needs: A Study and Analysis of User
interactions. Therefore, product
Queries on the Web," Information Processing &
Force Office of Scientific Research (AFOSR) funded
Management, vol. 36, pp. 207-227, 2000.
brand is an important usability
portions of this research.
[5] Marcus, A., "Branding 101," interactions vol. 11, pp. 14
variable in system design and
- 21, 2004.
evaluations. Future research involves
References
[6] Park, S., A. Harada, and H. Igarashi, "Influences of
in-depth quantitative and qualitative
[1] Eastman, C. M. and B. J. Jansen, "Coverage, Ranking,
personal preference on product usability," in Proceedings of
and Relevance: A Study of the Impact of Query Operators
analysis of experimental data, a series
Conference on Human Factors in Computing Systems: CHI
on Search Engine Results," ACM Transactions on
'06 extended abstracts on Human factors in computing
of experiment to tease apart the
Information Systems, vol. 21, pp. 383 - 411, 2003.
systems, pp. 87 - 92.
nuanced relationship between
[2] Hothkiss, G., "Eye Tracking Report: Google, MSN, and
[7] Wildemuth, B. M. and A. R. Carter, "The Perceived
perception of system performance and
Yahoo! Compared," Enquiro, Kelowna, BC, Canada 2006.
Affordances of Web Search Engines: A Comparative
[3] Jansen, B. J. and A. Spink, "How are we searching the
product brand, and how to incorporate
Analysis: SILS Technical Report 2002-02," Accessed on 1
World Wide Web? A comparison of nine search engine
January 2005 on the World Wide Web at
branding into the system design
transaction logs," Information Processing & Management,
http://ils.unc.edu/ils/research/reports/TR-2002-02.pdf.
vol. 42, pp. 248-263, 2005.
2476

---

Document Outline

• ??
• ??
• ??
• ??
• ??
• ??
• ??
• ??
• ??
• ??
• ??
• ??
• ??

---