What's Up CAPTCHA? A CAPTCHA Based On Image Orientation

What’s Up CAPTCHA?
A CAPTCHA Based On Image Orientation

Rich Gossweiler
Maryam Kamvar
Shumeet Baluja
Google, Inc.
Google, Inc.
Google, Inc.
1600 Amphitheatre Parkway
1600 Amphitheatre Parkway
1600 Amphitheatre Parkway
Mountain View, CA 94043
Mountain View, CA 94043
Mountain View, CA 94043
rcg@google.com
mkamvar@google.com
shumeet@google.com

ABSTRACT
Completely Automated Public Turing test to tell Computers and
We present a new CAPTCHA which is based on identifying an
Humans Apart. CAPTCHAs are designed to be simple problems
image’s upright orientation. This task requires analysis of the
that can be quickly solved by humans, but are difficult for
often complex contents of an image, a task which humans usually
computers to solve. Using CAPTCHAs, services can distinguish
perform well and machines generally do not. Given a large
legitimate users from computer bots while requiring minimal
repository of images, such as those from a web search result, we
effort by the human user.
use a suite of automated orientation detectors to prune those
We present a novel CAPTCHA which requires users to adjust
images that can be automatically set upright easily. We then
randomly rotated images to their upright orientation. Previous
apply a social feedback mechanism to verify that the remaining
research has shown that humans can achieve accuracy rates above
images have a human-recognizable upright orientation. The main
90% for rotating high resolution images to their upright
advantages of our CAPTCHA technique over the traditional text
orientation, and can achieve a success rate of approximately 84%
recognition techniques are that it is language-independent, does
for thumbnail images [27]. However, rotating images to their
not require text-entry (e.g. for a mobile device), and employs
upright orientation is a difficult task for computers and can only
another domain for CAPTCHA generation beyond character
be done successfully for a subset of images [15][19].
obfuscation. This CAPTCHA lends itself to rapid implementation
and has an almost limitless supply of images. We conducted
Figure 1 illustrates that some images are:
extensive experiments to measure the viability of this technique.
(A) easy for both computers and people to orient (because
the image contains a face, which can be detected and
Categories and Subject Descriptors
oriented by computers)
D.4.6 [Security and Protection]: Access Control and
(B) easy for humans to orient (because the image contains
Authentication
an object, e.g. a bird, that is easily recognized by humans)
but difficult for computers to orient (because the image
General Terms
contains multiple objects with few guidelines for meaningful
Security, Human Factors, Experimentation.
segmentation and the object in the foreground is of an
Keywords
CAPTCHA, Spam, Automated Attacks, Image Processing,
Orientation Detection, Visual Processing
1. I TRODUCTIO
With an increasing number of free services on the internet, we
find a pronounced need to protect these services from abuse.
Automated programs (often referred to as bots) have been
designed to attack a variety of services. For example, attacks are
common on free email providers to acquire accounts. Nefarious
bots use these accounts to send spam emails, to post spam and
advertisements on discussion boards, and to skew results of on-
line polls.
To thwart automated attacks, services often ask users to solve a
puzzle before being given access to a service. These puzzles, first
introduced by von Ahn et al. in 2003[2], were CAPTCHAs:

Copyright is held by the International World Wide Web Conference
Committee (IW3C2). Distribution of these papers is limited to
Figure 1: images with various orientation properties (left
classroom use, and personal use by others.
column: the image randomly rotated, right column: the
WWW 2009, April 20–24, 2009, Madrid, Spain.
image in its upright position).
ACM 978-1-60558-487-4/09/04.

---

irregular, deformable, shape).
and economic incentives for spammers to defeat CAPTCHAs, the
techniques introduced in academia to defeat CAPTCHAs are soon
(C) difficult for both people and computers to orient
likely to be in widespread use by spammers. To minimize the
(because the image is ambiguous and there is no “correct”
success of these automated methods, systems increase the noise
upright orientation)
and warping used in these CAPTCHAs. Unfortunately, this not
To obtain candidate images for our CAPTCHA system, we start
only makes it harder for computers to solve, but it also makes it
with a large repository and then remove images that a computer
difficult for people to solve – leading to higher error rates [8][9]
can successfully orient as well as those that are difficult for
and higher associated frustration levels.
humans to orient.
To address this, numerous alternate CAPTCHAs (including image
For example, all of the images returned from an image-search start
based ones) have been proposed [1][6][7][8]. In designing a new
as potential candidates for our system. We then use a suite of
CAPTCHA, the basic tenets for creating a CAPTCHA (from [10])
automated orientation detectors to remove those that can be set
should be kept in mind:
upright by a computer. We discuss the system used to
1.
Easy for most people to solve
automatically determine upright orientation in Section 2. We then
2.
Difficult for automated bots to solve
apply a social feedback mechanism to verify that the remaining
3.
Easy to generate and evaluate
images are easily oriented by humans. In order to identify images

that people cannot orient, we compute the variance of users’
It is straightforward to create a system that fulfills the first two
submitted orientations and reject images which have a high
requirements. The first requirement suggests the need for
variance. We discuss this social-feedback mechanism in detail in
usability evaluations, ensuring that people can solve the
Section 3.
CAPTCHA in a reasonable amount of time and with reasonable
Our CAPTCHA technique achieves high success rates for humans
success rates. The second requirement suggests that we test state-
and low success rates for bots, does not require text entry, and is
of-the-art automated methods against the CAPTCHA. In the
more enjoyable for the user than text-based CAPTCHAs. We
CAPTCHA proposed here, we ensure the automated methods can
discuss two user studies we have performed to demonstrate both
not be used to defeat our CAPTCHA by using them to filter
the viability and the user-experience of our system in Section 4.
images which can be automatically recognized and oriented.
In Section 5, we present directions for future study.
The third requirement is harder to fulfill; it is this requirement that
presents the greatest challenge to image-based CAPTCHA
1.1 BACKGROU D: CAPTCHAs
systems. The early success of the text-CAPTCHAs was aided by
Traditional CAPTCHAs require the user to identify a series of
the ease in which they could be generated – random sequences of
letters that may be warped or obscured by distracting backgrounds
letters could be chosen, distorted, and distracting pixels, noise,
and other noise in the image. Various amounts of warping and
colors, etc. added. Subsequent image-based CAPTCHAs were
distractions can be used; examples are shown in Figure 2.
proposed which required users to identify images with labels. The
Recently, many character recognition CAPTCHAs have been
difficulty with these systems is that they require a priori
deciphered using automated computer vision techniques. These
knowledge of the image labels. Reliable labels are not available
methods have been custom designed to remove noise and to
for most images on the web, so common techniques used to obtain
segment the images to make the characters amenable for optical-
labels included:
character recognition [3][4][5]. Because of the large pragmatic
(1) using the label assigned to an image by a search
engine,
(2) using the context of the page to determine a label,
(3) using images that were labeled when they were
encountered in a different task, or
(4) using games to extract the labels from users (such as
the ESP game [11]).
Unfortunately, many times the labels obtained by the former two
methods are often noisy and unreliable in practice because people
are needed to manually verify the labels. The latter two
approaches provide less noisy labels. However, even in the cases
in which labels can be obtained, it is necessary to be careful how
they are used. Asking the user to come up with the label may be
difficult unless many labels are assigned to each image.
Furthermore, unless exact matches are entered, similarity
distances between given and expected answers may be quite
complex to compute (for example, a number of measurements can
be used: edit distance, ad-hoc semantic distance, thesaurus
distance, word-net distance, etc.).
Figure 2: typical character recognition type CAPTCHAs (from
Google’s Gmail, Yahoo Mail, xdrive.com, forexhound.com )
Other, more interesting uses of labeled images, such as finding
sets of images with recurring themes (or images that do not belong

---

to the same set) are possible [10]. However, it is likely that when
2.1 LEAR I G IMAGE ORIE TATIO
a small set of N images is given, and the goal is to find which of
In order to identify images that are easy for computers to orient,
the N-1 images does not pertain to the same set (i.e. the
we pass the images through an automated orientation detection
anomalous CAPTCHA, as described in [10]), automated methods
system, developed by Baluja [19]. Although the particular
may be able to make significant inroads. For example, if N-1
machine learning tools and features used make this orientation-
images are of a chair in several different orientations and the
detection system distinct, the overall architecture is typical of
anomalous image is of a tree, the use of current computer-vision
many current systems.
techniques will be able to narrow down the candidates rapidly
(e.g. using local-feature detection [20] and the many variants
When the orientation detection system receives an image, it
[21]).
computes a number of simple transformations on the image,
yielding 15 single-channel images:
In the CAPTCHA we propose, we are careful not to provide the
user with a small set of images to compare. Any similarity
• 1-3: Red, Green, Blue (R,G,B) Channels.
computation must be done against the entire set of images
• 4-6: Y, I, Q (transformation of R,G,B) Channels.
possible – without any a priori filtering clues given. The success
of our CAPTCHA rests on the fact that orienting an image is an
• 7-9: Normalized version of R,G,B (linearly scaled
AI-hard problem. In the next section, we will review the many
to span 0-255).
systems that attempt to determine an image’s upright orientation.
• 10-12: Normalized versions of Y,I,Q (linearly
Although a few systems achieve success, their success is, when
scaled to span 0-255).
tested in realistic scenarios, limited to a small subset of image
• 13: Intensity (simple average of R, G, B).
types [19].
• 14: Horizontal edge image computed from
intensity.
2. DETECTI G ORIE TATIO
• 15: Vertical edge image computed from intensity.
The interest in automated orientation detection rapidly arose with

the advent of digital cameras and camera phones that did not have
For each of these single-band images, the system computes the
built-in physical orientation sensors. When images were taken,
mean and variance of the entire image as well as for square sub-
software systems needed a method to determine whether the
regions of the image. The sub-regions cover (1/2)x(1/2) to
image was portrait (upright) or landscape (horizontal). The
(1/6)x(1/6)
of
the
image
(there
are
a
total
of
problem is still relevant because of the large scale scanning and
91=1+4+9+16+25+36 squares). The mean and variance of
digitization of printed material.
vertical and horizontal slices of the image that cover 1/2 to 1/6 of
The seemingly simple task of making an image upright is quite
the image (there are a total of 20=2+3+4+5+6 vertical and 20
difficult to automate over a wide variety of photographic content.
horizontal slices) are also computed. In sum, there are 1965
There are several classes of images which can be successfully
features representing averages (15*(91+20+20)) and 1965 features
oriented by computers. Some objects, such as faces, cars,
representing variances, for a total of 3930 features. These
pedestrians, sky, grass etc. [22][23], are easily recognizable by
computers. It is important to note that computer-vision techniques
Rotated Input Images
have not yet been successful at unconstrained object detection;
therefore, it is infeasible to recognize the vast majority of objects
in typical images and use the knowledge of the object’s shape to
orient the image.
Instead of relying on object recognition, the majority of the
techniques explored for upright detection do not attempt to
understand the contents of the image. Rather, they rely on an
R
G
B
Y
I
R
G
B
Y
I
Q
A
H
assortment of high-level statistics about regions of the image
V


Q









n

v
n
n
n
n
o
n
e
o
r
e
r
(such as edges, colors, color gradients, textures), combined with a
o
o
o
o
r
o
r
r
r
r
r
m
r
a
i
z
t
m
i
m
m
m
m
g
o
c
a
statistical or machine learning approach, to categorize the image






e
n

t
a
l

E
orientation [12]. For example, many typical vacation images (such
l

E
d
g
as sunsets, beaches, etc.) have an easily recognizable pattern of
d
g
e
e
s

light and dark or consistent color patches that can be exploited to
s
Means (shown)

yield good results.
and variances of
blocks
Many images, however, are difficult for computers to orient. For
example, indoor scenes have variations in lighting sources, and
abstract and close-up images provide the greatest challenge to
both computers and people, often because no clear anchor points
or lighting sources exist.
The classes of images that are easily oriented by computers are
explicitly handled in our system. A detailed examination of a
recent machine learning approach in [19] is given below. It is
Feature vector, 1965 means and 1965 variances
incorporated in our system to ensure that the chosen images are
Figure 3: Features extracted from an image.
difficult for computers to solve.

---

‘retinal’, or localized, features have been successfully employed
when used as a CAPTCHA. The details of the image selection
for detection tasks in a variety of visual domains. Figure 3 shows
process and how the Adaboost classifiers are used are given in the
the features in detail.
next section.
The image is then rotated by a set amount, and the process
3. SELECTI G IMAGES FOR THE
repeats. Each time, the feature vector is passed through a
classifier (in this case a machine-learning based AdaBoost [25]
ROTATIO AL CAPTCHA SYSTEM
classifier that is trained to give a +1 response if the image is
As previously mentioned, a two-step process is needed to
upright and a -1 response otherwise). The classifier was
determine which images should be included in our CAPTCHA
previously trained using thousands of images for which the
system. First, in Section 3.1, we describe the automated methods
upright orientation was known (these were labeled with a +1), and
used to detect whether a candidate image should be excluded
were then rotated by random amounts (these rotations were
because it is easily oriented by a computer. In Section 3.2, we
labeled with -1). Although a description of AdaBoost and its
describe the social-feedback mechanism that can harness the
training is beyond the scope of this paper, the classifiers found by
power of users to further identify which images should be
AdaBoost are both simple to compute (are orders of magnitude
excluded from the dataset because they are too difficult for
faster than the somewhat worse-performing Support Vector
humans to orient.
Machine based classifiers for this task) and are memory efficient;
both are important considerations for deployment.
3.1 Removing Computer-Detectable Images
It is important not to simply select random images for this task.
As Figure 4 illustrates, when an image is given for classification,
There are many cues which can quickly reveal the upright
it is rotated to numerous orientations, depending on the accuracy
orientation of an image to automated systems; these images must
needed, and features are extracted from the image at each
be filtered out. For example, if typical vacation or snapshot
orientation. Each set of these features is then passed through a
photos are used, automated rotation accuracies can be in the 90%
classifier. The classifier is trained to output a real value between
range [14][15][19]. The existence of any of the cues in the
+1.0 for upright and -1.0 for not upright. The rotation with the
presented images will severely limit the effectiveness of the
maximal output (closest to +1.0) is chosen as the correct one.
approach. Three common cues are listed below:
Figure 4 shows four orientations; however, any number can be
used.
1.
Text: Usually the predominant orientation of text in an
image reveals the upright orientation of an image.
When tried on a variety of images to determine the correct upright
orientation from only the four canonical 90° rotations, the system
2.
Faces and People: Most photographs are taken with the
yielded wildly varying accuracies ranging from approximately
face(s) / people upright in the image.
90% to random at 25%, depending on the content of the image.
3.
Blue skies, green grass, and beige sand: These are all
The average performance on outdoor photographs, architecture
revealing clues, and are present in many travel/tourist
photographs and typical tourist type photographs was significantly
photographs found on the web. Extending this beyond
higher than the performance on abstract photographs, close-ups
color, in general, the sky often has few texture/edges in
and backgrounds. When an analysis of the features used to make
comparison to the ground. Additional cues found
the discriminations was done, it was found that the edge features
important in human tests include “grass”, “trees”,
play a significant role. This is important since they are not reliant
“cars”, “water” and “clouds” [27][16].
on color information – so black and white images can be captured;
albeit with less accuracy.
Ideally, we would like to use only images that do not contain any
of the elements listed above. All of the images chosen for
For our use, we use multiples of the classifiers described above.
presentation to a user were scanned automatically for faces and
180 Adaboost classifiers were trained to examine each image and
for the existence of large blocks of text. If either existed, the
determine the susceptibility of that image to automated attacks
image was no longer a candidate.1 Although accurate detectors do
not exist for all the objects of interest listed in (3) above, the types
of images containing the other objects (trees, cars, clouds) were
often outdoors and were effectively eliminated through the use of
Upright?
-0.6
the automated orientation classifiers described in Section 2.1.
If the image had neither text nor faces, it was passed through the
set of 180 AdaBoost classifiers in order to further ensure that the
Upright?
-0.2
candidate image was not too easy for automated systems. The
output of these classifiers determined if the image was accepted
Upright?
+0.7
into the final image pool. The following heuristics were used
when analyzing the 180 outputs of the classifiers:
Upright?
•
If the majority of the classifiers oriented the image
-0.7
similarly with a high confidence score, it was rejected.
The image was too easy.
Input
Rotated
Feature
Single
Input
Vector
Classifier

1 Additionally, all images were passed through an automated
Figure 4: Overview of the system in its simplest form.
adult-content filter [24]. Any image with even marginal adult-
content scores was discarded.

---

•
If the predictions of the classifiers together had too large
the automated techniques to exclude machine-recognizable
entropy, then the image was rejected. Because the
images, produces a dataset for our rotational CAPTCHA system.
classifiers are trained independently, they make
different guesses on ambiguous images. Some images
4. USER EXPERIME TS
(such as simple textures, macro images, etc.) have no
In this section, we describe two user studies. The first study was
discernible
upright
orientation
for humans or
designed to determine whether this system would result in a viable
computers. Therefore, if the entropy of guesses was
CAPTCHA system in terms of user-success rates and bot-failure
high, the image may not actually have a discernible
rates. The second study was designed to informally gauge user
correct orientation.
reactions to the system in comparison to existing CAPTCHAs.
These two heuristics attempt to find images that were not too
Since these were uncontrolled studies, we did not measure task-
easy, but yet possible to orient correctly. The goal is to be
completion times.
conservative on both ends of the spectrum; the images need to
neither be too easy nor too hard. The images were accepted when
4.1 Viability Study
no single orientation dominated the results, while ensuring that
The goal of this study was to understand if users would determine
there were still peaks in a histogram of the orientations returned.
the same upright orientation for candidate images in the rotational
There are many methods to make the selection even more
CAPTCHA system. We found that after applying a social-
amenable to people while remaining difficult for computers. It
correction heuristic (which can be applied in real time in a
has been found in [15] that the correct orientation of images of
deployed system), our CAPTCHA system meets high human-
indoor objects is more difficult than outdoor objects. This may be
success and high computer-failure standards.
due to the larger variance of lighting directionality and larger
amounts of texture throughout the image. Therefore, using a
4.1.1 Image Dataset
classifier to first select only indoor images may be useful.
The set of images used for our rotational CAPTCHA experiment
Second, due to sometimes warped objects, lack of shading and
was collected from the top 1,000 search results for popular image-
lighting cues, and often unrealistic colors, cartoons also make
queries2. We rejected from the dataset any image which could be
ideal candidates. Automated classifiers to determine whether an
machine-recognizable, according to the process described in
image is a cartoon also exist [26] and may be useful here to scan
Section 3. From the remaining candidate images, we selected a set
the web for such images. Finally, although we did not alter the
of approximately 500 images to be the final dataset which we used
content of the image, it may be possible to simply alter the color-
in our study. This ensures that our dataset meets the two
mapping, overall lighting curves, and hue/saturation levels to
requirements laid forth by [2]:
reveal images that appear unnatural but remain recognizable to
people.
•
First, that this CAPTCHA does not base its security in
the secrecy of a database. The set of images used is the
3.2 Removing Images Difficult for Humans to
set of images on the WWW, and is thus is non-secretive.
Further, it is possible to alter the images to produce
Orient
ones that can be made arbitrarily more difficult.
Once we have pruned from our data set images that a computer
can successfully orient, we identify images that are too difficult
•
Second, that there is an automated way to generate
problem instances, along with their solution. We
for a human to successfully rotate upright. To do this, we present
generate the problem instances by issuing an image
several randomly rotated images to the user in the deployed
search query; their solution (the image’s orientation)
system. One of the images presented is a “new” candidate image
defaults to the posted orientation of the image on the
being considered to join the pool of valid images. As large
web, but may be changed to incorporate the corrective
numbers of users rotate the new image we examine the average
offset found by the social-feedback mechanism.
and standard deviation of the human orientations.
To normalize the shape and size of the images, we scaled each
We identify images that are difficult to rotate upright by analyzing
image to a 180x180 pixel square and we then applied a circular
the angle which multiple users submitted as upright for a given
mask to remove the image corners.
image. Images that have a high variation in their submitted
orientations are those that are likely to have no clear upright
4.1.2 Experiment Setup
orientation. Based on this simple analysis from users, we can
500 users were recruited through Google-internal company email
identify and exclude difficult images from our dataset.
groups used for miscellaneous communications. The users came
This social feedback mechanism also has the added advantage of
from a wide cross-section of the company, and included
being able to “correct” images whose default orientation is not
engineers, sales associates, administrative assistants and product
originally upright – for example images where the photographer
managers. Users participated in the study from their own
may not have held the camera exactly upright. Though the
computer and were not compensated for their participation. Since
variance of the submitted orientation across users may be small,
this study was done remotely at the participant’s computer there
the average orientation may be different than the image’s posted
was no human moderator present. Participants received an email
orientation. Users will correct this image to its natural upright
position, compensating for the angle of the original image.

This social mechanism allows us to consistently correct or reject
2 Image queries are those which return a list of images, rather than
the images used in the CAPTCHA, which when combined with
a list of website URLs. For example, any query issued on
images.google.com would be considered an image query.

---

with a link to the experiment website which included a brief
introduction to the study:
“This experiment will present a series of
images one at a time. Each image will be
rotated to a random angle. Use the provided
slider to rotate the image until you believe it
is in its natural, upright position, then press
submit to go to the next image. This process
will continue until you have adjusted ten
images.”
Figure 5 shows a screenshot of an example trial in the viability
study written using cross-browser JavaScript and DHTML.
Figure 5: Screenshot of an example trial in the viability study
Figure 6: The first six images displayed to users


Each user was asked to rotate 10 images to their natural upright
Figure 7 illustrates the average and standard deviation of users’
position. The first six images and their offset angles were the same
final rotation angles for the first six images (the images which
for each user. We kept these trials constant to ensure that we
were shown to all of the users). There are some images for which
would have a significant sample size for some of the problem
users rotate very accurately (images 1, 5 and 6), and those which
instances. Figure 6 shows the first six images at the orientation
users do not seem to rotate accurately (images 2, 3 and 4). The
that they were shown to the users. The last four images and their
images which have poor results can be attributed to by three
offset angles were randomly selected at runtime. We did this to
factors, each of which can be addressed by our social feedback
evaluate our technique on a wide variety of images. For each trial,
mechanism:
we recorded the image-ID, the image’s offset angle (a number
between ±180 which indicated the position the image was
presented to the user), and the user’s final rotation angle (a
number between ±180 which indicated the angle at which the user
submitted the image).
4.1.3 Results
We have created a system that has sufficiently high human-
success rates and sufficiently low computer-success rates. When
using three images, the rotational CAPTCHA system results in an
84% human success metric, and a .009% bot-success metric
(assuming random guessing). These metrics are based on two
variables: the number of images we require a user to rotate and the
size of the acceptable error window (the degrees from upright
which we still consider to be upright). Predictably, as the number
Figure 7: average degrees from the original orientation that
of images shown becomes greater, the probability of correctly
each image was rotated.
solving them decreases. However, as the error window increases,

the probability of correctly solving them increases. The system
1. Some images are simply difficult to determine which way is
which results in an 84% human success rate and .009% bot
upright. Figure 8 shows one such image and plots the absolute
success rate asks the user to rotate three images, each within 16°
number of degrees-from-upright which each user rotated the
of upright (8-degrees on either side of upright).
image. Based on the standard deviation in responses, this image
is not a good candidate for social correction. We see that its

---

standard deviation was greater than the half of the error window;
it was deemed not to have an identifiable upright position, and
was rejected from the dataset.
image # 1
180
170
160
150
140
130
120
110
100
90
80
Figure 10: Two possible natural orientations of the image.
70
60
50
40

30
It is important to note that the decisions about whether an image
20
10
falls into one of the above categories can be made in real time by
0
a system that presents a user a “candidate” image in addition to
0
100
200
300
400
500
the CAPTCHA images. The “candidate” image need not be used
Figure 8: An image with large distribution of orientations.
to influence the user’s success at solving the CAPTCHA, but is

simply used to gather information. The user is not informed of
2. Some images’ default upright orientation may not correspond
which image is a candidate image.
to the users’ view of their natural upright orientation. We
designate the default upright orientation as the angle at which the
In our analysis, the human success rate is determined by the
image was taken originally. This is illustrated in the picture of
average probability that a user can rotate an image correctly.
the toy car (image #3). Figure 9 shows the original orientation of
However, we exclude any images which fall into case 1 or case 3
the image, in contrast to the orientation of the image which most
outlined above. Those images would be identified and
users thought was “natural”, shown in the graph. Based on the
subsequently rejected from the dataset by our social-feedback
low deviation in responses, this image is a good candidate for
mechanism. If an image falls in case 2, we corrected the upright
being “socially corrected”. If this image was used after the
orientation based on the mode of the users’ final rotation, as this
social correction phase, the “upright” orientation would be
could be similarly determined by the correction aspect of our
changed to approximately 60° from the shown orientation.
social-feedback mechanism.
Human success rates are influenced by two factors: the size of the
error window and the number of images needed to rotate. Table 1
image # 3
shows the effect on human success, as the size of the error
180
window and number of images we require a user to successfully
170
orient vary. The configurations which have a success rate of
160
150
greater than 80% are highlighted in green.
140
130

120
110
Table 1: Human-success rates (%), as number of images shown and
100
size of acceptable error window varies.
90
80
70
60
50
40
30
20
10
0
0
100
200
300
400
500

Figure 9: An image requiring social correction.


3. Some images have multiple “natural” upright positions. Figure

10 shows various orientations of the guitar image which could be
The viability of a CAPTCHA is not only dependent on how easy
considered upright. In our analysis we rejected any image which
it is to solve by humans, but it is also dependent on how difficult
had multiple upright orientations (indicated by a large standard
it is to solve by computers. Computer success rate is the
deviation in image rotation results). However, future versions of
probability that a machine can solve the CAPTCHA. No
this CAPTCHA system may choose to allow for multiple
algorithm has yet been developed to successfully rotate the set of
orientations, if there is a multi-modal clustering around a small
images used in our CAPTCHA system. A first pass at estimating
number of orientations.

---

a computer’s solution would be a random guess. Since our images
have 360 degrees of freedom for rotation, computers would have a
1/360 chance at guessing the exact upright orientation (to within
1°). The computer success rate of our CAPTCHA is based on two
factors: the window of error we would allow people to make when
rotating the image upright, and the number of images that they
would need to rotate. For example, if we allowed users to rotate
images in a 6-degree window (3° on either side of upright) the
machine success rate would be 6/360. If users were required to
rotate 3 images to their upright position, the computer success rate
would be decreased to (6/360)3. A CAPTCHA system which
displayed ≥ 3 images with a ≤ 16-degree error window would
achieve a guess success rate of less than 1 in 10,000, a standard
A.
acceptable computer success rates for CAPTCHAs [8]. It should
be noted, however, that these estimates are far too optimistic –
intelligent orientation detection systems will be able to assign
probabilities of upright orientations; thereby making more
intelligent, although still perhaps imperfect, guesses. The caveats
for intelligent guessing are also equally applicable to text-based
CAPTCHA systems; for each character to be recognized, large
numbers of incorrect characters can be reliably withdrawn from
consideration. Therefore, in a manner similar to increasing the
difficulty
of
text-based
CAPTCHAs,
we
can
simply
B.
modify/perturb/degrade the image, or increase the number of
images to de-rotate.
In its simplest instantiation, we used an image-based CAPTCHA
system that requires a user to rotate at least three images upright
with a 16 degree error window (8-degrees on either side of
upright). In order to generate data for the social-correction system,
an additional image (or multiple images), the “candidate images”,
can be shown with the required CAPTCHA images.
4.2 Happiness Study
The goal of this study was to informally determine what type of
CAPTCHA users preferred to use.
C.
4.2.1 Experiment Setup
Figure 11: Snapshots of the user-happiness experiment.
Sixteen users were recruited to participate in the study through an
11a) One trial in the “rotate image” task.
email to an internal Google company email group and were
11b) One trial in the “decipher text” task.
compensated for their time3. The users were selected from a
11c) The side-by-side comparison presented to users after they had
cross-section of departments within Google: the users consisted of
completed the “rotate image” and “decipher text” tasks.
10 sales representatives and six employees from other departments
including Engineering, Human Resources, Operations, and
needed for an effective system. The 15 images that users rotated
Finance. All users participated in the study from a Firefox
were randomly selected from the web and processed as described
Browser on a desktop computer located in a usability lab on the
earlier, and we generated a random angle to offset each image.
Google Campus.
These 15 images were always presented in the same order to
users. Users were instructed to press the submit button after
This study asked users to do two tasks: to rotate a set of images
adjusting the images in each trial to their upright orientation.
into their natural, upright positions (Figure 11A) and to type
distorted text into a textbox (Figure 11B). Each task, “rotate
Before the “decipher text” task (Figure 11B), users were told we
image” and “decipher text”, had five trials, and the order of these
were measuring their ability to accurately read distorted text. Each
tasks was counterbalanced across users.
trial consisted of users deciphering text consisting of six letters.
We randomly chose five CAPTCHAs from Yahoo’s CAPTCHA
Before the “rotate image” task (Figure 11A), users were told we
base and presented them to users in the same order. Users were
were measuring their ability to accurately rotate an image to its
instructed to enter the text they believed to be in the image and
natural, upright position. Each trial consisted of three images
press the submit button.
shown to a user on the page. We chose three images because our
previous study indicated that at least three images would be
After users completed these 10 tasks, they were presented with a
side-by-side comparison of the “rotate image” and “decipher text”

(Figure 11 C).
3 Users were compensated with their choice of a $15 Amazon.com gift
certificate, a $15 iTunes gift certificate, or a 30 minute massage coupon
for participating in the study.

---

4.2.2 Results
68.75% of users (11 users) preferred rotating images, and 31.25%
of users (5 users) preferred deciphering text.
Among the comments from users who preferred the rotational
approach indicated they thought that method was “easy”, “cool”,
“fun” and “faster”. One user stated that he preferred “visual cues
over text”, and many users referenced feeling like they were at an
eye exam while deciphering the text.
Only two of the five users who stated their preference as
“deciphering text” provided insight to their choice. One user
pointed to an implementation flaw (that the slider should retain
focus even when the mouse left its bounding box) as the reason he
did not like the rotational approach, while the other user pointed
Figure 12: Example rotating an image on a mobile device.
to familiarity with the text CAPTCHA, and more absolute input
mechanism as the rationale for her preference. “I prefer
[deciphering text] since it requires simple keyboard inputs which
are absolute. With rotating pictures I found myself continually
Finally, another interesting aspect to this system is related to
making fine adjustments to make them perfectly upright, therefore
adoption and user perception. Most CAPTCHAs are viewed as
taking a slight bit longer to accomplish. Also, I’m much more
intrusive and annoying. To alleviate user dissatisfaction with
familiar with [deciphering text] since it’s what most internet
them, we can use images that keep the user within the overall
portals use for security purposes.”
experience of the website. For example, on a Disney sign-up
page, Disney characters, movie stills, or cartoon sketches can be
From these two studies, we conclude that not only is the rotational
used as the images to rotate; eBay could use images of objects that
task a viable one, but compared to the standard deciphering text,
are for sale; a Baseball Fantasy Group site could use baseball-
users may prefer it.
related items when creating a user account.
5. VARIATIO S A D FUTURE WORK
6. CO CLUSIO S
There are a number of interesting extensions to this CAPTCHA
We have presented a novel CAPTCHA system that requires users
system that we can investigate and deploy. The fundamental
to adjust randomly rotated images to their upright orientation.
system presents n images, of which m of the images are new
This is a task that will be familiar to many people given the use of
candidates (m < n). In alternative implementations, we can also
early digital cameras, cell phones with cameras, and even the
present n images and ask the user to select p of them to rotate.
simple act of sorting through physical photographs. We have
We suspect that there will be useful trends from this system;
preliminary evidence that shows users prefer rotating images to
difficult images will be chosen less frequently than other images.
deciphering text as is required in traditional text based
This gives us further evidence to identify images to exclude from
CAPTCHAs. Our system further improves traditional text-based
our CAPTCHA system.
CAPTCHAs in that it is language and written-script independent,
and supports keyboard-difficult environments.
There are numerous sources for candidate objects to rotate.
Beyond images, we can also introduce views from 3D models (or
It is important that random images are not chosen for this task;
with advanced graphics capabilities, users can interact with the
they must be carefully selected. Many typical vacation and
3D models themselves). These models, being more austere, can
snapshots contain cues revealing upright orientation. We ensure
remove many of the features such as lighting and horizons on
that our CAPTCHA can not be defeated by state-of-the-art
which automated orientation mechanisms rely. Styles can be
orientation detection systems by using those systems to filter
applied to remove strong edges. For example, we could use the
images that can be automatically recognized and oriented. In
Golden Gate bridge model without lighting effects and without
contrast to traditional text based CAPTCHAs which introduce
the sky/ocean horizon. It also gives us another dimension of
more noise and distortion as automated character recognition
rotation, greatly increasing the number of possible answers,
improves, we currently do not need to alter or distort the content
making it even harder for computers to randomly guess the correct
of the images. As advances are made in orientation detection
angle. Furthermore, the difficulty in these tasks can be
system, these advances will be incorporated in our filters so that
parameterized.
those images that can be automatically oriented are not presented
to the user. The use of distortions may eventually be required.
In our experiments, users moved a slider to rotate the image to its
upright position. On small display devices such as a mobile
Some of the major pitfalls associated with other proposed image-
phone, they could directly manipulate the image using a touch
based CAPTCHA systems do not apply to our CAPTCHA system.
screen, as seen in Figure 12, or can rotate it via button presses.
A priori knowledge of the image’s label is not needed, which
This may be particularly useful in cases in which there is no
makes examples for our system easier to automatically generate
character keyboard or where keyboard entry is error prone. User
than other image-based CAPTCHA systems. Furthermore, it is
interfaces that cycle through, scale, or otherwise engage the user
harder for bots to solve than the image-based CAPTCHAs that
based on the constraints of the display and input capabilities can
require a user to identify a common theme across a set of images,
be developed, measured, and compared for utility.
since the set of images to compare against is not closed.

---

Finally, our system provides opportunities for a number of
[11] Ahn, L.V., Dabbish, L. (2004) Labeling Images with a
interesting extensions. First, the set of images selected can be
Computer Game, CHI-2004.
chosen to be more interesting or valuable to the end-user by
[12] Vailaya, A., Zhang, H., Yang, C., Liu, F., Jain, A. (2002)
displaying those that are related to the overall theme of the
“Automatic Image Orientation Detection”, IEEE Transactions on
website. Second, more aggressive social-correction can be used
Image Processing. 11,7.
through the presentation of multiple images of which only a few
must be uprighted; this gives real, and immediate, insight into
[13] Wang, Y. & Zhang, H. (2001), “Content-Based Image
which images may be too hard for users. Third, the large number
Orientation Detection with Support Vector Machines” in IEEE
of 3D models being created for independent applications, such as
Workshop on Content-Based Access of Image and Video
Google’s Sketch-Up, can be used as sources of new images as
Libraries. pp 17-23.
well as full-object rotations.
[14] Wang, Y., & Zhang, H. (2004) “Detecting Image Orientation
based on low level visual content” Computer Vision and Image

Understanding, 2004.
7. ACK OWLEDGME TS
[15] Zhang, L, Li, M., Zhang, H (2002) “Boosting Image Orientation
Detection with Indoor vs. Outdoor Classification”, Workshop on
Many thanks are extended to Henry Rowley and Ranjith
Application of Computer Vision, 2002.
Unnikrishnan for the text-identification system. Thanks are also
given to Kaari Flagstad Baluja for her valuable comments.
[16] Luo, J. & Boutell, M. (2005) A probabilistic approach to image
orientation detection via confidence-based integration of low

level and semantic cues, IEEE Transactions on Pattern Analysis
and Machine Intelligence, v27,5 pp.715-726.
8. REFERE CES
[17] Lyu, S. (2005) Automatic Image Orientation Determination

with Natural Image Statistics, Proceedings of the 13th annual
[1] Shahreza, A., Shahreza, S. (2008) “Advanced Collage
ACM international conference on Multimedia, pp 491-494
CAPTCHA”, Fifth International Conference on Information
Technology, 1234- 1235
[18] Wang, L., Liu, X., Xia, L, Xu, G., Bruckstein, A., (2003)
“Image Orientation Detection with Integrated Human Perception
[2] von Ahn, L., Blum, M., Hopper, N. and Langford, J.
Cues (or which way is up)”, ICIP-2003.
CAPTCHA: Using Hard AI Problems for Security. Advances in
Cryptology, Eurocrypt 2003. Pages 294-311.
[19] Baluja, S. (2007) Automated image-orientation detection: a
scalable boosting approach, Pattern Analysis & Applications,
[3] Huang, S.Y., Lee, Y.K., Bell, G. Ou, Z.h. (2008) “A Projection-
V10, 3.
based Segmentation Algorithm for Breaking MSN and YAHOO
CAPTCHAs”, The 2008 International Conference of Signal and
[20] Lowe, D.G., Distinctive Image Features from Scale-Invariant
Image Engineering
Keypoints, International Journal of Computer Vision, v.60 n.2,
p.91-110, November 2004
[4] Chellapilla K., Simard, P. “Using Machine Learning to Break
Visual Human Interaction Proofs (HIPs),” in L. K. Saul, Y.
[21] Tuytelaars, Tinne, Mikolajczyk, K., A Survey on Local Invariant
Weiss, and L. Bottou, editors, Advances in Neural Information
Features, preprint, Foundations and Trends in Computer
Processing Systems 17, pp. 265–272. MIT Press
Graphics and Vision 1:1, 1-106.
[5] Mori, G., Malik, J. (2003) “Recognizing Objects in Adversarial
[22] Yang, M.H., Kriegman, D.J., Ahuja, N. (2002) “Detecting Faces
Clutter: Breaking a Visual CAPTCHA”, in Computer Vision and
in Images”, IEEE-PAMI 24:1
Pattern Recognition (CVPR-2003).
[23] Bileschi, S., Leung, B. Rifkin, R., Towards Component-based
[6] Elson, J., Douceur, J. Howell, J., Saul, J., (2007) Asirra: A
Car Detection, 2004 ECCV Workshop on Statistical Learning
CAPTCHA that Exploits Interest-Aligned Manual Image
and Computer Vision.
Categorization, in Proceedings of the 14th ACM conference on
[24] Rowley, H., Jing, Y., Baluja, S. (2006), Large-Scale Image-
Computer and communications security.
Based Adult-Content Filtering, International Conference on
[7] Golle, P. (2008) Machine Learning Attacks against the Asirra
Computer Vision Theory and Applications.
CAPTCHA, to appear in in Proceedings of the 15th ACM
[25] Freund, Y., R. Schapire, “Experiments with a New Boosting
conference on Computer and communications security.
Algorithm” (1996), in Machine Learning, Proceedings of the
[8] Chellapilla, K., Larson, K., Simard, P., Czerwinski, M.,
Thirteenth International Conference – 1996.
Designing Human Friendly Human Interaction Proofs (HIPs),
[26] Lienhart, R., Hartmann, A. (2002) Classifying images on the
CHI-2005.
web automatically, J. Electron. Imaging Vol 11, 445
[9] Yan, J., Ahmad, A.S.E., (2008) Usability of CAPTCHAs Or
[27] Luo, J., Crandall, D., Singhal, A., Boutell, M. Gray,R.,
Usability issue in CAPTCHA design. In Symposium on Usable
“Psychophysical Study of Image Orientation Perception”,
Privacy and Security (SOUPS) 2008.
Spatial Vision, V16:5.
[10] Chew, M., Tygar, D. (2004) Image Recognition CAPTCHAs, in
Proceedings of the 7th International Information Security
Conference (ISC 2004)

---