A Multimodal Communication with A Haptic Glove: On the
Fusion of Speech and Deixis Over a Raised Line Drawing
∗
†
Francisco Oliveira
Francis Quek
Virginia Tech
Virginia Tech
2202 Kraft Drive
2202 Kraft - 1130 KWII
Blacksburg, Virginia, USA
Blacksburg, Virginia, USA
oliveira@cs.vt.edu
quek@cs.v.edu
Categories and Subject Descriptors
H.5 [Information Interfaces and Presentation]: Haptic
ABSTRACT
I/O
Mathematics instruction and discourse typically involve two
modes of communication: speech and graphical presenta-
1.
INTRODUCTION
tion. For the communication to remain situated, dynamic
According to the American Foundation for the Blind, 45%
synchrony must be maintained between the speech and dy-
of individuals with severe visual impairment or blindness
namic focus in the graphics. In normals, vision is used for
have a high school diploma [5]. Around 80% of sighted stu-
two purposes: access to graphical material and awareness of
dents receive the degree. The National Federation of the
embodied behavior. This embodiment awareness keeps com-
Blind (NFB) estimates that in the United States alone, more
munication situated with visual material and speech. Our
than 93,000 children on school age are blind [19]. Further-
goal is to assist blind students in the access to such instruc-
more, blind students are typically one to three years behind
tion/communication. We employ the typical approach of
their seeing counterparts [27].
sensory replacement for the missing visual sense.
Haptic
There is evidence that blind students are capable of spa-
fingertip reading can replace visual material. For the em-
tial reasoning, and by extension, mathematical reasoning
bodied portion of the communication, we want to make the
[15, 18, 8, 14]. Assuming that the environmental conditions
blind student aware of the deictic gestures performed by
which affect learning (e.g. family conditions) are equal, the
the teacher over the graphic in conjunction with speech. We
reason for this deficit must lie either in the paucity of learn-
propose the use of haptic gloves paired with computer vision
ing material or in the communication of that material to
based tracking to help blind students maintain reading focus
students who are blind.
A variety of devices to produce
on a raised line representation of a graphical presentation to
raised line drawings rapidly are available. While there is
which the instructor points while speaking. In this initial
room for improvement in the development of better tactile
phase of our research, we conducted three experiments that
display devices, this paper explores the understudied area of
show that: 1) The gloves convey sense of direction; 2) The
improving mathematics instruction by aiding the communi-
gloves do not interfere in fingertip reading; 3) A person can
cation between teacher and student.
navigate with the help of this system while listening to a
Human to human communication has to be ‘situated’. In
story; 4) It is possible to fuse the information received from
brief, the discourse must maintain common ground between
both modes. We discuss these findings in this paper.
the communicants [3]. Since mathematical instruction typ-
ically involves a spatial/graphical component, this common
ground must be maintained in the communication with re-
General Terms
spect to both speech and the graphical material. In commu-
Multimodal, awareness, embodiment, gesture, sensory re-
nication between sighted people in the presence of an object
placement
of discussion, the fusion of gestural information (e.g. point-
ing), speech (what is said), and the object referenced (the
∗Francisco Oliveira is a PhD candidate at Virginia Tech
deictic field) are required to situate the discourse and facili-
†Francis Quek is the Director of the Center for Human Com-
tate comprehension [17, 2, 9]. For example, in a typical lec-
puter Interaction at Virginia Tech
ture setting, seeing students can fuse the information about
the presentation graphic and the speech of the teacher, us-
ing the embodied behavior of the teacher to fuse the salient
portion of the graphic with the co-temporal speech.
Permission to make digital or hard copies of all or part of this work for
For students who are blind, the graphical material may be
personal or classroom use is granted without fee provided that copies are
replaced with raised line material, but access to the gestu-
not made or distributed for profit or commercial advantage and that copies
ral/deictic behavior of the teacher is an open problem. For
bear this notice and the full citation on the first page. To copy otherwise, to
example, a thorough analysis of the techniques and equip-
republish, to post on servers or to redistribute to lists, requires prior specific
ment available to enhance learning in a lecture setting was
permission and/or a fee.
done by Supalo [24], a chemistry graduate student at Penn-
PETRA ’08 Athens, Greece
Copyright 200X ACM X-XXXXX-XX-X/XX/XX ...$5.00.
sylvania State University, who is blind. The problem with

---

the equipment used by Supalo and others in their experi-
ments is that none of them raises the student’s awareness of
the teacher’s behavior. The teacher-student communication,
a critical link in the learning environment, was left out.
In this paper, we present a haptic glove system designed to
guide the hand of the student who is blind over a raised line
version of the graphical presentation used by the teacher.
Our approach is based on sensory replacement, where tactile
exploration of the document substitutes visual access. Using
this strategy, the student should be able to read the tactile
material as the teacher references its contents during the
Figure 1: One glove model and the controller
lecture.
Following on our multimodal fusion model, we test our
system for three levels of information fusion. First, we test
if subjects are able to track a teacher’s pointing direction-
position with our device. Second, we test if subjects are able
to fuse the directional information with fingertip reading of
raised line media (or if there is interference between our
haptic device and the tactile reading), and finally, we test if
subjects are able to fuse co-temporal speech with the haptic
device while reading raised line media. We present promis-
ing results of our experiments that suggest that continued
research in instructional conditions is warranted.
Figure 2: A possible way to convey North using 5 x
4 glove
2.
APPROACH
We enhance teacher/student communication by giving the
blind student access to the pointing behavior of the teacher
the pocket produces the sensation of vibration. We used
by means of a tactile cut-off glove (leaving the fingertips free
thinnest and the most stretchable fabric we could find to
for reading). The glove has a set of vibrating motors that
ensure close contact between the user’s palm and the actu-
can be activated in various patterns. The system tracks the
ator.
teacher’s pointing hand on a presentation graphic (e.g. a
The rectangular glove (configuration A) has twenty actua-
poster or projection) and guides the student to the corre-
tors arranged in a 5x4 grid (shown in Figure 1). The square
sponding position on a raised-line graphic to which she has
one (configuration B) is assembled in a 4x4 grid. Finally, the
access. We will discuss the system and the experiments we
round glove (configuration C) has its 12 motors mounted as
conducted.
the hours of an analog clock.
3.1.1
The Directions
3.
THE SYSTEM
The goal of our research is to provide students who are
blind with access to the teacher’s deictic gestures. Given the
3.1
The Haptic Gloves
speed of hand movements, it is critical to convey informa-
We built and tested three different glove models. Each
tion about the location of the deictic focus as fast as possible.
glove contains a set of vibrating motors arranged in a spe-
The gloves have low resolution; the maximum number of ac-
cific configuration. Configuration A is rectangular, B is a
tuators on a glove is twenty (configuration A). Hence, rapid
square, and configuration C employs a circle of vibrators. A
signaling is important.
controller box with a PIC18F452-I/P and a set of driving
To convey the location of the deictic focus, the gloves
transistors control the firing patterns of these motors. The
provide eight directional signals: North, Northeast, East,
glove is connected to the controller box via serial cable. Fig-
Southeast, South, Southwest, West and Northwest. These
ure 1 shows the controller box with the rectangular glove.
directions are presented through the patterns of vibration of
The controller box is, in turn, connected to a computer also
the glove. In our design, these vibration patterns are data-
via serial cable. The program on the microcontroller con-
driven. The pattern is stored as a list of 3-tuples. Each 3-
trols the timing and intensity of the vibration of each indi-
tuple represents an ‘activation command’ by specifying the
vidual motors on the gloves. Each motor or actuator has a
actuator ID, the intensity of vibration, and the time delay
unique address and its vibration intensity has sixteen differ-
before performing the next command. A time delay of zero
ent levels. We selected the PIC18F452-I/P because it has
milliseconds represents simultaneous activation. For exam-
4 independent 8-pin output control registers for simultane-
ple “5 10 0, 7 10 30, 3 8 0, 4 8 10, 5 0 0, 7 0 30, 3 0 0, 4
ous control of up to 32 devices. We employ the pulse width
0 0” tells a glove to fire up actuators 5 and 7 to intensity
control mechanism of the microcontroller to produce a sense
10 simultaneously, wait 30 msec and then fire up actuators
of varying intensity. The microcontroller can set the motor
3 and 4 at intensity 8, wait 10 msec and stop actuators 5
to vibrate at its highest intensity simply by providing the
and 7, and wait 30 msec then stop actuators 3 and 4. These
motor current long enough so that it reaches full spinning
patterns are stored in plain text files. Each glove has its own
speed. Setting vibration intensity to zero brings a motor to
collection of pattern files. This data-driven strategy allows
a halt. The motors are assembled in copper tubes, which
us to change the way a direction is signaled without chang-
are put into the glove’s small pockets. The spinning within
ing the source code. Furthermore, we can easily add new

---

glove models.
We tried several different patterns for each direction on
each glove to find one that produces a strong, clear and short
signal. We wanted to increase signal salience, and therefore
reduce memory load [26]. The lesser memory load the better
our chances to enable multimodal interaction.
During pilot studies, we first tested a “wave pattern”. In
the 5 x 4 glove, for example, “North” was conveyed as vi-
Figure 3: Focal Disparity (direction) computed from
brating actuators 18, 13, 8, and 3 sequentially at intensity
TPA and PIF
10 for 30 msec each, with a short 5 msec pause in between.
This is specified in our data file as “18 10 30, 18 0 5, 13 10
we used a 6-level processing pyramid. To ensure that inter-
30, 13 0 5, 8 10 30, 8 0 5, 3 10 30, 3 0 0” to produce the
action issues are not confounded with tracking errors, we
virtual upward motion shown in Figure 2.
asked the subjects to wear a blue tape on top of their finger
With the wave pattern, the direction was clearly per-
and the PIFs were painted green on a black sheet of paper
ceived. The pattern resembled those lights in highways indi-
( Figure 3).
cating detours. However, this approach costs some precious
We created an imaginary circle with 10 pixels of radius
milliseconds to completely send a directional signal. The
with origin at the PIF’s coordinates. We considered that
start-stop interval between the vibrations has to be long
participant had reached the PIF when he entered this circle.
enough to allow the sequencing to be perceived. This “wave”
pattern, although very easily perceived, was abandoned be-
4.
THE EXPERIMENTS
cause of the long duration of the signal.
We devised three experiments. Each one was designed
To keep the signal short, we decided to pick the smallest
to answer one specific question.
Twenty-five members of
number of actuators necessary to convey a direction and vi-
the Virginia Tech community participated (19 males and 6
brate them all together at the highest intensity possible. For
females).
Their age ranged from 21 to 52 years old and
this, we chose the actuator or actuators with most signifi-
averaged 29.35. One participant was not able to complete
cance for a given direction. The most significant actuator
experiments two and three, but we did collect data from her
for the North direction would be the one with address three
trial at experiment one.
in figure 1. South would be actuator 18. For Northwest, we
One important note is that none of the participants were
used actuator 1. For East and West directions, we chose to
blind. It is believed that blindness is associated with su-
vibrate actuators 6, 11 and 10, 15, respectively.
perior non visual perception. It is consensus that any ad-
As for the vibration duration, it was empirically set to 30
vantage of the blind is due not to heightened sensitivity,
milliseconds. Less than that, the motors will not spin at full
but rather to the development and refining of perceptual
speed, making the vibration less perceivable.
skills with practice [10]. The basis for such practice-related
perceptual improvement is the remarkable plasticity of the
3.1.2
Tracking
Central Nervous System [22]. Pascual-Leone and Torres [20]
We used real time computer vision based tracking. In the
used somatosensory evoked potentials and transcranial mag-
setup, we have a downward looking camera that tracks the
netic stimulation (TMS) to demonstrate that the Braille-
student’s hand movements. The goal here is to get the x,y
reading finger has an expanded representation in sensorimo-
position of the participant’s “reading fingertip”, which in the
tor cortex of blind Braille readers. Van Boven [25] et al
future versions of this system will be the student’s Tactile
report that grating resolution at the fingertip is nearly 30%
Point of Access - TPA [21]. The camera feeds the system
percent better in blind Braille readers than controls with
at thirty frames per second. For the experiments described
normal vision, and is also better by a similar magnitude on
here, we had two fixed targets or Points of Instruction Focus
the Braille-reading finger relative to other fingers of sub-
- PIFs [21]. The PIFs had fixed coordinates. For each frame
jects who are blind. This study used gratings consisting of
received, the system calculates the direction from the cur-
alternating ridges and grooves that were impressed into the
rent TPA to the next PIF. Once the direction is found, we
fingerpad oriented either along or across the long axis of the
convey it through the glove. To keep tracking and signaling
finger.
as independent as possible from each other, they were imple-
Blind Braille readers can identify Braille-like dot-patterns
mented into two different computational threads. It might
almost 5% more accurately than sighted subjects [6]. Steven
occur that the TPA changes its position while a signal is still
et al [23] show that the blind can detect gaps that are over
being sent. This could occur due to quick hand movement
15% narrower, distinguish the orientation of lines that are
or if the signal is too lengthy. If this happens, the system
nearly 40% shorter, and discriminate dot-patterns bearing
aborts that signaling and starts sending the new direction.
minute spatial offsets in the hyperacuity range (i.e., below
This dynamic tracking approach avoids the “piling up” of
the limits of spatial resolution) with about 50% lower thresh-
old directions that make no sense.
olds. With practice, however, sighted subjects can match
For the experiments, we used a 2.33 GHz Intel Core 2 Duo
the tactile performance of the blind [7].
Mac Book Pro with 2 GB RAM to which the firewire camera
We decided to use normals for this phase of our research
and the controller box for the gloves were attached.
and not overuse our limited number of students who are
For the tracking, we employed the Lucas-Kanade tracking
blind. This permits us first to establish a baseline of the
algorithm [1, 16] (we used the OpenCV [11] implementa-
efficacy of the devices over a larger population. We expect
tion). The frames captured had 640x480 pixels, the camera
that the blind subjects would perform better in the tasks.
was set at approximately 1.5m above the desk. For efficiency,
We also understand that the findings described on this paper

---

Table 2: Confidence Interval for hit percentages
Glove
Trials
Mean
Std
Lower
Upper
Model
%
Error
95%
95%
4x4
9
79.16%
5.50
67.75
90.57
5x4
8
78.12%
5.83
66.02
90.23
Round
8
39.06%
5.83
26.95
51.16
Figure 4: Experiment I - A participant choosing the
The participants filled out a questionnaire after they fin-
direction he perceived
ished the experiment. In one of the questions, we asked them
to grade how well they distinguished the directions. We call
Table 1: Response time in seconds per glove model
this number the level of reported perception - LRP. The mean
Glove
Trials
Mean
Std
Lower
Upper
response time among the participants that reported higher
Model
in Sec
Err
95%
95%
LRP was 0.3312 seconds, with f: 3.1868 and p<0.0258, thus
in Sec
in Sec
significantly better. We could not find any significant differ-
4x4
55
0.592
0.047
0.499
0.686
ence between wearing the glove on the dominant hand and
5x4
48
0.553
0.050
0.453
0.653
the non-dominant hand.
Round
40
0.544
0.055
0.434
0.653
Perception results.
We ran one-way Anova on the data collected. Figure 5a
shows the results for the gloves in terms of correctly per-
can be applied to the blind.
ceived directions per glove model (hits). The x-axis shows
We tested the three models of glove on each hand (dom-
the glove models. The y-axis shows hit percentages. If the
inant and non-dominant) four times. We wanted each user
participant could not feel the direction, he would be guessing
to perform all three experiments so that we could track
and right on 1/8th or 12.5% of the trials. The value on bold
progress.
It took one hour for one condition (one hand,
on Table 2 (Confidence interval for hit percentages) is the
and one glove configuration) alone, making it impractical to
worst case: The lower end of the 95% confidence interval is
test our subjects on all gloves and both hands. Hence, the
26.95%. From this we can infer that the gloves, at different
participants wore the same glove model on the same hand
rates, do deliver the sense of direction.
throughout all three experiments. The trials were counter-
We cannot conclude that glove 4x4 performed better than
balanced.
the 5x4 model. However, the round glove yielded signifi-
Participants had a chance to practice before each experi-
cantly worse perception results.
ment.
The level of reported perception also appears to play an
important role on the number of hits. Figure 5b shows that
4.1
Experiment I
participants who had higher levels of reported perception
This experiment was designed to answer the question:
performed better with on this task.
Can the gloves convey directional information?
Eight di-
It seems that it does not matter if the glove is worn on
rection signals were sent randomly to the glove. Before each
either hand (Figure 5c).
A closer look tells us the glove
trial, the experimenter asked the participant whether she
worn the dominant hand performed slightly worse than on
was ready to receive the signal. Upon an affirmative an-
the non-dominant hand. However, this difference is not sta-
swer, the experimenter clicked a button to start sending the
tistically significant.
signals. The participant was asked to quickly and accurately
tap on the arrow corresponding to the perceived direction
4.2
Experiment II
on a touch-sensitive pad (Figure 4). This avoids the prob-
Experiment II was designed to answer the question: Does
lem of lexical access where the participant has to think of
the vibration interfere with fingertip reading? This interfer-
the “name of the direction”. The graphical direction presen-
ence may come from the vibration of the motors on the palm
tation on the horizontal tabletop also reduces the problem
of the hand and adjacent areas or because the glove did not
of coordinate transform for the subject and simulates the
fit well for some reason.
orientation of reading a raised-line graphic.
Figure 3 shows a typical board used in this experiment.
To increase redundancy and reduce confusion, we arranged
We assigned two PIFs on the board for each participant.
the display so that the arrows’ locations correspond to the
The glove helped the subject’s navigation to the PIF. When
directions on a map [26, 4]. We recorded the direction sent,
the TPA entered the PIF area, the experimenter asked the
the direction perceived and the participant’s response time.
participant what number she was reading. Each board had
six numbers, collection of dots forming numbers like those on
Response time results.
a dice. We chose this representation because seeing people
We ran one-way Anova at a 95% confidence interval on
are not used to Braille, but normally are familiar to dice or
the data collected. All the analyses on this document were
domino.
done using this confidence interval. We can see in Table 1
Participants were aware that there were six numbers on
that none of the models yielded shorter response times. In
the board. Therefore, if they guess they would be correct
fact the results are quite similar. We understand that, with
1/6th or 16% of the times. Figure 5d shows the means at
practice, these numbers would decrease.
95% of confidence interval.
Y-axes show the correct an-

---

(a) Hit percentages per
(b) Average hit percent-
(c) Hits and misses by
(d) Hit percentage per
glove model
age by level of reported
hand dominance
glove model
perception
Figure 5: Charts from Experiments I and II
Table 3: Speed (cm/sec) in different glove models
Table 4: One of the five stories
Glove
Trials
Mean
Std
Lower
Upper
A friend is thinking of moving out of his
2
bedroom apart-
Model
Speed
Error
Speed
95%
95%
ment. He found another apartment for almost the same price but
Error
Speed
with
3
bedrooms. He hasn’t come to a decision yet because
4x4
13
3.557
0.591
2.354
4.7619
the current apartment is in a very beautiful place. He has until
5x4
12
3.107
0.615
1.854
4.3608
the end of the month to make up his mind, or face a 5 percent
Round
12
2.541
0.643
1.232
3.8503
increase in the rent and still live in the same place.
swers (hits) – the participants answering correctly the num-
ber question.
Auditory-Haptic displays. These tasks have their own goals
None of the glove had the 16% value included on their
and demands. It is not just a matter of redundancy. There-
confidence interval. The results do not allow us to conclude
fore, the question to be answered by this experiment is: Can
that any glove model outperformed other. We also could
a person fuse the information obtained from both modes?
not find any significant difference on wearing the glove on
To answer this question, we told five different one-paragraph-
the dominant or on the non-dominant hand.
long stories to each participant. The stories refer to num-
The question this experiment tried to answer is very im-
bers, which are displayed on the boards similar to the one
portant for future research on the area. If we have found an
shown on Figure 3. Table 4 has one of the stories.
important decrease on fingertip sensibility due to the use of
The experimenter did not say the numbers two and three
these gloves, we would probably be forced to abandon this
that appear on the text. On the other hand, number five
technology.
was said. He told the story while the participant navigated
We also analyzed the times the participants took to reach
to the PIFs just like she did on experiment two. The ex-
the PIFs.
A good system would help the participant to
perimenter adjusted the story telling speed according to the
quickly get to the PIF. We ran one-way Anova to try to
participant’s distance to the PIF to keep the communica-
identify if a particular glove model yielded faster speed (Ta-
tion situated. When the participant reached the PIF, glove
ble 3). The results were inconclusive. Non-significant dif-
stopped vibrating, indicating the user that she has reached
ferences between the use of gloves on either dominant or
the PIF. After a 2 sec pause, the glove started vibrating
non-dominant hand were found.
again indicating the direction to the next PIF. By that time,
Moreover, we compared speed and distance to PIFs. We
the experimenter resumed the story telling. At the end of
wondered if longer distances would mean slower speeds once
each story, four questions were asked.
The experimenter
the participant would proceed slower because the PIF was
read the questions and the alternatives.
further away. Would this decrease the participant’s confi-
To correctly answer some of the questions, the participant
dence on the signal he was receiving? The answer seems to
needed information that was not present on speech, she had
be No. According to the regression performed on the data,
to acquire it from tactile reading. Again, if the participant
differences on distances explain only 15.11% of the differ-
were guessing the answers she would get only 1/4th of the
ences on speed.
answers right. To discourage guessing, we also presented
a 5th option, “I don’t remember”, which was computed as
4.3
Experiment III
wrong.
The experiments get more complex as they get more sim-
We ran one-way Anova on the data collected. Again, we
ilar to what is expected in a classroom. Students who are
found no significant difference among the glove models and
blind will have to navigate with the help of the system while
between dominant and non-dominant hand. Figure 6a shows
paying attention to the teacher. Two tasks using different
the average number of correct answers obtained from par-
modes will be competing for attentional resources. Cross-
ticipants wearing different glove models.
modal time sharing is better then intramodal [26]. Joeong
Even though we cannot say that a particular glove yielded
et al [12] found good recall rates with users working on
better performance, we can see that the lowest end of the

---

Table 5: Averages of correct answers by glove model
Glove
Trials
Mean %
Std
Lower
Upper
Model
Error
95%
95%
4x4
8
77.66%
0.051
67.00%
88.31%
5x4
8
84.79%
0.051
74.13%
95.44%
Round
8
78.59%
0.051
67.94%
89.25%
(a) Average of correct answers
per cross-modal time sharing
confidence interval is 67.00% (bold value in Table 5) of cor-
rect answers. This number is way above the 25% expected
for guessing.
We also wondered if the participants improved their per-
formance as more stories were told. We plot on the percent-
age of correct answers per story, consolidating data from all
hand/glove combination (Figure 6e). A possible explanation
to this graph would be a learning curve followed by fatigue.
This was the third experiment and they were performing
different tasks from almost one hour. Interestingly, subjects
who gave the highest score to the post-questionnaire ques-
tion: “I could keep listening to what was being said while
(b) Speed per-cross-modal time
sharing
the glove was vibrating” , had a significant higher average of
correct answers, 3.59 (f:13.9564, p<0.0001) ( Figure 6b).
We also compared the navigation speeds per glove model,
(Figure 6c). The round glove performed significantly worse
than the other models.
We observed a constant and consistent increase of speed
as the participants listened to the stories. It is also inter-
esting how speed relates to cross-modal time sharing. The
timesharing degree shown on the x-axis of Figure 6d is the
answer the participant gave to the question of how well the
participant could listen to the story while the glove was vi-
brating – the higher the score, the higher the speed. How-
(c) Average of speed per glove
ever, speed explains only 6.05% of the number of correct
model
answers, according to the regression performed.
4.3.1
Post-Questionnaire
After the all three experiments, the participants answered
a questionnaire. We discuss the results in the section.
Sixty percent said they felt comfortable wearing the gloves,
considering all models together. The percentages per glove
were: 50.00%, 88.88% and 37.75% for the 5x4, 4x4 and
round gloves respectively.
Fifty five percent affirmed they could perfectly feel the
direction. The percentages per gloves were: 62.50%, 88.88%
and 12.50% for the 5x4, 4x4 and round gloves respectively.
(d) Average of correct answers
Forty eight percent could not tell weather they would per-
per cross-modal time sharing
form better if they wore the glove on the dominant hand.
The percentages per gloves were: 62.50%, 55.56% and 25%
for the 5x4, 4x4 and round gloves respectively.
Seventy five percent could keep listening to the stories
while they navigate. The percentages were: 87.50%, 62.50%
and 75% for the 5x4, 4x4 and round gloves respectively.
5.
DISCUSSION
The results seem encouraging, but could they be better?
What can we do to improve the system? What conclusions
can be drawn from the experiments?
(e) Percentage of correct an-
Let’s start with the glove models. We built only one size
swers as stories were told
per glove model. One of participants has such a small hand
that it was impossible for her to distinguish any direction.
Figure 6: Charts from Experiment III

---

Table 6: Confusion Matrix
Glove
Direction
Direction Percieved
Model
Sent
N
N
E
S
S
S
W
N
To-
E
E
W
W
tal
5x4 DH
N
1
3
0
0
0
0
0
0
4
5x4 NDH
N
3
0
0
0
0
0
0
1
4
4x4 DH
N
4
1
0
0
0
0
0
0
5
Figure 7: Locations of the receptive field centers of
4x4 NDH
N
4
0
0
0
0
0
0
0
4
334 globrous skin mechanoreceptive units
Round DH
N
1
0
2
0
0
0
0
1
4
Round NDH
N
1
0
0
0
0
0
2
1
4
5x4 DH
NE
0
4
0
0
0
0
0
0
4
We had to find her a substitute. We certainly had some less
5x4 NDH
NE
2
2
0
0
0
0
0
0
4
4x4 DH
NE
1
4
0
0
0
0
0
0
5
extreme fit problems among the other participants. We will
4x4 NDH
NE
1
3
0
0
0
0
0
0
4
probably need to build customized gloves in the future.
Round DH
NE
0
1
2
0
1
0
0
0
4
This “fit” problem must be analyzed more thoroughly. Jo-
Round NDH
NE
0
3
0
1
0
0
0
0
4
hansson et al [13] studied the distribution of tactile units in
5x4 DH
E
0
0
4
0
0
0
0
0
4
the glabrous skin area of the human hand. We can see in
5x4 NDH
E
0
0
3
0
1
0
0
0
4
4x4 DH
E
0
1
4
0
0
0
0
0
5
Figure 7 that some areas of the hand have much more re-
4x4 NDH
E
0
0
4
0
0
0
0
0
4
ceptive centers than others. We cannot place the actuator
Round DH
E
0
0
1
1
2
0
0
0
4
too close to one another because this would make harder to
Round NDH
E
0
1
3
0
0
0
0
0
4
distinguish one direction from other [26]. At this point of
5x4 DH
SE
0
0
0
4
0
0
0
0
4
5x4 NDH
SE
0
0
0
2
2
0
0
0
4
our research, we did not even know if the vibration would
4x4 DH
SE
0
0
0
5
0
0
0
0
5
interfere on tactile perception. More investigation is needed.
4x4 NDH
SE
0
0
0
4
0
0
0
0
4
From the present study, it is not statistically possible to
Round DH
SE
0
0
1
1
2
0
0
0
4
conclude that better results were obtained from the partic-
Round NDH
SE
0
0
0
3
1
0
0
0
4
5x4 DH
S
0
0
0
0
3
1
0
0
4
ipants who had a better actuator/skin area contact. How-
5x4 NDH
S
0
0
0
0
3
0
1
0
4
ever, in all tasks, those who reported higher degrees of per-
4x4 DH
S
0
0
0
1
4
0
0
0
5
ception obtained better results. Furthermore, one source of
4x4 NDH
S
0
0
0
1
3
0
0
0
4
research this study did provide was a confusion matrix Table
Round DH
S
1
0
0
2
1
0
0
0
4
Round NDH
S
0
0
0
0
4
0
0
0
4
6. Analyzing this matrix, we saw that in one fourth of the
5x4 DH
SW
0
0
0
0
0
4
0
0
4
times North was perceived as Northeast on the 4x4 glove.
5x4 NDH
SW
0
0
0
0
0
4
0
0
4
Northeast, East and Southwest had a pretty bad hit rate
4x4 DH
SW
0
0
0
0
0
5
0
0
5
for the round glove. Maybe this is because the actuators re-
4x4 NDH
SW
0
0
0
0
3
1
0
0
4
sponsible for delivering these directions on that model were
Round DH
SW
0
0
0
0
1
3
0
0
4
Round NDH
SW
0
0
0
0
2
2
0
0
4
contacting a poorly in-nerved skin area or not contacting at
5x4 DH
W
1
0
0
0
0
0
3
0
4
all.
5x4 NDH
W
0
0
0
0
0
0
4
0
4
It is interesting to see that even though only 37.75% of
4x4 DH
W
1
0
0
0
1
0
3
0
5
those who tested the round glove felt comfortable wearing
4x4 NDH
W
1
0
0
0
0
1
2
0
4
Round DH
W
0
0
0
0
0
2
2
0
4
it, and only 12.50% of them could not perfectly feel the direc-
Round NDH
W
0
0
0
0
0
0
3
1
4
tions, seventy five percent of them could listen to the stories
5x4 DH
NW
3
0
0
0
0
0
0
3
6
while navigating. This data conforms to the performance of
5x4 NDH
NW
5
0
0
0
0
0
0
4
9
this group on the third experiment: They correctly answered
4x4 DH
NW
8
0
0
0
0
0
0
3
11
4x4 NDH
NW
7
0
0
0
0
0
0
3
10
the questions about the stories over 75% of the time.
Round DH
NW
2
0
0
1
0
1
1
1
6
We also could not find any significant performance dif-
Round NDH
NW
1
0
0
0
0
1
1
2
5
ference between those who wore the glove on the dominant
Legend: 5x4 DH means the 5x4 glove model worn on the dominant
hand and the ones who wore on the non-dominant hand.
hand. NDH = Non-dominant hand
In the third experiment, the experimenter had to adjust
the speed of his speech to participant’s navigation speed.
This suggests that we need some way to inform the teacher
that there is someone in the audience that is not keeping up
with the lecture.
We also need to improve the tracking system. The Lucas-
Kanade tracking algorithm is based on optical flow which is
not stable enough when the object being tracked moves fast
or moves out of the scene. We encountered an orientation
issue. To maintain the consistency between the real world

---

direction and that one being sent by the glove, the user must
[16] B. Lucas and T. Kanade. An iterative image
keep his finger in an orthogonal position with respect to the
registration technique with an application to stereo
desk where the instructional material lays.
vision. In Proc. of 7th International Joint Conference
on Artificial Intelligence, pages 674–679, 1986.
6.
ACKNOWLEDGEMENTS
[17] D. McNeill. Hand and Mind: What Gestures Reveal
about thought. U. of Chicago Press, Chicago, 1992.
This research has been supported by the National Science
[18] S. Millar. Movement cues and body orientation in
Foundation Human and Social Dynamics Program, Grant
recall of locations by blind and sighted children.
number NSF-IIS- 0451843. We also acknowledge our collab-
Quarterly Journal of Psychology, A37:257–279, 1985.
orators on the broader research: David McNeill of the Uni-
[19] N. F. of the Blind. Blindness statistics. National
versity of Chicago, and Mary Ellen Bargahuff, Heidi Turmer,
Federation of the Blind, 2008. http://www.nfb.org.
and Jeffrey Vernooy of Wright State University
[20] A. Pascual-Leone and F. Torres. Plasticity of the
sensorimotor cortex representation of the reading
7.
REFERENCES
finger in braille readers. Brain, 116(1):2230–2236,
1993.
[1] J.-Y. Bouguet. Pyramidal implementation of the lucas
[21] F. Quek, D. McNeill, and F. Oliveira. Enabling
kanade feature tracker. Technical report, Intel
multimodal communications for enhancing the ability
Corporation, 2000.
of learning for the visually impaired. In ICMI ’06:
[2] C. B¨
uhler. The deictic field of language and deictic
Proceedings of the 8th international conference on
words. In R. Jarvella and W. Klein, editors, Speech,
Multimodal interfaces, pages 326–332. ACM, 2006.
Place, and Action, pages 9–30. John Wiley and Sons,
[22] K. Sathian. Practice makes perfect: Sharper tactile
London, 1982.
perception in the blind. Neurology, 54(12):Editorial,
[3] H. Clark. Using Language. Cambridge: Cambridge
2000.
University Press, 1996.
[23] J. Stevens, E. Foulk, and M. Patterson. Tactile acuity,
[4] H. Clark and H. Brownell. Judging up and down.
aging and braille reading in long-term blindness. J
Journal of Experimental Psychology: Human
Exp Psychol Applied, 2:91–96, 1996.
Perception and Performance, 1(4):339–352, 1975.
[24] C. Supalo. Techniques To Enhance Instructors’
[5] A. F. for the Blind. Educational attainment. American
Teaching Effectiveness with Chemistry Students Who
Foundation for the Blind, American Foundation for
Are Blind or Visually Impaired. Journal of Chemical
the Blind, 2008. http://www.afb.org.
Education, 82(10):1513, 2005.
[6] E. Foulk and J. Warm. Effects of complexity and
[25] R. Van Boven, R. Hamilton, T. Kauffman, J. Keenan,
redundancy on the tactual recognition of metric
and A. Pascual-Leone. Tactile spatial resolution in
figures. Percept Mot Skills, 25:177–187, 1967.
blind braille readers. Neurology, 54:2230–2236, 2000.
[7] A. Grant, T. MC, and S. K. Tactile perception in
[26] C. Wickens and J. Hollands. Engineering Psychology
blind braille readers: a psychophysical study of acuity
and Human Performance. Prentice Hall, 2001.
and hyperacuity using gratings and dot patterns.
[27] J. Williams. Nationwide shortage of teachers for blind
Percept Psychophys, 62:301–312, 2000.
students must be corrected. National Federation of the
[8] R. N. Haber, L. R. Haber, C. A. Levin, and
Blind: Advocates for Equality, Canadian Blind
R. Hollyfield. Properties of spatial representations:
Monitor, 14, 2002.
Data from sighted and blind subjects. Perception and
http://www.nfbae.ca/publications/cbm14
Psychophysics, 54:1–13, 1993.
/cbm14.php?article=27.
[9] E. Hinrichs and L. Polanyi. Pointing the way: A
unified treatment of referential gesture in interactive
discourse. Papers from the Parasession on Pragmatics
and Grammatical Theory, Chicago Linguistics Society,
22nd Meeting:71–78, 1986.
[10] M. Hollins. Understanding blindness. Lawrence
Erlbaum Associates, New Jersey, 1989.
[11] C. Intel. Open cv, 05/30/2007 2006.
[12] W. Joeong and M. Gluck. Multimodal bivariate
thematic maps with auditory and haptic display. In
Proceedings of the 2002 International Conference on
Auditory Display, Kyoto, Japan, 2002.
[13] R. Jonhansson and A. Valbo. Tactile sensibility in the
human hand: relative and absolute densities of four
types of mechanoreceptive units in glabrous skin. The
Journal of Physiology, 286:283–300, 1979.
[14] J. M. Kennedy. Drawing and the Blind. Yale Press,
New Haven, CT, 1993.
[15] B. Landau, E. Spelke, and H. Gleitman. Spatial
knowledge in a young blind child. Cognition,
16:225–260, 1984.

---

Document Outline

• Introduction
• Approach
• The System
  • The Haptic Gloves
    • The Directions
    • Tracking
• The Experiments
  • Experiment I
  • Experiment II
  • Experiment III
    • Post-Questionnaire
• Discussion
• Acknowledgements
• References

---