Situation Awareness in Dynamic Decision-Making: Effects of practice and working memory

Situation Awareness in Dynamic Decision-Making: Effects of practice and working
memory

Cleotilde Gonzalez and Jacob Wimisberg
Dynamic Decision Making Laboratory
Social and Decision Sciences
Carnegie Mellon University
Pittsburgh, PA 15213

ABSTRACT
(a) Objective
The goal of this research is to investigate the effects of task practice on SA, to investigate
how cognitive ability —in particular, working memory— moderates this practice effect on SA,
and to investigate effects of the SA measurement procedure (covering or uncovering the display
while queries are answered).
(b) Background
Task practice and working memory influence situation awareness (SA). However, the
dynamics of the relationship between working memory and SA over time are not well
understood, particularly when it comes to using different SA measures.
(c) Method
This research reports an experiment in which different SA measurement methods were
used while participants played a computer simulation over several days. SA was measured using
two different query methods, a covered method in which queries were asked while the display
was blanked out and an uncovered method in which queries were asked while the display was
shown. A working memory measure was collected from participants.
(d) Results

---

Improving Situation Awareness in Dynamic Decision Tasks

1
Results indicate that the relationship between working memory and SA diminishes with
task practice and differs between a covered and uncovered method; and that the SA improvement
with practice depends on the way in which SA is measured.
(e) Conclusion
We discuss the implications of these findings to understanding SA development, SA
measurement, and the relationship of working memory in this process.
(f) Application
The results from this study have clear implications for systems design, for the design of
learning aids and for SA measurement.


Please address correspondence to:

Cleotilde Gonzalez
Social and Decision Sciences
Carnegie Mellon University
Pittsburgh, PA 15213
conzalez@andrew.cmu.edu
(412) 268-6242
(412) 268-6938 (fax)


Short title: Improving Situation Awareness in Dynamic Decision Tasks
Key words: situation awareness, dynamic decision-making, working memory, learning

---

Improving Situation Awareness in Dynamic Decision Tasks

2
INTRODUCTION
Situation awareness (SA), our understanding of what is happening around us and what
could happen in the future, is a complex construct that plays a key role in the performance of
dynamic tasks, such as flying an airplane, driving an automobile, and conducting military
operations. Common definitions of SA are: “the ability to extract and integrate information in a
continuously changing environment and to use such information to direct future actions”
(Wickens, 2000); “the perception of the elements in the environment within a volume of time
and space, the comprehension of their meaning and the projection of their status in the near
future” (Endsley, 1988). Often, SA is divided into three levels: perception, the awareness of cues
and information in the environment; comprehension, the retention, interpretation and
combination of perceptual information to provide meaning; and projection, the ability to
anticipate future events (Endsley, 2000b).
Prior research has assumed that task practice influences SA, with much of the literature
deriving optimal SA from interviews with subject matter experts (Endsley, 2001; Endsley, Bolte,
& Jones, 2003; Shebilske, Goettl, & Garland, 2000). Studies intended to experiment with the
effects of task practice on SA often rely on the comparison between experts and novices (Durso
et al., 1995; Sohn & Doane, 2004) rather than on the process by which novices improve their SA
(Endsley, 2001; Shebilske et al., 2000). Task practice may allow people to develop “the mental
models, schema, and goal-directed processing that are critical for SA in most domains” (Endsley
et al., 2003), but we unfortunately know little about how task practice contributes to SA
development. We believe that an important first step in designing more effective displays and
training interventions is to understand how individuals naturally improve their SA through
practice; thereafter, methods to advance this skill may be designed accordingly.

---

Improving Situation Awareness in Dynamic Decision Tasks

3
Like task practice, cognitive mechanisms such as working memory are commonly
assumed to influence SA (Endsley, 1995; Endsley & Robertson, 2000). Working memory is a
limited human cognitive capacity that must split between temporarily holding information and
strategically manipulating that information for future action (Baddeley, 1992). Although there is
some clear evidence regarding the influence of working memory on SA (Durso, Bleckley, &
Dattel, in press), we only have a weak understanding of the relationship between working
memory and experience: that working memory plays a larger role in SA among novices than
among experts ( Sohn & Doane, 2004; Ericsson & Kintsch, 1995; Ericsson & Staszewski, 1989).
The role of working memory on SA is dependent on the way in which SA is measured
(Durso, et al., in press). A common method to measure SA is query. In this method a task
simulation is stopped at random points and a participant answers a set of queries about the
situation. Queries may be answered while the simulation display is not visible or covered
(Endsley, 1995) or while the display is visible, uncovered (Durso et al., 1995). Durso and
colleagues found that adding SA to a battery of cognitive tests (including working memory tests)
improved prediction of performance only when SA was measured with a query method in which
information was visible. When SA was measured with a not visible method, SA did not improve
the prediction of the cognitive tests, suggesting that the relationship between cognitive tests and
SA depends on the way SA is measured.
The goals of this paper are threefold. The first goal is to investigate the effects of task
practice on SA. Second, is to investigate how cognitive ability —in particular, working
memory— moderates this practice effect on SA. The third goal is to investigate effects of the SA
measurement procedure (covering or uncovering the display while queries are answered).

---

Improving Situation Awareness in Dynamic Decision Tasks

4
Relevant to describing the potential importance of working memory to specific levels of
SA during task training, we also aim to conduct exploratory analyses on measures of perception,
comprehension and projection. Differential effects for the three levels of SA may help explain
and understand more thoroughly the process by which individuals improve SA with practice and
how working memory is related to SA.
The role of working memory on SA development
Working memory is a bottleneck for the development of SA (Endsley & Robertson,
2000). For example, forgetting information reportedly leads to approximately 9% of SA errors
(Jones & Endsley, 1996). Therefore, we expect a correlation between working memory and SA,
specifically, increased working memory (higher scores on a working memory test) is expected to
lead to better SA.
The relationship dynamics between working memory and SA over time (practice) is less
obvious. Researchers have investigated the relationships between cognitive abilities and
performance over task practice. Ackerman (1988) predicted that initial task performance
demands general and content abilities, but the need for these abilities diminishes with task
practice. However, the change in correlations between cognitive abilities and performance
through practice depends on the complexity of the task. For example, the strength of the
relationship between cognitive ability and task performance increases as the complexity of a task
increases (Kyllonen, 1985). Ackerman (1992) also found that for simple tasks the ability-
performance correlations may decrease with task practice, but for complex tasks these
correlations may remain consistent across task practice. We have found consistent task
performance and working memory correlations through practice in complex dynamic decision
making tasks, like that used in the present study (Gonzalez, Thomas, & Vanyukov, 2005).

---

Improving Situation Awareness in Dynamic Decision Tasks

5
Current research on the correlation between ability and performance leads us to believe
that given the complexity of dynamic tasks, the correlation between SA and working memory
will remain constant through practice.
Hypothesis 1. Correlation between scores in a working memory test and SA scores in a
complex decision task will remain consistent and not increase throughout task practice.
The role of SA measurement
Any study on the role of working memory on SA will be influenced by how SA is
measured (Gugerty, 1997). There are multiple ways to measure SA, directly and indirectly,
subjectively and objectively. Direct measures often involve queries asking participants to recall
or recognize events from their experience (Gugerty, 1997). Indirect measures infer SA from the
individual’s performance. For example, SA has been measured as the time it takes decision
makers to detect irregularities in an environment (Sarter & Woods, 1992).
Subjective measures rely on a user’s self-assessment of SA (Jones, 2000), while objective
measures often query participants to recognize a situation and then compare their views of the
situation to reality (Endsley, 2000a). The Situation Awareness Global Assessment Technique
(SAGAT), created and validated by Endsley (1987, 1990, 1995), is an example of a direct and
objective measure of SA. We chose to use SAGAT in the current study because it uniquely
combines objectivity and directedness, and it is a well-documented measure of SA.
With SAGAT, a simulation of a dynamic task is paused at random times and the display
is blanked out. Individuals are queried concerning the current state of the simulation (e.g.,
“What is the current simulation time?”). SAGAT does not require user self-assessment or any
inferences of user behavior. SAGAT is seemingly unintrusive, due to the short (usually less than

---

Improving Situation Awareness in Dynamic Decision Tasks

6
one min.) and random interruptions (Bolstad & Endsley, 1990). Further, no performance
differences have been found with stops of up to five minutes in length (Endsley, 1995).
SAGAT’s requirement of blanking out the display while the operator answers the queries
puts demands on individuals for memory recall to answer the SAGAT queries (Endsley, 2000a;
Gugerty & Tirre, 2000). Conversely, a method in which one allows individuals to view a display
while they answer SA queries may make the answers too obvious (Durso et al., 1995; Gugerty &
Tirre, 2000). Thus, the effects of working memory on SA would depend on how SA is measured.
We planned an experiment using two versions of SAGAT employing both covered and
uncovered displays, similar to the way Durso et al. ( 1995) investigated novice and expert chess
players.
In the covered display condition (the standard version of SAGAT), the task is paused and
the display is blanked out while participants answer queries on perception, comprehension and
projection. In the uncovered display condition, all the information remains in view while the
participant answers the queries. Consequently, in addition to the empirical investigation of
widely accepted assumptions of the experience and working memory relationships over SA, this
paper also attempts to shed light on a methodological controversy of how different SAGAT
versions (blanking the screen or not while answering queries) put demands on working memory.
Like Durso et al. (1997), we expect higher SA scores in the uncovered condition, since all
of the information an individual needs to answer the queries is within view when SA is
measured. We expect individuals may use query information in order to improve their SA with
practice. Similarly, we expect lower SA scores and weaker effects of practice on SA for the
covered condition, as exposure to queries with uncovered displays may contribute to practice.

---

Improving Situation Awareness in Dynamic Decision Tasks

7
Hypothesis 2. Any change on SA with practice will depend on the way SA is measured.
Increases in SA due to practice in the covered condition will be less than increases in SA in the
uncovered condition.
We also expect a stronger relationship between working memory and SA in a covered
versus uncovered display because individuals must recall information from memory while
answering the queries rather than using the information available on the screen.
Hypothesis 3. The correlation between working memory and SA will be greater in the
covered than the uncovered condition.
Further, it seems that different effects of working memory in the covered and uncovered
conditions would depend on the level of the SA queries. For example, working memory will be
essential to answer perception queries (level 1) under a covered display, because responses to
perception queries depend particularly on the visibility of information (Endsley, 2000a). In
contrast, working memory may be less important to answer the same perception queries under an
uncovered display, because there is no need to remember information (the display is visible). In
addition, the need for working memory in responding to queries on level 2 and 3 of SA
(comprehension and projection) is less obvious. For example, it is possible that less working
memory is necessary to answer projection rather than perception queries, because correct
responses to these projection queries may depend more on the individual’s experience with the
task (long-term memory) rather than on holding and manipulating current information (i.e.,
working memory) . However, it is also possible that projection queries will demand working
memory (in both, the covered and uncovered displays) because of the need for mental simulation
and pattern-matching needed to respond to those queries (Endsley, 2000b).

---

Improving Situation Awareness in Dynamic Decision Tasks

8
In this paper, we tested the above hypotheses and explored the effects of working
memory on the three levels of SA for the covered and uncovered conditions.
METHODS
Water purification plant: A dynamic decision-making task
The research reported here involved a computer simulation of a dynamic decision-
making task called water purification plant (WPP). This simulation has been used in many
empirical investigations (Gonzalez, 2004, 2005; Gonzalez, Thomas et al., 2005). WPP is a
resource allocation and scheduling task, representative of many real-world tasks. As reported in
previous research (Gonzalez, Vanyukov, & Martin, 2005), WPP features the characteristics of
dynamic decision-making environments as well as a simplified interface that helps in both
explaining the task and learning it more quickly.
In WPP the decision maker plays the role of an operator whose goal is to distribute water
to different locations before given deadlines (see Figure 1).
[[Insert Figure 1 here]]
The operator activates the pumps assigned to each tank, letting water flow into adjacent
tanks in an effort to pump the water through several chains of tanks before the deadlines. Water
flows only from left to right in the system. The system’s 23 tanks are interconnected in a tree-
like structure. Each tank has two pumps, and only five pumps can be active at any given time
(participants are told that electricity constraints prevent them from using more than five pumps
concurrently). Consecutive tanks (e.g., Tank 3, Tank 15, and Tank 21) compose a chain and have
the same deadline (in this case, 8:00 p.m.).
The simulation provides an indicator that tracks the number of pumps in use (shown at
the top-left corner in Figure 1). Each of the pumps deliver water at the rate of 1 gallon every 2

---

Improving Situation Awareness in Dynamic Decision Tasks

9
minutes; thus, when two pumps are active within one tank, the delivery rate is 1 gallon per
minute. Each pump may be operating in one of four states, identifiable by the color of the
indicator bar above each pump: off (red), on (green), cleaning (yellow), or in queue (purple).
Pumps switch to cleaning mode after they are turned off (either by the operator or by the system
when there is no more water to pump in the tank), and each pump requires 10 minutes of
simulation time to finish cleaning. While pumps are cleaning, the user can select and queue other
pumps (within the five-pump limit), which become active as soon as a cleaning pump turns red.
In WPP, exogenous events (i.e., the arrival of water at different times throughout the
simulation) harm the status of the system if no action is taken (e.g., water will remain in the
chains after the deadline expires). The user’s actions, which are restricted by limited resources,
also affect the status of the system. WPP is a complex task because it involves multiple variables
(e.g., pumps, water, and deadlines) and because some of the relationships among these variables
are nonlinear. For example, pumping water out of a tank before the deadline reduces the number
of gallons missed, but pumping water out of a tank after the deadline has expired is ineffective.
Also, the more often a pump is opened, the more time it spends in cleaning mode and the less
time it is available for pumping water through the chain. Finally, WPP is ‘opaque’ in the sense
that many of the system’s characteristics are not visually represented and therefore are
identifiable only by user inference. For example, later deadlines have longer chains, causing
more water to accumulate in the tanks, a situation that may affect the task prioritization
suggested by the deadlines.
The main performance measure in WPP is the cumulative number of gallons of water
remaining in the tanks after the expiration of all deadlines (shown in the top-left corner of the
screen in Figure 1). The system’s has a total capacity per trial of 1,080 gallons of water.

---