PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 46th ANNUAL MEETING -- 2002 303

Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting -- 2002

ONCE MORE WITH FEELING:

AUGMENTING RECOGNITION PRIMED DECISION
MAKING WITH AFFECTIVE FACTORS

Eva Hudlicka & Jonathan Pfautz*

Psychometrix Associates, Inc., Blacksburg, VA
*Research Laboratory of Electronics, MIT, Cambridge, MA

Although quintessentially human, emotions have, until recently, been largely ignored in the human
factors cognitive engineering / decision-making area. This is surprising, as extensive empirical evidence
indicates that emotions, and personality traits, influence human perception and decision-making. This is
particularly the case in crisis situations, when extreme affective states may arise (e.g., anxiety). The
development of more complete and realistic theories of human perception and decision-making, and
associated computational models, will require the inclusion of personality and affective considerations.
In this paper, we propose an augmented version of the recognition-primed decision-making theory,
which takes into consideration trait and state effects on decision-making. We describe a cognitive
architecture that implements this theory, and a generic methodology for modeling trait and state effects
within this architecture. Following an initial prototype demonstration, the full architecture is currently
being implemented in the context of a military peacekeeping scenario.

INTRODUCTION stimuli, and a bias towards negative appraisal

Both stable characteristics (traits) and transient
emotions and moods (states) influence behavior. (This
is, of course, in addition, to cognitive ability
differences, skill level and training, individual histories,
and contextual factors.) Much empirical and some
theoretical research exists regarding the nature of these
influences on perceptual, cognitive, and motor
processes (e.g., Matthews et al., 2000; Matthews &
Deary, 1998; Williams et al., 1997; LeDoux, 1989;
Ekman and Davidson, 1994). These influences exist
both at the “lower” levels of processing (e.g., attention
orientation during an acute fear episode, increased
working memory capacity correlated with positive
affect), and at “higher” levels involving goals, situation
assessments, expectations, and self schemas (e.g.,
complex feedback relationships between affective state
and self-schemas (Matthews et al., 2000a)).
As might be expected, traits tend to exert their
influence via more stable structures (e.g., types of
schemas stored in long-term memory (LTM),
preferential processing pathways among hc tiona l
components), whereas states tend to produce transient
changes that influence the dynamic characterisiics of a
particular cognitive or perceptual process (e.g., attention
and WM capacity, speed, and accuracy). For example,
low emotional stability (also referred to as neuroticism)
is correlated with a predominance of negative, threat-
and self-related schemas in long term memory, as well
as schemas pertaining to affect (Matthews et al., 2000).
It is also correlated with attentional and situation
assessment preference for self-related and affective
(Matthews et al., 2000). Low emotional stability is also
associated with increased likelihood of negative affect
and punishment-avoiding behavior, while high
extraversion is associated with increased likelihood of
positive affect and reward-seeking behavior (Matthews
and Deary, 1998).
Traits also contribute to the dynamic characteristics
of affective state generation and expression, which is
one of the key mechanisms through which traits exert
their influence. In other words, particular trait value
combinations map onto specific values of affective state
trigger thresholds, growth and decay rates, and
intensity levels. Thus, for example, low emotional
stability correlates with lower trigger thresholds, steeper
growth, slower decay and higher intensity for fear and
anxiety states, as well as more generalized expressions
of anxiety (Rothbart, 1994; Matthews and Deary, 1998).
However, attempts to operationalize the nature of
these influences at a sufficient level of detail to allow
computational modeling have only recently begun to
emerge. Such models are particularly relevant to
cognitive engineering and decision-making research
undertaken within human factors, since they promise to
produce computational models of cognition and
performance that more closely resemble the human user
1 operator.
While much progress has been made in developing,
refining and evaluating theories of situation assessment
and decision-making (e.g., Klein, 1997; Endsley, 2000)
and some studies focus on individual differences
(Gugerty and T h e , 2000) and stress (e.g., Cannon-
304 PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 46th ANNUAL MEETING -- 2002
Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting -- 2002
Bowers and Salas, 1998), little attention has been paid
to explicitly incorporating affective and personality
influences within these theories, at a level that would
enable computational modeling. A notable exception is
the work of Matthews (e.g., Matthews et al., 2000b).
Ignoring these factors leads to incomplete or
erroneous task analysis. The resulting system and user
interface designs may then lack the necessary
accommodations to these pervasive trait and state
effects on cognition and behavior, with consequent risks
of suboptimal or inappropriate user / operator
performance.
In this paper, we describe an augmented theory of
recognition-primed decision-making model (RPD)
(Klein, 1989), that takes into account the effects of
personality traits and emotion states. The theory is
embedded within a symbolic cognitive architecture,
which is parameterized to support the modeling of a
broad range of individual differences, with emphasis on
traits and states. The architecture explicitly represents
processes and structures necessary to model the effects
of these factors on attention, situation assessment and
action selection. The model is currently being evaluated
in the context of a Stability and Support Operation
(SASO) military peacekeeping scenario.
METHOD
Two key issues need to be addressed in developing
the affect-augmented version of the RPD theory, and an
associated computational model:
0 What is the best means of modeling trait and state
effects within a cognitive architecture?
0 What additional processes and mental constructs
must be introduced to the existing RPD theory, to
enable the modeling of trait and state effects?
To address the first issue, we developed a generic
methodology for representing the effects of a broad
range of individual differences factors, including traits
and states, within a cognitive architecture: the
Methodology for Analysis and Modeling of Individual
Differences (MAMID). The underlying assumption of
the approach is that the combined effects of individual
differences can be modeled by varying the cognitive
architecture parameters that control both the processing
and the structure of the architecture and its constituent
modules and knowledge-bases.
To address the second issue, we analyzed existing
empirical evidence and theories regarding the effects of
traits and states on perception, cognition, and behavior,
and the associated mechanisms mediating these effects.
This analysis led to the identification of processing and
structural requirements that motivated the design of the
cognitive arihitecture described below. The
requirements included the following:
0 Explicit representation of specific processes within
dedicated architecture modules (e.g., attention,
situation assessment, expectation generation, affect
appraisal and goal management).
0 Explicit representation of a number of internal mental
constructs: cues, situations, expectations and goals;
with goals representing the primary means of
coordinating perception and action selection, and
mediating the influences of states and traits.
0 Explicit representation of the mechanisms of
reciprocal influences among mental constructs and
the user / operators’s affective states.
0 Dedicated module for deriving the user / operator’s
affective state, resulting from the current situational
and internal operating context.
0 Distinction between task and self constructs, to
accommodate differences in performance focus,
resulting from differences in trait / state profiles,
and uniform processing of these constructs via the
same underlying mechanisms, which supports
smooth shifts in focus between task- and self-
relevant perception, processing and performance.
DESCRIPTION
We briefly outline the modeling methodology and the
architecture below.
Individual Differences Modeling Methodology
The core component of our approach is a generic
methodology for modeling individual differences within
symbolic cognitive architectures, via parametric
manipulations of the architectural processes and
structures. A key objective is to provide flexibility
regarding the ypes of factors selected, and the nature of
their influence on perception and cognition. This
approach is described in more detail elsewhere
(Hudlicka, 2002b; Hudlicka & Pfautz, 2002a; Hudlicka,
1997; also appears in Pew & Mavor, 1998, pp. 261-
267). The distinct individual differences factors are
mapped onto distinct configurations a simulated agent’s
cognitive architecture parameters, which in turn
produce different processing within the architecture,
and ultimately lead to different behavior.
The effects of individual differences factors can be
modeled in two ways: (1) by manipulating the
architecture processing parameters (e.g., working
memory and attention capacity and accuracy), to reflect
a particular personality trait (e.g., preference for self-
vs. task-relevant cues), or particular emotion (e.g., bias
towards detection of threatening cues), thereby biasing
 PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 46th ANNUAL MEETING -- 2002
Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting -- 2002
the behavior in a particular direction; and by (2)
manipulating the architecture content (e.g., structure
and content of the long term memory stores), to reflect
the desired characteristics (e.g., stable trait-based
perceptual and decision-making biases, etc.). Figure 1
provides an illustration of the general relationship
between the traits / states, the architecture parameters,
and the architecture. Table 1 lists examples of cognitive
architecture parameters.
Figure 1: Schematic Illustration of MAMID
Trait / State Modeling Approach and Architecture
Table 1: Examples of Architecture Parameters
Available for Modeling Individual Differences
Cognitive Architecture
The structure of the architecture was motivated in
part by cognitive theories regarding attention, memory,
decision-making; by the existing conceptual structure of
the RPD model; by empirical evidence .regarding
trait/state effects; and by theories of the mechanisms
mediating these effects.
Thus, an explicit attention module was necessary to
demonstrate the trait / state induced attentional biases;
goals were necessary to demonstrate the influence of
goal-situation mismatch on affect generation;
expectations were necessary to demonstrate effects of
expectation valence on affect and action selection; and
a distinction between self- and task-relevant mental
305
constructs was necessary to demonstrate the trait-
influenced differences in self vs. non-self focus.
These considerations led to the design of a symbolic,
integrated cognitive architecture implementing an
affect-augmented recognition-primed decision-making
model. The architecture consists of the following
modules: sensory pre-processing, which translates the
incoming raw data into high-level task-relevant
perceptual cues; attention, which filters the incoming
cues and selects a subset for M e r processing;
situation assessment, which integrates individual cues
into an overall situation assessment; expectation
generation, which projects the current situation into one
or more possible future states; aflect appraisal,, which
derives the affective state fiom both static factors (traits,
individual history) and dynamic factors (current
affective state, current situation, goal, expectation); goal
selection, which selects the most relevant goal for
achievement; and action selection, which selects the
most suitable action for achieving the current goal
within the current context.
Consistent with the RPD model, the processing
emphasis is on the perception of relevant cues, their
integration into a task-relevant situation, and the
selection of an appropriate action by mapping the
situation onto an action within the available behavior
repertoire. Figure 2 illustrates the cognitive
architecture, its constituent modules, and the mental
constructs that comprise the input and output of these
modules; (i.e., cues, situations, expectations, goals, and
actions). The early version of the MAMID cognitive
architecture was loosely based on the SAMPLE
architecture (Zacharias et al., 1995; Harper &
Zacharias, 2002), which also implements the RPD
model, but places more emphasis on the early
perceptual filtering and does not include expectations
and goals, or affective processing and the effects of
traits and states.
Architecture Parameter Space
The effects of individual differences are modeled by
translating distinct trait / state profiles (e.g., high
extraversion, low neuroticism, low anxiety, etc.) into
specific values of the architecture parameters. These
then determine the speed and capacity limitations of the
individual modules, and control the ranking, selection,
and processing speed of the individual constructs within
the modules.
306 PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 46th ANNUAL MEETING -- 2002
Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting -- 2002
Figure 2: MAMID Cognitive Architecture, Mental
Constructs and Emotions that Comprise the Input and
Output of the Architecture Modules
This affect-augmented RPD model is thus able to
account for behavior which might otherwise appear
‘irrational’ (e.g., shift in focus away from the task when
a crisis situation is occurring), by modeling the explicit
threat-induced shift to a self-relevant goal of anxiety-
reduction versus a task-relevant goal to continue the
mission.
FINDINGS
Following an initial prototype demonstration
(Hudlicka and Billingsley, 1999), the full architecture is
currently being implemented and nearing completion.
The current demonstration scenario is within a military
context, where multiple commanders are defined in
terms of distinct individual differences profiles, and
their behavior is modeled by instances of the cognitive
architecture. Alternative mission outcomes can then be
observed, resulting from individual commander’s
distinct reactions to the same set of events (e.g.,
encountering a hostile crowd en route to an objective).
For example, when two distinct commander types
(e.g., ‘low anxious’ and ‘high anxious’) encounter the
same situation (e.g., a hostile crowd blocking their
route), their behavior differs significantly. This is due to
a variety of state / trait effects across the spectrum of
the decision-making process.
During attentional cue filtering, an anxious
commander will focus on threatening cues and neglect
or downgrade other cues. This will result in highly-
ranked cues regarding the crowd attack reaching the
situation assessment module. There, the anxious
commander will interpret the crowd attack as highly
threatening, in part because of the highly-ranked high-
threat cues and lack of balancing and contextual cues
providing evidence to the contrary, and in part due to
trait-associated predominance of threatening schemas in
LTM (content bias), and state-associated preference for
recalling such threatening schemas (process bias).
This interpretation will further increase the highly
anxious commander’s state of anxiety, causing him to
select a self-relevant goal to reduce anxiety, and a task-
relevant goal to defend against attacking enemy. These
goals will then cause the high-anxious commander to
select inappropriate or non-optimal actions. First,
addressing the ‘reduce-anxiety’ goal, the commander
will attempt to reduce anxiety by excessive, irrelevant
communication behavior, not warranted by the current
situation. Second, addressing the ‘defend-from-danger’
goal, the commander will fire into the crowd, also not
an appropriate action given the relative size and combat
effectiveness of an armored unit versus a crowd.
In contrast, the low-anxious commander will not be
subject to these trait and state biases, either by the
process-based focus on threatening cues, situations, and
self and danger related goals, or by the content of the
corresponding LTM structures (e.g., threat and danger
schemas, self schemas, and anxiety-reduction and self-
defense activity repertoire). Figure 3 provides a
summary of the effects of these distinct profiles for each
of the cognitive architecture modules.
Figure 3: Distinct Commander Profdes and Behaviors
DISCUSSION
We have operationalized the conceptual RPD model
within a computational cognitive architecture and
augmented the existing model structure with a dedicated
affect appraisal module and explicit representations of
trait and state effects on perfonnance. Preliminary data
from the initial MAMID prototype, and the on-going
full-implementation, demonstrate distinct differences in
individual commander behavior, as a fimction of their
individual profiles, and consequent significant
differences in the final mission outcome. The modeling
environment allows the analyst to trace the specific trait
 PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 46th ANNUAL MEETING -- 2002
Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting -- 2002
/ state effects through the decision-making sequence,
starting with incoming cues and ending with the final
outgoing action selection. The environment also
supports flexible manipulation of a variety of factors
and parameters, to fine-tune the model performance.
Model validation will be performed once the full-
implementation is complete. Both a heuristic
evaluation by subject matter experts (psychologists and
military commanders), and an in-depth comparison with
empirical data are planned. However, due to lack of
sufficiently detailed empirical data, some of this
validation process may be limited to input / output
validation.
One of the key contributions of a computational
approach is the ability to generate hypotheses for
targeted empirical studies. A tightly-coupled
theoretical-modelingand applied-empirical program of
research promises to produce detailed characterizations
of the effects of traits and states on performance, and
the associated mediating mechanisms.
CONCLUSIONS
The ability to explicitly model effects of traits and
states on performance will contribute to the
development of more realistic human performance
models. Such realism is particularly necessary as a
variety of 'mission-critical' decision aids and
computerized systems become increasingly ubiquitous.
To ensure effective human-machine interaction,
across a variety of tasks and populations, cognitive
engineers must take into account the broad range of
individual differences effects on performance.
Existing results, and future developments, of the
MAMID cognitive architecture and modeling
methodology are relevant from both a theoretical and
from an applied perspective. The ability to model and
explore alternative theories regarding mechanisms
mediating trait and state effects is valuable for basic
research and understanding of these pervasive effects,
both positive and negative, on performance. The ability
to model distinct operator / user types, based on a
definition of their individual differences profile, has
applications in a variety of areas of interest to the
cognitive engineer, including user interface design and
task allocation; individual and team training
environments; performance analysis and assessment;
and behavior modeling, simulation, and prediction.
307
ACKNOWLEDGMENTS
The research described in this paper was supported in part by US
Army Contract DASWOI-00-C-3000. W e would like to
acknowledge continued support of the COTR Dr.Bob Witmer, and
the contributions of the following individuals: Dr.Greg Zacharias of
Charles River Analytics provided valuable discussions and
suggestions regarding the simulation algorithms, Prof. William
Revelle of Northwestern University made suggestions regarding the
individual differences factor selection, and Mr. Ted Fichtl of The
Compass Foundation was the subject matter expert. Mark Turnbull
of Contented Software developed the world simulation module. We
also acknowledge the efforts of the Psychometrix software team:
Lisa Buomano, Jim Helms, Craig Ganoe
REFERENCES
Cannon-Bowers, J. A. & Salas, E. (1998). Decision Making Under Stress.
Washington, DC: APA.
Ekman, P. & Davidson, R.J. (1994). The Nature of Emotion. Oxford..
Endsley, M. R. (2000). Theoretical Underpinnings of Situation Awareness: A
Critical Review. In Situation Awareness Analysis and Measurement,
M.R. Endsley & D.J.Garland, eds. Mahwah: NJ: LEA.
Endsley, M. R. (1995). Measurement of Situation Awareness in Dynamic
Systems. Human Factors, 37(1).
Gugerty, L.J. & Tim, W.C. (2000). Individual Differences in Situation
Awareness. In Situation Awareness Analysis and Measurement, M.R.
Endsley and D.J. Garland, eds.Mahwah: NJ: LEA.
Harper, K. & Zacharias, G. (2002). Modeling Attention Allocation and
Multitasking in Computational Human Behavior Representation. In
Proceedings of the IShConference CGF-BR,May, 59-66.
Hudlicka, E. & Pfautz, J. (2002a). Architecture and Representation
Requirements for Modeling Effects of Behavior Moderators. In
Proceedings of the 11" Conference on CGF-BR,May, 9-20.
Hudlicka, E. (2002b). This time with feeling: Integrated Model of Trait and
State Effects on Cognition arid Behavior. Applied AI, 16:1-31.
Hudlicka, E. (1997). Modeling Behavior Moderators in Military HBR
Models. TR # 9716. Lincoln, MA: Psychometrix Associates, Inc.
(sections appear in Pew and Mavor, 1998)
Hudlicka, E. and Billingsley, J. (1999). Representing Behavior Moderators in
Military Human Performance Models In Proceedings of the 8 '
Conference on CGF-BR. Orlando, FL.
Klein, G. A. (1997). The recognition-primed decision (RPD) model: Looking
back, looking forward. In C, Zsambok & G. Klein (Eds.), Naturlistic
decision making. Mahwah,NJ: Erlbaum.
Klein, G. A. (1989). Recognition-Primed Decisions. Advances in Man-
Machine Systems Research, 5,47-92.
LeDoux, J.E. (1989). Cognitive-Emotional Interactions in the Brain.
Cognition and Emotion, 3(4), 267-289.
Matthews, G., Denybeny, D., & Siegle, G.J. (2000a) Personality .and
emotion: Cognitive science perspectives. In S.E. Hampson (Ed.),
Advances in personality psychology (Vol. 1). London: Routledge.
Matthews, G., Davies, D.R., Westerman, S.J. and Stammers, R.B. (2000b).
Human Performances Cognition, Stress, and individual differences.
Philadelphia, PA: Taylor and Francis.
Matthews, G., and Deary, 1.J. (1998). Personality Traits. Cambridge, U K :
Cambridge.
Pew, R.W. and Mavor, A.S. (1998). Representing Human Behavior in
Military Simulations. Washington, DC.National Academy Press.
Williams, J.M.G., Watts, F.N., MacLeod, C., and Mathews, A. (1997).
Cognitive Psychology and Emotional Disorders. NY: John Wiley.
Zacharias, G.L., Miao, A.X, Illgen, C., and Yara, J.M. SAMPLE: Situation
Awareness Model for Pilot-in-the-Loop Evaluation. In Proceedings of
the Ist Annual Conference on Situation Awareness in the Tactical Air
Environment. Wright-Patterson AFB, OH: CSERIAC.