Computers & Graphics 42 (2014) 4258

Contents lists available at ScienceDirect

Computers & Graphics

journal homepage: www.elsevier.com/locate/cag

Technical Section

A human cognition framework for information visualization$

Robert E. Patterson a, Leslie M. Blaha a, Georges G. Grinstein b, Kristen K. Liggett a,n,
David E. Kaveney c, Kathleen C. Sheldon b, Paul R. Havig a, Jason A. Moore c
a
Human Effectiveness Directorate, Air Force Research Laboratory, Wright-Patterson AFB, OH, United States
b
Department of Computer Science, University of Massachusetts, Lowell, MA, United States
c
Information Directorate, Air Force Research Laboratory, Rome Research Site, NY, United States

art ic l e i nf o a b s t r a c t

Article history: We present a human cognition framework for information visualization. This framework emphasizes
Received 10 July 2013 how top-down cognitive processing enables the induction of insight, reasoning, and understanding,
Received in revised form which are key goals of the visual analytics community. Specically, we present a set of six leverage
17 March 2014
points that can be exploited by visualization designers in order to measurably inuence certain aspects
Accepted 21 March 2014
Available online 21 April 2014
of human cognition: (1) exogenous attention; (2) endogenous attention; (3) chunking; (4) reasoning
with mental models; (5) analogical reasoning; and (6) implicit learning.
Keywords: Published by Elsevier Ltd.
Information visualization
Visual analytics
Human cognition
Cognitive augmentation
Visual attention

1. Introduction augmentation issue and propose that well-designed visualizations

Information visualization refers to the interdisciplinary eld
concerned with the visual representation of complex information
in ways that enhance understanding [1]. The eld draws from such
disciplines as computer science, graphics, visual design, psychology,
mathematics, and business. The role of information visualization is
to leverage the functioning of the human visual system in an effort
to provide insight about abstract information [2], to help humans
resolve logical problems, to think and reason [3], and to provide
help in understanding data [4]. High-level cognitive functioning,
such as developing insight, reasoning, and understanding, is engaged
by visualization techniques because visual perception possesses
special properties [3], is attuned to visual images [2], and it performs
pattern recognition [4].
These statements, and others like them, are likely to be true,
and have been a focus of the visual analytics community. Yet they
lack details and specicity in the linkage between visualizations
and the induction of high-level cognitive reasoning and under-
standing. To articulate the mechanisms and processes supporting
high-level cognition, we need to place information visualization
within a detailed foundation of cognitive psychology. Accordingly,
we see information visualization fundamentally as a human cognition-
This article was recommended for publication by M. Chen.
n
Corresponding author.
E-mail address: kristen.liggett@us.af.mil (K.K. Liggett).
induce reasoning and understanding by inuencing high-level
cognitive processes such as retrieval from long-term memory, in
addition to leveraging human visual perception capabilities.
Our focus in this paper is on a human cognition framework for
information visualization which makes direct contact with under-
lying cognitive processes that enable the induction of insight,
reasoning, and understanding. We specically conjecture that
there is a set of leverage points a given visualization designer
might exploit in order to inuence human cognition in the visual
analytics process.
Patterson [5] identied a number of principles for information
display design in order to shift the emphasis in much of the
display community from entertainment to task-oriented represen-
tations of information. In this paper, our aim is to offer a broader
framework for the visualization design process through the
identication of key leverage points.
Research on human cognition has been utilized to varying
degrees in interaction design and analysis by the HumanCompu-
ter Interaction (HCI) and Human Factors and Ergonomics (HFE)
communities. For example, the HCI community has primarily
focused on performance evaluations of two or more designs for
a given task, where best performance on measures such as
response time and accuracy is taken to indicate design effective-
ness. Many of the design principles prominent in the HCI/HFE
community are summarized as lists of Dos and Don'ts, which
provide very general guidance [68]. The supporting rationale that
connects these guidelines to perceptual/cognitive capabilities are

http://dx.doi.org/10.1016/j.cag.2014.03.002
0097-8493/Published by Elsevier Ltd.
 R.E. Patterson et al. / Computers & Graphics 42 (2014) 4258
not obvious, so the actual implementation of the guidance is often
interpreted by the designer. More recently, a few authors have
embarked on more theoretical approaches, such as Huang et al. [9]
who developed a measure of cognitive load they called mental
effort.
Many topics in human factors-based design have not been
explored by the visualization community in much depth. Much of
the current methodology for designing visualization tools and
interfaces is still ad hoc and informal. Only a few visualization
designs utilize perceptual and cognitive theories. Tory and Moller
[10] suggested that, because many areas of perception and cogni-
tion research are likely not utilized to their full potential, further
work in this area is promising. These authors provided a survey of
possible perception and cognition support for information visua-
lization, such as methods for improving perception of 3D shape,
techniques to more easily distinguish and highlight objects, new
interaction methods and input devices (e.g., real-world props),
faster rendering for better interactivity, interfaces to make transfer
function specication easier, and methods of reducing memory
load (e.g., detail and context displays).
It is in the spirit of Tory and Moller [10] that we offer our
human cognition framework for information visualization. We rst
present a selected review of the literature on information visua-
lization. Next is a section that presents our human cognition
framework for information visualization, which includes a theore-
tical perspective and an overview of human cognition. We then
present a set of six leverage points with specic suggestions for
assessing the design choices through human performance mea-
surement. The leverage points we provide tie a particular design
principle to human cognitive processes so the designer under-
stands why a particular recommendation is made. We illustrate in
the next section the implementation of our framework in the
visualization design process and a case study. Finally, we relate our
perspective to some other theoretical frameworks for human
cognition which may complement the implementation and assess-
ment of our human cognition framework for visualization.
2. Information visualization
Information visualization has several design reference models,
many of which now include the users interpretation of the
visualization; almost all focus on the synthesis process and
present alternative taxonomies for classication or synthesis
methods. For example, in a pipeline design model based on Card
et al. [3] (Fig. 1) the focus is on displaying data, and the involve-
ment of the user's cognition is left undened.
In one early synthesis model, Bertin [11] used perceptually
based graphical symbols and marks to generate the visual repre-
sentations. Aspects of these symbols were position, shape, size,
brightness, color, orientation, texture and motion. There are other
models [3], some of which include other perceptual modalities
(e.g., audio or tactile), but none offer explicit treatment of the
cognitive activity engaged by the user in the visual design process
[12].
Formalization marked the beginning of a change from the early
models. Mackinlay [13] developed the rst computational
approach to rule generation with the Automatic Presentation Tool
(APT). Wehrend and Lewis [14] believed that by categorizing all
visualizations (or at least the large number with which they
started), the catalog could be the target of an automatic visualiza-
tion selector based on suitable values. Roth et al. [15] used
composition to create more complex displays, including exten-
sions to 3D. Keller and Keller [16] dened visualization goals;
Shneiderman [17] used data types for classication; Ward et al. [1]
added user interaction.
43
Fig. 1. Typical pipeline of information visualization design found in the visualiza-
tion literature. The pipeline's focus is on displaying data, and the role played by the
user's cognition is unspecied.
The incorporation of human cognition into the visualization
process demarcates the modern approach. Casner [18] was one of
the rst to start generating perceptual task-driven displays from
decision-making task descriptions. The effects of visualizations on
the user's ability to mentally visualize data were examined by
Zacks, Tversky and colleagues [1922]. Finally, North [23] tried to
quantify the user's insights gained through the use of visualization.
Thus, some modern theories do mention perception and
cognition in their synthesis models, but most do so in a limited
fashion; the connection of the visualizations to actual perceptual
or cognitive processes is usually vague and leads to no or few
measurable outcomes or guidelines. The approach of simply
tacking on a nal stage called User, as in Fig. 1, is very proble-
matic because it ignores the complex nature of human cognition
as the user engages with the visualizations [24,25].
The compelling need for a precise understanding of human
cognition in the design of information visualizations can be appre-
ciated by considering a study by Elting et al. [26]. In this study,
participants interpreted multiple joint and conditional probabilities
presented in four different graphical formats: a numerical table, a
pie chart, a divided (stacked) bar chart, and an icon display. Elting
et al. [26] found that the numerical table was the most preferred
display format, yet it produced a lower level of decision-making
accuracy relative to the icon display. The icon display was preferred
by no participants and disliked by a quarter of the participants, but
it produced the highest level of decision-making accuracy. The pie
chart and bar graphs, which were also preferred representation
techniques, yielded the lowest accuracy. Thus, visualization
designers cannot rely only upon the subjective preferences of users
because subjective preference may not be a reliable indicator of
objective performance. Instead, designers should take the cognitive
processes of the user into account in order to ensure that objec-
tively measured performance is facilitated.
Designing an information visualization for good analytical rea-
soning is akin to designing graphical presentations of data for
information extraction rather than data availability [27]. In data
availability, data is displayed so that the burden of identifying,
remembering, and drawing inferences from information is placed
on the user. In information extraction, data is displayed to facilitate
human cognitive processing, which takes advantage of graphical
primitives or elementary cognitive codes, like length, angle, or area.
But we can go much deeper than such primitive codes. We can
directly link information visualization with high-level cognitive
processes such as reasoning and thinking. In doing so, we present
a human cognition framework for information visualization and
argue that visualizations should engage and promote high-level
cognitive functioning.1
1
It is important to note that we are explicitly stating that a well-designed
visualization should inuence high-level cognitive functioning within the environ-
mental context for which it was designed but will not necessarily be effective
outside of that context.
44 R.E. Patterson et al. / Computers & Graphics 42 (2014) 4258

3. Human cognition framework for information visualization extraction is consistent with the well-known propensity of
humans to remember the meaning and interpretation of pictures
3.1. Theoretical perspective but forget much of their physical details, even after only a 10-min
retention interval [39,40].
Our theoretical perspective is that sensory and perceptual
processing occurs within a context of top-down inuence that
3.2. Overview of human cognition
involves the search for a meaningful interpretation of stimulation.
This means that visual perception should be conceptualized as
This overview of human cognition is based on what is called
being more than simply bottom-up engagement of low-level,
the dual-systems theory, or dual-process theory, that has been
feature-detection processes that sequentially feed into higher-
proposed as an organizing framework in both contemporary
level cognition, which has been the traditional treatment of visual
cognitive and social psychology [12,4145]. In presenting this
perception in most textbooks [28]. Conceptualizing the role of
theory, it should be pointed out that not all researchers share
visual perception as a strictly feed-forward process risks leading us
the assumptions of this theory, and other frameworks of human
to the assumption that the designer's task is one of just providing
cognition exist in the published literature. For example, the
sets of visual features congured in certain ways so as to help
intuitive system very likely involves multiple systems of a given
individuals perceptually organize visual displays. While this is an
type rather than a single system [45]. Moreover, the intuitive
important component of the total cognitive process, the connec-
system(s) may engage in high-level cognition rather than being a
tion between perception and cognition is more complicated. More
simple pattern-recognition process based solely on statistical
importantly, if we limit our designs to the facilitation of visual
regularities: the intuitive system may integrate information and
perception alone, we miss the opportunity to inuence cognition
engage in response selection [46], and the intuitive system may
in more powerful ways.
extract meaning from patterns of stimulation [47]. Despite the lack
Human cognition arises from the dynamic interplay between
of full consensus on the exact nature of these two systems or
bottom-up and top-down processing. Our human cognition
processes, analytical and intuitive, we believe that the dual-
framework for visualization design emphasizes harnessing
systems or dual-process theory offers a reasonable organizing
the top-down processing. With bottom-up processing, the
structure on which to base our framework.
visual stimulus is ltered by arrays of biological mechanisms
The core ideas found in the dual systems theory literature are
called receptive elds, which represent the zones of selective
that human cognition derives from an interplay between an
stimulation on the retina that project to different cell types
analytical reasoning system (composed of working memory and
throughout various stages of the visual system. The receptive
long-term declarative memory) and a rapid, autonomous intuitive
elds of early levels of vision are selectively stimulated, for
or implicit system that entails implicit pattern recognition and
example, by lines of varying width and orientation, whereas
procedural long-term memory [43]. Both the analytical and
receptive elds at higher levels are selectively activated by more
intuitive systems involve encoding, and they feed into a common
complex congurations [29].
decision point. Consequently, the dual systems are heavily reliant
Top-down processing guides the way in which the bottom-up
on both bottom-up and top-down processing. According to Evans
information is processed in order to activate organized knowledge
and Stanovich [45], the rapid, autonomous processes yield default
structures represented in long-term memory, which is why we
responses unless interfered with by the higher-order analytical
emphasize top-down processing in our framework. Patterson [5]
reasoning processes.
discussed the inuence on perception of top-down processes, and
Logically, both the analytical system and the intuitive system
evidence for this abounds in the cognitive and perceptual psy-
entail pattern recognition. For example, the analytical reasoning
chology literatures. One example of such top-down inuence on
system would be responsible for analogical reasoning. Analogical
perception comes from Weisstein and Harris [30], who showed
reasoning is dened as the transfer of inferences from a relation-
that individuals could identify a briey ashed line segment
ship of elements in one domain (the analogue) to a relationship of
more accurately when it was part of a unied drawing than when
elements in another domain (the target), which could be con-
the line was in a less coherent conguration (see also [31,32]).
strued as a form of pattern recognition. But the analytical system is
If the visual system was a simple feed-forward hierarchy, with
also involved in other forms of reasoning as well, such as
lower-level activity feeding into higher-level mechanisms which
deductive reasoning, which entails the process of reasoning from
respond to more complicated objects, we would expect the
general premises to reach logical conclusions. The intuitive system,
opposite result.
however, is dened solely by pattern recognition based on the
Additional evidence that cognition derives from the dynamic
learning of statistical regularities. Accordingly, in our framework,
interplay of top-down and bottom-up processes comes from
we dene the analytical reasoning system as working memory and
Grill-Spector and Kanwisher [33], who reported that individuals'
long-term (declarative) memory, and the intuitive system as
performance on a visual categorization task was as fast and
pattern recognition and long-term (procedural) memory.
accurate as on a detection task, indicating that the observers
Fig. 2 depicts the ow of information in human cognition,
already knew the object's category (semantic information) when
beginning with an impinging stimulus on the left and moving
they detected its presence. Oliva and Torralba [34] reported that
through a semi-parallel series of cognitive mechanisms until a
the semantic information of a scene can be extracted from
decision is made and the user responds accordingly, either to
a 200 ms exposure. Thorpe et al. [35] reported that, based on
affect the stimulus in some way or to complete a task at hand. We
event-related potentials, object recognition could occur within the
now turn our attention to explicating the processes in this
rst 150 ms of visual-processing time. Biederman and colleagues
information ow, including the critical feed-back mechanisms
[3638] showed faster speed or greater accuracy for object
throughout the system that create the dynamic, top-down inter-
identication when the object was presented within a coherent
play so critical in cognitive performance.
real-world scene than when the object was presented in a jumbled
scene.
Bottom-up ltering by the visual system thus is an embedded 3.2.1. Encoding
substrate within a neural architecture designed to efciently External information enters the perceptual and cognitive pro-
extract the meaning from visual stimulation. This idea of meaning cessing systems via encoding, which refers to the conversion of a
 R.E. Patterson et al. / Computers & Graphics 42 (2014) 4258
Fig. 2. Overview of human cognition based on a dual-process framework for reasoning and decision making, which forms the backbone of our human cognition framework
for information visualization. The ow of information proceeds from left to right, beginning with an impinging stimulus on the left and ending with a decision being made
on the right. The components of human cognition shown here, namely encoding, working memory, pattern recognition, long-term memory, and decision making, are
discussed in the text.
visual image into a neural representation in human memory.2
In many cases, the neural representation persists only briey,
without extensive cognitive processing, and is held in the visuo-
spatial temporary store (Fig. 2), which is often referred to as visual
short-term memory. Simultaneously, the encoded stimulus may
activate a series of cognitive pattern-recognition mechanisms.
Some encoded stimulus representations may activate enduring
pattern or information traces, which are known collectively as
long-term memory.
Only the information that is encoded can be used for subse-
quent processing. Based on goals and the demands of the present
task, the user may have some idea of the information needed to be
extracted from a visual stimulus and can direct attention to
stimulus attributes to assist in their encoding. Thus, encoding is
intimately related to the top-down inuences of attention and
memory.
3.2.2. Attention
Attention can be dened as the process of focusing cognitive
resources on selected aspects of the environment while ignoring
others. It can be inuenced both by the properties of the environ-
ment as well as by the top-down cognitive processes.
Exogenous attention refers to the capture of attention by the
presence of a triggering stimulus in the visual eld, often in the
periphery. Egeth and Yantis [49] refer to exogenous attention as
stimulus-driven, as opposed to endogenous attention, which is
goal-directed. Certain visual attributes that distinguish a stimulus
from its background (e.g., color, orientation, motion), which are
called singletons, can serve as the triggering feature, especially
abrupt visual onsets or bright colors. For example, a moving target
stands out among stationary distractors, and a fast-moving object
stands out among slow-moving objects [50]. While this kind of
stimulus-driven attention can be important in certain monitoring
applications, such as when a blinking light alerts operators, this
would represent a small portion of the visualization task domain
because users typically never look at a visualization without some
task in mind. Nonetheless, a well-designed visualization could
2
For detailed review of perceptual encoding, see Kosslyn [48].
45
leverage singletons to alert a user to a certain aspect or changing
detail of the visualization that the designer deemed important.
Exogenous attention often occurs just prior to, or in parallel
with, stimulus encoding, generally preceding the processes of
working memory. However, there is some evidence that some
working memory processes can affect exogenous attention [51].
This is consistent with statements by Egeth and Yantis [49] who
argued that both the properties of stimulation [51] and the
expectations and goals of the observer (i.e., top-down inuence)
can determine whether a stimulus will capture the observer's
attention.
Endogenous attention refers to the volitional and active deploy-
ment of processing capacity to an external stimulus, or an internal
state, in a goal-directed fashion [52,53]. Endogenous attention
plays a critical role in working memory and executive control, as
we will explicate later. Once information gains access to working
memory, it must be maintained in an active state for the duration
it is relevant to a given task. One method for maintaining
information in working memory is rehearsal. Rehearsal involves
covert shifts of endogenous attention; the latter therefore plays a
signicant role in memory maintenance [52,54,55].
Because it is under conscious control, endogenous attention is
one way in which the cognitive mechanisms act in a top-down
manner to guide the processing of a visualization. Endogenous
attention is an important type of attention because users typically
never look at a visualization without some task in mind. Atten-
tional distractions, or other conditions that encourage goal neglect
[56], have the potential to cause the contents of working memory
to decay or be displaced which might impair the user from gaining
information from the visualization.
Wickens [57] described a multiple resource model of mental
capacity, which is analogous to attention. In this model, decre-
ments in performance that occur when multiple tasks are being
performed can be predicted from values taken on four dimensions:
processing stages (perceptual-cognitive versus selection and
action execution), processing codes (spatial versus verbal-linguis-
tic), modalities (visual versus auditory), and, if the tasks involve
vision, type of visual channels (focal versus ambient vision).
Greater task interference occurs when two tasks are similar along
these dimensions, and less interference occurs when two tasks are
different along these dimensions.
46 R.E. Patterson et al. / Computers & Graphics 42 (2014) 4258

Thus, visualizations should be designed so as not to produce 3.2.5. Long-term memory

interference when a user is concurrently performing a second task. Central to the internal cognitive processing in Fig. 2 is long-
This would involve making sure that the visualization and the term memory. Long-term memory refers to the neural representa-
second task are different along the four dimensions proposed by tion of information and knowledge that is relatively enduring.
Wickens [57]. Long-term memory can be subdivided into separate systems (not
reected in Fig. 2): a declarative memory system and a non-
declarative memory system.
The declarative memory system involves conscious recollection
3.2.3. Working memory
of facts and events. Declarative memory is further subdivided into
Working memory refers to a dynamic form of conscious mem-
semantic memory, capturing meaning and relational knowledge,
ory that provides an individual with the ability to perform
and episodic memory, capturing memory for events and beha-
complex mental operations [52,5861]. A core component of
viors. The non-declarative memory system includes several sub-
working memory is the central executive (see Fig. 2, upper middle
systems [75,76]. One important non-declarative subsystem is
box), which performs a variety of important functions through
procedural memory, which encodes sequences and statistical
endogenous attention: selecting targets to be encoded in accor-
regularities [77,78], and is involved in the development of percep-
dance with current goals; directing resources away from irrelevant
tual motor skills. Note that the two ways in which long-term
distractors; activating declarative long-term memory representa-
memory interacts with working memory and pattern recognition
tions; selecting responses; and maintaining information in work-
reect the analytical and intuitive systems comprising the dual-
ing memory for further processing [5]. Visual inputs are
systems framework discussed above [12,4144].
maintained in a visuo-spatial temporary store, which is a part of
Long-term memory can exert many forms of top-down inu-
working memory, for parsing and for comparison during visual
ence across the perceptual and cognitive processes. Long-term
tasks.3
memory has a direct top-down connection to working memory
As information comes into working memory, the information is
because long-term declarative memory can inuence the chunk-
mentally parsed into a structured organization via chunking.
ing of patterns or information in working memory. Pattern
Chunking refers to the mental process of grouping elements into
recognition is directly inuenced by long-term memory because
larger units based on their meaning [62], which would involve
long-term procedural memory can enable the act of pattern
retrieving representations from long-term memory. Because work-
recognition. Woodman et al. [79] suggests that long-term memory
ing memory has a xed capacity, chunking increases the amount
may be directly involved in the selection of targets to be encoded
of incoming information that can be encoded and subsequently
in accordance with current goals (i.e., top-down inuence). How-
processed or remembered. In the past, this limit was thought to be
ever, the top-down inuence of long-term memory on the encod-
about seven items [60], but recent estimates put this limit at four
ing processes is indirect and mediated by the many activities of
chunks [63]. The amount of information contained in each chunk
the working memory system. Importantly, encoded information
can be quite large [64].
can become directly represented in long-term memory owing to
For example, De Groot [64] showed that master-level chess
its emotional quality, for example, or strong sensory associations.
players were able to reconstruct from memory the legally cong-
ured position of chess pieces placed on a chess board almost
perfectly after a 5 s viewing (as opposed to randomly placed 3.2.6. Decision making
pieces), which was based on the chess masters' abilities to The nal stage of cognitive processing, on the right side of
perceive structure among the chess pieces and encode them in Fig. 2, is decision making. Decision making represents the process
mental chunks [6569]. We see in the process of chunking that the of selecting an option from the set of alternatives available for a
visual input the chess positions served as retrieval cues which given task, which may result in a response. The decision compo-
triggered the long-term memories of patterns of attack and nent of the model receives input from both working memory and
defense. The latter provided the top-down component and was pattern recognition. The feed forward connection from pattern
the basis for the chunking. The triggering of long-term memory recognition indicates that a user has an ability to render a decision
during encoding can also allow the expert player to anticipate based on implicit pattern recognition (using procedural memory),
subsequent chess moves [70]. which is called intuitive in the dual-process framework [12,4144].
Decision making also has a bi-directional connection to working
memory which indicates that a user has an ability to render a
decision based on conscious deliberation (supported by working
3.2.4. Pattern recognition
memory), which is called analytical in the dual-process frame-
Working in parallel with the central executive of working
work. The decision making process can also guide the activities of
memory, encoded representations can also be involved in an
working memory via top-down inuence, by updating goals or
implicit pattern-recognition process. While a number of cognitive
eliminating choices under consideration.
abilities rely upon pattern recognition (e.g., analogical reasoning),
While our basic model presented in Fig. 2 has a general serial
in the present context this kind of pattern recognition refers to the
structure (e.g., stimulus, working memory, decision and response),
recognition of statistical regularities encountered in the environ-
we note that this process need not always have a distinctive
ment, including within a visualization. Here, the incoming statis-
starting and stopping point. Our perceptual systems are constantly
tical pattern would serve as a retrieval cue which would trigger
sampling the world, so many of these processes are occurring in
representations in long-term procedural memory. The latter pro-
parallel.
vides the top-down component and is the basis for the recognition
response. This kind of pattern recognition is a ubiquitous and
robust process that underlies intuitive decision making [43,7173] 4. Information visualization and leverage points
and possibly the development of expertise [74].
In considering the interplay between human cognition and infor-
3
Models of working memory also contain an analogous temporary store for
mation visualization, we will assume that the cognitive processing
other sensory modalities, such as the auditory temporary store for auditory begins when an observer is presented with a new visualization.
information. Cognitive processing will continue until some decision resulting from
 R.E. Patterson et al. / Computers & Graphics 42 (2014) 4258
that visualization is made or some action is taken. This, for example,
may simply result in a change to the visualization for further
scrutiny, such as zooming in on a subset of data. A change to the
visualization begins the cognitive processing again. It is through this
iterative process that the goals and intentions of the user directly
inuence the information available for encoding, the crux of how
top-down processes inuence the information available to the
encoding mechanisms.
Before continuing to our leverage points, mention should be
made of a framework for information visualization that also draws
heavily from the human cognition literature [28]. Ware's frame-
work is a three-stage hierarchical framework consisting of the
following: a lower stage that processes the visual scene into
elements of form, color, texture, and motion; a middle stage that
forms and perceives patterns; and a higher stage that selects (via
attention) objects and patterns for visual queries. This is a strongly
bottom-up framework in which information processing starts at
the lowest stage and proceeds on up to the highest stage. At the
highest stage, there is brief mention of visual problem solving
involving pattern perception and working memory. There is also
brief mention of semantic networks, verbal-propositional mem-
ory, and three stages of creative problem solving. Nonetheless, this
framework remains a bottom-up model that only mentions high-
level cognition in the simplest of details. Yet as we have illustrated,
cognition involves top-down inuence on these bottom-up pro-
cesses. Visual analytics engages high-level cognition, and so we
emphasize that the visual design process must consider the role of
top-down inuence in such cognitive processes as chunking and
reasoning. Our leverage points, discussed next, reect this top-
down philosophy.
4.1. Leverage points
We can utilize our knowledge of the cognitive processes out-
lined above to identify ways in which those processes can be
facilitated or guided by an information visualization [80,81]. To aid
the information visualization designer, we dene a set of visuali-
zation leverage points, which are access mechanisms in the
information visualization design process which encourage a
designer to choose visualization parameters that will improve
knowledge extraction. The leverage points are derived from
knowledge of the cognitive system in Fig. 2 in order to inuence
attention and memory.
Previously, the concept of a leverage point, or touch point, has
been used to identify ways in which technology can aid intelli-
gence analysts in developing situational awareness [82]. Pirolli and
Card [83] identied several leverage points where technology
might improve sensemaking in intelligence analysts. However,
the concept of leverage point in these studies was not specic to
underlying cognitive processes nor information visualization
per se, except for one leverage point which assisted working
memory by ofoading information onto external devices. All of
our leverage points are specic to an underlying cognitive process,
or to the interaction between cognitive processes, highlighted in
Fig. 2, which further enables us to delineate the ways in which
each can be assessed using measurements of human behavioral
performance in standard cognitive psychology testing paradigms.
As we outline our six leverage points, we will suggest at least one
type of test that could be implemented to measurably gauge the
efcacy of that leverage point.
We note that in basic cognitive behavioral experiments that
may be used to assess the leverage points, response time and
accuracy are usually two primary measures of interest. Often,
a well-designed visualization should promote faster and more
accurate decisions and responses. Depending on the domain or
visual analytic task of interest to the user or designer, other
47
aspects of performance may be used, such as number of items
recalled from a visualization, order or items recalled from a series
of visualizations or animation, ratings of condence or preference,
subjective assessments of difculty or workload, etc.
Eye movements are often interpreted as a behavioral proxy for
the focus of attentional resources by assuming the observer must
be attending to that which the eyes are foveating, which can be a
useful endeavor in some perceptual research. However, we believe
that the important dynamics of visual cognition are related to
memory retrieval mechanisms, which our framework emphasizes
and which eye movements do not directly measure. Because our
focus is on top-down processing and meaning extraction, we do
not consider issues such as eye movements to be critical to our
framework as it is in other theories [28]; meaning can be extracted
from a visual display within a single glance [34,35] or over many
glances. Thus, we will emphasize the assessment of cognitive
performance through behavioral measures and metrics, leaving
neural and physiological metrics to future efforts.
We should note that several authors have proposed abstract
conceptual frameworks for conducting research on assessing the
efcacy of visualizations. For example, Elmqvist and Yi [84]
proposed a pattern-based approach to the design of such studies,
which includes study patterns (high-level patterns that deter-
mine the design of an entire study), methods patterns (mid-level
patterns that determine the design of a given experiment), and
trial patterns (low-level patterns dealing with the design of
individual tasks and trials). Lam et al. [85] proposed a scenario-
based approach to the design of visualization-evaluation studies,
which includes evaluating visual data analysis and reasoning,
evaluating user performance, evaluating user experience, eval-
uating environments and work practices, evaluating communica-
tion through visualization, automated evaluation of visualizations,
and evaluating collaborative data analysis.
Finally, in many, if not all, cases, the usefulness of a given
leverage point will depend upon the task being executed, such
that a given technique may prove superior for one task but not for
another task. (We also are not advocating using as many leverage
points as possible in a given visualization.) For example, with
respect to task dependency, Sanyal et al. [86] investigated users'
perception of uncertainty while performing two types of search
tasks (e.g., nd data points that were the least uncertain) and two
types of counting tasks (e.g., count data features). Four common
techniques were employed for representing uncertainty in 1D and
2D data (error bars, scaled size of glyphs, color-mapping on glyphs,
or color-mapping on data surface). The results showed that
differences in performance among the four techniques depended
upon the type of task being performed. Accordingly, we offer the
six leverage points below in the spirit of discovery: we believe that
these six leverage points, being derived from our reading of the
literature on human cognition, represent the strongest potential
recommendations for visualization design, but that the actual
efcacy of one or another leverage point for a given application
ultimately must be determined empirically. The rst three lever-
age points involve lower-level processes related to encoding and
perception.
Leverage point 1: capture exogenous attention utilize salient
cues to drive exogenous attention, alerting users to changes in or
important attributes of a visualization: Elements can be incorpo-
rated into visualizations, especially visualizations that change over
space or time, that engage exogenous attention because they are
different from the surrounding elements [87]. Healey and Enns
[88] suggested visualization guidelines for utilizing both color and
texture cues for target detection using multivariate data (see also
[8991]). Failure to have attention captured by a stimulus because
it is not expected and attention is directed elsewhere can lead to a
situation in which the stimulus is not consciously registered at all
48 R.E. Patterson et al. / Computers & Graphics 42 (2014) 4258
[92], a phenomenon called inattentional blindness [51,93]. Thus,
it is critical to use design elements that are difcult for an observer
to ignore to minimize the potential for inattentional blindness.
Although our framework focuses on top-down processing, we
have included Leverage Point 1, which entails a bottom-up
process, for completeness. Exogenous attention is depicted on
the left side of Fig. 2.
Because exogenous attention can provide information to the
stimulus encoding mechanisms and, in turn, the working and
long-term memory mechanisms, the visual cues can be chosen so
as to imply a reason that a visual element is noteworthy. For
example, motion can convey a change in relationship to the
observer, such as when an element appears to move closer to an
observer in a perspective display to indicate some kind of primary
status. Alternatively, an element that moves off screen could
indicate the elimination of an element/datum, or a ashing
element might indicate the existence of data points that are novel
in a new iteration of a visualization. Leverage Point 1 works not
only to engage exogenous attention but also to provide additional
memory cues.
Assessing exogenous attention cues: The success of a cue for
attracting exogenous attention can be assessed through a compar-
ison of observers performance with and without the cue added to
a visualization. As an example, an experiment could entail having
users make same/different judgments about two visualizations
on each of a set of trials. On different trials, changes in the
visualization would be cued or uncued. Response time and
accuracy could be compared across the cued and uncued trials to
see if the cue enhanced observers ability to detect the change. For
example, users could be asked to make same/different judgments
about two visualizations that would be employed for depicting
hierarchies. On different trials, one of the visualizations could be
changed by adding a new level to the hierarchy, which could be
cued via an arrow pointing to the new level, or uncued. Response
time and accuracy could be compared across the cued and uncued
trials to see if the arrow enhanced the users' ability to detect the
new level of the hierarchy.
Garlandini and Fabrikant [94] proposed a unique method for
assessing the efcacy of different stimulus features for attracting
exogenous attention. These authors assessed four commonly used
features for designing 2D maps, namely size, brightness, hue, and
orientation, within the change detection icker paradigm. This
paradigm entails having users try to detect changes in a scene
when a blank eld separates the two alternating images (i.e., the
original image alternating with the modied image), which can be
very difcult to do (this difculty is termed change blindness).
Garlandini and Fabrikant [94] found that size was the fastest and
most accurate feature for change detection, and orientation was
the slowest and least accurate. Nonetheless, the authors suggested
that all four features were attention guiding because their
participants performed signicantly above chance. The authors
argued that their results provide support for the ordering of
features as discussed by Bertin [11].
Leverage point 2: Guide endogenous attention provide appro-
priate organization of material or interaction options to assist
endogenous attention and minimize distracting information: Endo-
genous attention can be leveraged when visualizations help guide
and focus attention and minimize distractions, which would allow
the information relevant to a given task to be encoded and
maintained in working memory. The top-down allocation of
endogenous attentional resources may be assisted by using an
appropriate organization of material in the visualization (e.g.,
using clear labels for the structure of a visualization) or by giving
control of appropriate cues (e.g., arrow pointing to relevant
information) to the users, which may help them to maintain the
information in working memory. Minimizing distracter information
can be enhanced by deleting extraneous details from a visualization
(e.g., users zooming in on particular details). Endogenous attention
is depicted at the top of Fig. 2 as a component of working memory.
An example of how endogenous attention can be guided comes
from a computational modeling study by Hornof [95] who inves-
tigated how visual layout affects the cognitive strategy that people
use when searching computer displays. For this study, we take the
concept of cognitive strategy to mean the way in which endo-
genous attention was consciously and volitionally deployed. Hor-
nof reported that an unlabeled layout (items presented without
group headings) produced search times that were predicted by a
random search strategy, which we interpret as endogenous atten-
tion being deployed randomly. A labeled layout (items presented
with group headings) produced search times that were predicted
by a hierarchical search strategy (i.e., group labels systematically
searched rst, followed by group contents), which we interpret as
endogenous attention being deployed hierarchically.4 Visual
search performance in Hornof's study can be explained largely in
terms of the way in which endogenous attention was deployed to
accomplish a given task.
Assessing endogenous attention cues: Metrics that could be used for
indexing the inuence of endogenous attention would be to measure
the speed and accuracy with which a person can recognize a particular
piece of information presented with one kind of organization (e.g.,
unlabeled layout), compared to the speed and accuracy with which a
person can recognize the same information presented with a different
kind of organization (e.g., labeled layout). Such metrics, for example,
could be used in studies similar to the design used by Hornof,
described above.
Leverage point 3: facilitate chunking choose visualization para-
meters that provide strong grouping cues to facilitate the chunking of
information, which will minimize the effects of working-memory
capacity limitations: Designers should construct visualizations so
that they provide strong retrieval cues for relevant knowledge
structures in long-term memory. The activation of the knowledge
structures would then become part of working memory and be able
to exert a top-down inuence over encoding and chunking [66,67].
We note that chunking can refer to grouping information in space
or across time. In the visualization design process, chunking can be
implemented by grouping image features in space to represent
patterns over time.
There are several ways to promote chunking, such as the use of
common image parameters (e.g., color or shape) that promote
grouping by their self-similarity or the use of Gestalt principles of
grouping (e.g., good continuity, proximity, closure, common fate).
For example, in a Treemap, areas that are represented with the
same hue are likely to be interpreted as related. Because chunking
refers to the tendency to group elements into larger units based on
their meaning or learned associations, the use of image features to
assist in chunking should take meaning into account; that is, the
image features of the visualization should provide retrieval cues
for long-term memory of the knowledge domain in question.
Working memory (which is involved in chunking) is shown at the
top of Fig. 2.
Assessing chunking: Metrics that could be used for indexing the
occurrence of chunking would be to measure the accuracy of the
contents of chunks, or the number of chunks, derived from a
visualization. Verbal protocol methods based on the work of Ericsson
and Simon [96] could be employed. Or experiments might assess
speed or accuracy of responses to associations of information between
or within chunks. One example of the use of color and continuity to
4
A single target-only layout produced a strategy in which the eyes moved to
the sudden appearance of the target, which we interpret as an action of exogenous
attention.
 R.E. Patterson et al. / Computers & Graphics 42 (2014) 4258 49

Fig. 3. The ThemeRiver visualization (top) was developed as an alternative to the stacked bar chart (bottom) for showing trends over time. Havre et al. [97] showed that the
ThemeRiver successfully promoted better identication of macro trends in time series data, which is a consequence of their strong use of color and continuity cues for
chunking information. See text for a more thorough discussion of how this supports leverage point 3.

promote chunking can be found in ThemeRiver visualization techni-
que [97], shown in the top of Fig. 3. The ThemeRiver visualizes the
dynamic changes in the semantic content of a set of documents by
using color to code each theme and then continuous curves to show
the prevalence of a theme at each point in time. Compared to a
stacked bar chart of the same data (Fig. 3, bottom), the use of good
continuity with the curves enabled users to more easily group the
colors into macro trends and to follow the trends over time, measured
by verbal protocol reports.
The next three leverage points involve reasoning, analogical
reasoning, and implicit learning, which are relatively high-level
cognitive processes. One may ask whether models of high-level
cognition in the psychological literature are sufciently validated
and sufciently predictive to support a visualization design and
evaluation program. Our answer is that there is abundant research
by cognitive psychologists that relate to Leverage points 46 (see
citations given in the previous sections) such that we feel
comfortable in making the suggestions offered in the following
paragraphs. Not surprisingly, the following suggestions will need
to be tested empirically in future evaluation studies.
Leverage point 4: aid reasoning with mental models organize
information based on mental models so as to provide strong retrieval
cues for knowledge structures in long-term memory to aid reasoning.
Reasoning is the ability to draw inferences and conclusions in
order to solve problems. Reasoning involves an interaction among
knowledge representations in long-term memory as they become
activated in working memory, under control of the Central Execu-
tive of working memory. There have been a number of theoretical
approaches attempting to explain the structure of these knowl-
edge representations, including (1) a form of mental logic from
which inferences are drawn [98]; (2) domain-specic rules of
inference relating to social exchange [99]; (3) the mental compu-
tation of probabilities [100]; or (4) mental models of imagined
possibilities from which inferences are derived [101]. Long-term
memory is shown in the middle of Fig. 2; working memory is
depicted at the top of Fig. 2.
These theories are controversial. Consistently across the exam-
inations of these theories, there does seem to be substantial
support for the role of mental models in human reasoning
[102,103]. A mental model refers to an organized knowledge
structure that involves imagined possibilities and projection
[102104] which is similar to the idea of sensemaking [83]. In
the literature on expertise development, mental models do seem
to play an important role in providing a framework upon which
50 R.E. Patterson et al. / Computers & Graphics 42 (2014) 4258
expertise is built [105107]. Thus, a well-designed visualization
could serve as a retrieval cue for the mental model of a given
knowledge structure in long-term memory and thus exert a top-
down inuence over encoding and chunking.
The domain-specic mental models of expert individuals can
be studied and captured by certain methods of cognitive task
analysis (CTA) [104,108,109]. Klein and Hoffman [104] noted that
CTA methods for capturing mental models include having the
participant think aloud during a solution to a problem or submit to
an interview afterwards. Such think-aloud protocols can entail the
expert describing reasons for making a given decision or for
adopting a particular strategy. Subject-matter experts may be
asked to identify one of the several alternative models as closest
to their beliefs, or to judge which of several alternatives would be
solved with a strategy or technique most similar to a target
problem. Experts may also be asked to adopt a particular mental
model while completing a typical decision making task while an
investigator observes results.
Klein and Hoffman [104] also suggested that different kinds of
relationships are represented by a range of mental models, which
may require various techniques to best be elucidated by the CTA
process. Conceptual relationships can be represented with a
concept map, while activity relationships are better represented
by a cause-effect diagram or process diagram. Spatial relationships
are captured well by a topological diagram or map, and device
activity can be effectively shown by a cause-effect sequence,
diagram, or stories. Temporal relationships are captured well in a
script, while organizational relationships are highlighted by a
wiring diagram showing roles and functions. Dependency rela-
tionships are well-illustrated with a plan or storyboard. These
different ways of representing various kinds of relationships
provide clues as to the form that visualizations might take when
serving as retrieval cues for mental models.
We acknowledge that some researchers consider asking people
to describe their mental models to be a highly problematic
experimental method given that it is entirely subjective. None-
theless, we believe that the process of deriving mental models
using structured techniques, such as CTA, is sufciently developed
in the literature on reasoning and sensemaking (e.g. [102,103])
that it is worthwhile to try to leverage the concept when designing
information visualizations. Accordingly, Tversky et al. [110] have
proposed what they call the Congruence Principle. The Congru-
ence Principle states that the content and format of a visualization
should correspond to the content and format of the concepts/
mental model to be conveyed. For example, Zacks et al. [111]
showed that people conceive events as goal-directed sequences of
discrete steps. This would be a form of temporal chunking of
events based on meaning, which would be related to the user's
mental model. This is perhaps one reason why animations are
typically not better at conveying dynamic information than
equivalent still diagrams; the animations are presented in con-
tinuous time while people's mental models of events are repre-
sented in discrete steps [110]. Zacks et al. [111] present methods
for uncovering the mental models of dynamic events of indivi-
duals, such as having an individual segment videos of events into
separate units.
Another approach for aiding reasoning with mental models is
to employ color-concept associations for effective discrimination
of categories, such as using red for anger or green for money
[112]. Lin and colleagues proposed the term semantically reso-
nant to refer to colors that evoke a given association with the
physical appearance of objects, common metaphors, or other
linguistic or cultural conventions. These authors introduced an
algorithm for automatic selection of semantically resonant colors
for representing data for discrimination in a controlled study. The
results showed that both expert-chosen and algorithm-derived
semantically resonant colors improved speed equally on chart
reading tasks compared to a standard palette.
Borgo et al. [113] investigated whether adding visual embellish-
ments (e.g., adding pictures or icons to data sets) to data
visualizations aided memorization, visual search, and concept
comprehension. A dual-task paradigm was employed wherein a
secondary task acted to divide attention which presumably made
the effects of visual embellishments more observable. The authors
found that visual embellishments improved memory for the
information depicted in the visualizations, yet the embellishments
also slowed the speed of visual search.
We believe that there are many studies in the information
visualization literature that address the issue of mental models,
but they do so only implicitly. That is, we believe that the
recommendations made by many authors serve to enhance users'
mental models, or are congruent with users' mental models, but
the authors themselves do not acknowledge this. For example,
Plaisant et al. [114] presented guidelines for using a novel tree
browser that include the use of semantic zooming, maximizing the
number of levels opened at any one time, maintaining landmarks
to help orient users, and using data-aware zooming controls (i.e.,
keep relevant data in view). All of these suggestions which
collectively would serve to maximize the amount of meaningful,
relevant information being displayed should enhance the users'
mental models. Other techniques for congruence and enhancing
users' mental models would include the use of space-time cubes
for representing complex spatio-temporal patterns [115], and the
use of an arrow-based visualization for depicting off-screen
objects [116].
Interestingly, Stasko et al. [117] examined the efcacy of two
visualization tools for depicting hierarchies, the Treemap method
(rectangular) versus the Sunburst method (circular), as shown in
Fig. 4. Performance trends were better, and preferences higher,
with the Sunburst method. Participants reported that the Sunburst
method was better for conveying structure and hierarchy because
they are explicitly depicted in the Sunburst but only implied by the
nesting in the Treemap. Thus, the Sunburst is more compatible
with the traditional node-link hierarchy, which is the hierarchical
structure mental model of many naive observers, with the addi-
tional space-saving benets of the circular layout.
Liu and Stasko [118] have also emphasized the role of mental
models when individuals interact with information visualizations.
These authors assume that reasoning should be understood as a
process of constructing and simulating a mental model. They also
posit that interaction with an external visualization is analogous to
mental simulation by the user; the external visualization serves to
augment the user's internal mental model. Finally, Liu and Stasko
argue that interaction between an external visualization and the
user's mental model serves three purposes: (1) external anchoring,
(2) information foraging; and (3) cognitive ofoading. We agree
with many of the ideas presented by these authors. Nonetheless,
our framework is different from Liu and Stasko in that we discuss
in more detail more of the cognitive processes that may be
involved in information visualization (e.g., attention, chunking,
implicit learning) and we also present six leverage points that may
enhance the design of information visualizations.
Assessing reasoning with mental models: One metric that could
be used for indexing reasoning would be to measure the degree in
which a person's self-described mental model adheres to the
organization by which a given visualization is designed or deviates
from the mental model suggested by the visualization. If queries
are available in a visualization, another metric that could be used
for indexing reasoning would be to measure the number of queries
that users employ before they can answer questions of inference
based on the material at some given level of accuracy. Tversky
et al. [119] suggested that faster reaction times can indicate that a
 R.E. Patterson et al. / Computers & Graphics 42 (2014) 4258 51

Fig. 4. Two examples of space-lling visualizations representing hierarchical data. The top visualization is a Treemap; the bottom visualization is a Sunburst. Stasko et al. [117]
showed that performance trends were better and preferences higher with the Sunburst method, and we speculate this is because the Sunburst more accurately maps to the
participants' mental models of hierarchical data. See text for a more thorough discussion of how this supports leverage point 4.

given format of a visualization is congruent with the format of an
individual's mental model.
Leverage point 5: aid analogical reasoning structure information
so as to provide strong retrieval cues for knowledge structures
(mental models) to aid in analogical reasoning: Analogical reasoning
involves the transfer of inferences from a relationship of elements
in one domain (the analogue) to a relationship of elements in
another domain (the target) [120123]. It can be argued that the
analogue serves as a mental model for the target. Analogical
reasoning involves an interaction among knowledge representa-
tions of the analog in long-term memory as they become activated
in working memory, under control of the Central Executive of
working memory. Analogical reasoning is one of the more funda-
mental thinking skills possessed by humans, enabling us to
generalize knowledge and experience to new situations [124,125].
Here, we are proposing that visualizations be created that have
analogous patterns in common with users' mental models; see
below. Long-term memory is shown in the middle of Fig. 2; working
memory is depicted at the top of Fig. 2.
Many individuals use analogies for reasoning provided that the
connection between analogue and target is recognized. Visualiza-
tions that make analogous situations more explicit can help users to
realize the embedded relationships via long-term memory retrieval
or the similarity of patterns across domains by consistent repre-
sentations that trigger pattern recognition. Pedone et al. [126]
found that most individuals made mental connections between a
cancer radiation treatment problem (target) and a fortress defense
problem (analogue) when a simple animation display was used to
convey moving lines converging onto a target (see also [127,128]).
The animation served as a retrieval cue for the solution to the
fortress problem (analogue) represented in long-term memory,
which was then be applied to the cancer radiation problem (target).
52 R.E. Patterson et al. / Computers & Graphics 42 (2014) 4258
Burkhard and Meier [129] reported an evaluation of an informa-
tion visualization called the Tube Map Visualization (Fig. 5) as an
alternative to a traditional Gantt chart. This visualization was a
visual metaphor for communicating the creation of a long-term
quality development process in an educational center for health
care professionals. The Tube Map Visualization depicted the devel-
opment of the project; each tube line represented a target group
and each station represented a given milestone. The visualization
was printed as a poster and displayed at prominent locations in the
organization. The results of a questionnaire showed that the Tube
Map Visualization was a powerful metaphor to communicate a
complex project to different target groups. The Tube Map Visualiza-
tion is an example of analogical reasoning because metaphors can
be considered analogies if they share primarily relational informa-
tion [130], as is the case in this visualization.
Assessing analogical reasoning: Analogical reasoning can be
indexed by the speed or accuracy of performance when users are
asked to complete an analogy without using any visualization
versus when asked to complete the analogy while using a
visualization (e.g., as Pedone et al. [126] had done with their
animation display). Alternatively, the generality of a visualization
to novel data can be probed by allowing users to interact with it
under one data/information scenario, and then assessing their
performance on a series of questions about a second scenario
using the same visualization.
Leverage point 6: encourage implicit learning develop training
regimes for implicitly learning about statistical regularities within a
visualization: Implicit learning refers to the process of learning
without being able to verbalize easily what has been learned
[131,132]. Aslin et al. [133], Fiser and Aslin [134,135], Saffran et al.
[136], Turk-Browne et al. [137,138], and Patterson et al. [139] have
shown that individuals can easily learn about statistical regula-
rities. Thus, visualizations could be developed for training pattern
recognition via implicit learning by presenting statistical regula-
rities to users over a series of trials [43,139]. The precise nature of
the patterns used for training would depend upon the task domain
of interest. Such patterns can involve geometric shapes or letters
seen simultaneously across multiple trials, or objects or letters
seen sequentially within each trial, for a number of trials. Follow-
ing learning, the encoded visual representation for the statistical
pattern would provide the retrieval cue for the implicitly learned
representations in long-term procedural memory, which would
provide the top-down component for the recognition.
To the degree that implicit learning has not been fully
exploited by the visualization community, we see leverage point
6 as a call for a program of research on statistical learning that
speaks to visualization design. This may be especially important
given that both the analytical and intuitive processes likely play a
role in cognitive functioning and decision making. The pattern
recognition process is shown at the bottom of Fig. 2; long-term
memory (including procedural memory) is depicted in the middle
of Fig. 2.
Assessing cues that promote implicit learning: One metric that
could be used for indexing the advantage of implicit learning would
be to compare the performance (e.g., accuracy, speed) of different
groups of individuals on a given visualization task with and without
training. For example, imagine a hypothetical task in which users
were required to detect a given vehicle that was going to be parked
on a city street in a simulated scenario of activity in a given city.
Suppose that the vehicle always appeared within a given context
(e.g., same time of day but different street; or same street but different
time of day). Having the users view the scenario repeatedly over time
would help them pick up important cues (i.e., statistical regularities)
for detecting the presence of the target vehicle. Such learning could
occur without the users being consciously aware of the cues that they
learned (it would be implicit).
4.2. Efforts to automate the visualization design process
Our human cognition-based visualization leverage points can
be related to efforts by other researchers to nd principled ways of
automating the visualization design process. Mackinlay [13] out-
lined a presentation tool (APT) for automating the visualization
process with the intent to exploit the capabilities of the human
visual system using expressiveness and effectiveness criteria. The
leverage points outlined above could be used to establish efcacy
criteria through the various metrics of performance suggested
here to determine if the visualization exploits the capabilities of
the human cognitive system as well as the human visual system.
Howe and colleagues [140,141] suggested that the impediment
to understanding data is no longer the lack of visualizations and
difculty in acquiring data but the enormous amount of data and
the difcultly in visualizing it for understanding. Their Vizdeck
software uses preprocessing of the data and, based on their
ndings, presents possible visualizations to the analyst using a
thumbnail/card deck style dashboard that quickly allows the user to
scroll through suggested visualizations without trying to automa-
tically determine an optimal visualization. The thumbnails
encourage pattern recognition and the deployment of endogenous
attention to the examination of the solution space by the user.
This allows chunking of information in the visualization, which
relates to our Leverage point 3, and may suggest further avenues of
research.
Gotz and Wen [142] generated visualization recommendations
by monitoring user behavior. Their analysis/visualization prototype,
Harvest, collected user interaction patterns and matched them to
dened patterns. Based on the matches, Harvest recommended
visualization alternatives which resulted in improved efciency and
lower error rates on specic tasks. This pattern recognition model
ties in to leverage point 4, aid reasoning with mental models, and
leverage point 6, encourage implicit learning. The collection of user-
interaction patterns with the visualization task can be assessed to
see the kind of mental models users typically use and visualizations
can be adjusted to take advantage of the ndings.
5. Implementation: a cognitively enhanced visualization
design process
Fig. 6 illustrates a cognitively enhanced visualization design
process that highly integrates the user's cognition. Engagement
with the visualization leverage points throughout this process
plays a key role in the design of the visualization as well as
serving in the role of the consumer of the visualization. Rather
than just ending at the user (like Fig. 1), the user's cognitive
processes are now also central to the design process, shown at
the top of Fig. 6. In the human cognition framework-guided
visualization design process, the user is the central player. The
designer is guided by the user's cognition, through considering
his/her typical characteristics, intended task, cognitive capabil-
ities and state, all possibly determined via consulting the litera-
ture, or perhaps through direct interaction where she/he can
supply direct experience and domain knowledge. The designer
then denes abstract transformations that consume data and
produce a visualization instance. This visual output (e.g., visual
features on a display screen) [143] is then presented to the user
for consideration and cognitive processing. Feedback from the
user (e.g., based on performance measures or subjective ratings)
then would lead to potential renement of the visualization. Note
that this dynamic design process could occur over hours, days,
weeks, or months (or more). Critically, the designer is responsible
for employing knowledge about human cognition throughout the
visualization development process. Engagement by the designer
 R.E. Patterson et al. / Computers & Graphics 42 (2014) 4258 53

Fig. 5. The top visualization is a Gantt chart of a project; the bottom is a representation of the same project using a subway map as a visual metaphor for the representation
of information. See text for a more thorough discussion of how this supports leverage point 5. Bottom image reproduced from [129].

with our leverage points can improve the process as she/he
explicitly and systematically works through the cognitive pro-
cesses that will be engaged by the user for a specic task.
Additionally, when employing knowledge about human cogni-
tion in the design of a visualization, it is important for the
designer to work with subject-matter experts to fully understand
the tasks that will be undertaken within a certain domain and/or
working with particular types of data.
When the initial design of a visualization is insufcient, one
may need to rene it via an iterative process of redesign and
repeated testing. Accordingly, Zacks and Tversky [144] noted that
there are two ways to design humancomputer interfaces: top-
down and bottom-up.5 The top-down approach applies general
principles derived from cognitive research to a specic interface.
The bottom-up approach systematically varies the parameters of a
given task to determine their optimal values, and the design of the
interface is rened by iterative testing. Zacks and Tversky noted
that, because it sometimes may be difcult to make decisions
about specic designs guided by general cognitive principles, the
use of a hybrid method may be more useful: the top-down
5
Critically, these two approaches to designing a visualization should not be
confused with bottom-up and top-down cognitive processing.
54 R.E. Patterson et al. / Computers & Graphics 42 (2014) 4258

Fig. 6. Visualization design process. The design process is conceptualized with the user's cognition as being essential to both ends of information visualization: cognition
serves in the role of consumer of a given visualization as well as being the primary force driving its design. This is a human cognition-centered design process. Note that this
dynamic design process could occur over hours, days, weeks, or months (or more).

principles are used to suggest promising directions and the change in the motion of the dots around T 470 serves to engage
bottom-up approach then renes specic cases. Our leverage Leverage Point 1 Capture Exogenous Attention. At one point the
points provide some guidance in a similar hybrid fashion, but, visualization, the dots move much faster and in a non-random
importantly, they suggest ways to measure the efcacy of parti- pattern to exit the building, which is very distinct from the
cular choices or renements. animation frames prior to that time. This signals a change in the
human behavior represented by the dots, and an analyst can infer
that this demarcates an important event.
Thus, the change in patterns of motion also facilitates temporal
6. Visual Analytics Science and Technology challenge case
chunking (leverage point 3 facilitate chunking). The domain
study
knowledge that a hypothetical IED was detonated inside a building
together with patterns of motion allowed the viewer to temporally
We can explore the application of our leverage points with one of
chunk the events into four groupings: pre-detonation (few dots
the Visual Analytics Science and Technology (VAST) Challenges from
moved in different directions at the beginning of the simulation),
2008 [145].6 In the 2008 VAST Mini-Challenge 4 Evacuation Traces,
the person placing the IED (the dot that stopped behind a wall),
the participants were given a hypothetical scenario that involved an
detonation (sudden mass exodus of most dots out of the building),
improvised explosive device (IED) placed and detonated in a govern-
and the location of the IED (near dots that never moved pre-
ment building. Radio-frequency identication (RFID) technology
sumably because they represented people close to the detonation
was active in the building at the time of the attack which provided
who died).
a time history containing the location of each person in the
Leverage point 4 aid reasoning with mental models is engaged
building. The participants in the challenge created visualizations
throughout the visualization by the use of the static building map.
from the raw time series data of each RFID's location at different
The animated dots are portrayed on an overhead blueprint-like view
points in time so that the location of the IED and the RFID of the
of the building, which provided a natural, familiar representation
individual responsible for detonating it could be ascertained.
(mental model) of the scenario environment, facilitating reasoning
A dynamic visualization solution submitted by Simeone and
about human actions within a building. The schematic of the building
Paolo [146] utilized an animation coupled with a top-down view
and the pattern of the dots over time served as a retrieval cue which
of a schematic of the hypothetical building. Screenshots from their
triggered the long-term memory of emergency evacuation plans that
solution are shown in Fig. 7, illustrating four key time points in the
can be applied to the new scenario.
sequence of events. At the beginning of the animation, a few dots
moved in different directions, and one of the dots appeared to stop
behind a wall. At another point in time, most of the dots suddenly
7. Other cognitive modeling frameworks
appeared to exit out of the building, including the one who
stopped behind the wall. In the nal time segment, a few dots
Our human cognition framework for information visualization
located in one area of the building never moved while all others
can be compared to other cognitive modeling frameworks. One
were outside the building.
framework is that offered by Scaife and Rogers [147]. In their
The information from this challenge could have been repre-
review of the literature on graphical representation, these authors
sented in a number of ways (text document, temporal display, etc.)
emphasized the importance of the interaction between external
but the dynamic visualization in this particular example success-
representations and internal mental representations when carry-
fully utilizes several visualization leverage points. First, the distinct
ing out different cognitive tasks, a perspective they termed
external cognition. These authors abstracted three central charac-
6
The VAST Challenge is an annual event sponsored by members of the Visual teristics from which to explain external cognition: computational
Analytics Community, and it culminates at the IEEE VAST Symposium during IEEE ofoading (an external representation reduces cognitive effort),
VisWeek. The objective of this activity is to present large hypothetical data sets to representation (different external representations with the same
the community and challenge community members to create visualizations that
enhance an understanding of the data set and solve complex problems presented
abstract structure affect problem-solving), and graphical con-
as scenarios. For more information on the VAST Challenge and related data sets, the straining (graphical representation constrains inferences). Our
reader is referred to http://www.cs.umd.edu/hcil/vastchallenge/. framework for information visualization emphasizes top-down
 R.E. Patterson et al. / Computers & Graphics 42 (2014) 4258
processing whereas bottom-up processing would be a reection of
Scaife and Rogers external cognition idea, and both bottom-up and
55
top-down processing would be involved in Scaife and Rogers'
internal mental representation.
Another perspective is the computational modeling framework
based on production-system architectures [148], such as the
Executive-Process/Interactive Control (EPIC) model [149] and the
Adaptive Control of Thought-Rational (ACT-R) model [150]. Both
EPIC and ACT-R are computational cognitive architectures that
consist of a set of interconnected processors or modules that
simulate various parts of human information processing (e.g.,
perceptual-motor and memory). Our framework is different from
these models in that our framework is not a computational model
of cognition. Consequently, our framework can be agnostic when it
comes to the debate between the symbolic versus connectionist
approach to modeling cognition. EPIC and ACT-R belong to the
symbolic approach for which it is assumed that cognition consists
of symbolic representations that can be manipulated according to
algorithms. The connectionist approach, on the other hand,
assumes that cognition emerges from the activity of intercon-
nected networks of simple elements. We note that although our
framework is mute as to these arguments, it is likely that any of
these computational cognitive models might be instantiated so as
to model the visual analytics process, which might then be use to
test visualizations for specic tasks, but such work is beyond our
present scope.
8. Concluding remarks
In our human cognition framework for information visualization,
we conjecture that well-designed visualizations should engage and
promote high-level cognitive functioning, such as gaining insight,
reasoning, and understanding.
A well-designed visualization attracts attention to important
features of the display. This serves to minimize the potential for
inattentional blindness and the user ignoring important informa-
tion. Well-designed visualizations also focus endogenous attention
on task-relevant goals and minimize distractions that detract
attentional resources from the visual analytics processes.
A well-designed visualization promotes chunking and shapes
encoding by serving as retrieval cues for knowledge representa-
tions (e.g., mental models) in long-term memory. Interactions with
the visualization enable users to reorganize the relevant details for
further encoding or scrutiny. Serving as long-term memory retrie-
val cues also supports reasoning, thinking, and decision making.
Well-designed visualizations may present patterns in a training
regime, inducing implicit learning and creating the foundation for
procedural memory.
In our view, we cannot offer a simple, exhaustive list of visual
attributes that engender good visualization design which promote
high-level cognition like reasoning, because good design will be
content-domain specic. Instead, we offer our leverage points to
provide a way to measurably link human cognition to information
visualization, with an emphasis on top-down inuence.
Fig. 7. Screenshots from Simeone and Paolo [146] proposed solution to the VAST
2008 Mini-Challenge 4 Evacuation Traces. The full dynamic visualization depicted
movements of individuals, represented with their unique associated radio-fre-
quency identications (RFIDs), as green dots around a building oor plan over a
time period spanning the placement and detonation of a hypothetical improvised
explosive device (IED) in the building. The images illustrate the four time periods
indicated by changes in the movements of the dots: beginning (T 0), time of IED
placement (T 394) where the highlighted dot stops behind a wall, immediately
following IED detonation (T 472), and end of simulation (T 674). This visualiza-
tion illustrates use of several visualization leverage points, as discussed in the text.
(For interpretation of the references to color in this gure caption, the reader is
referred to the web version of this article.)
56 R.E. Patterson et al. / Computers & Graphics 42 (2014) 4258
Conceptualizing visualizations as engaging and promoting
high-level cognition places great emphasis on the need to under-
stand the characteristics of high-level knowledge structures in
human cognition. This is a very difcult enterprise that will
require much research effort beyond the suggestions given in this
paper. But we believe that our framework serves an important role
in explicitly articulating fundamental connections between visua-
lization and such high-level cognitive processes, together with the
ways to assess the efcacy of visualization design choices via the
measurement of human performance.
References
[1] Ward M, Grinstein G, Keim D. Interactive data visualization: foundations,
techniques, and applications. Natick, MA: A.K Peters, Ltd.; 2010.
[2] Hearst M. Search user interfaces. Cambridge: Cambridge University Press;
2009.
[3] Card SK, Mackinlay JD, Shneiderman B. Readings in information visualiza-
tion: using vision to think. San Francisco: Morgan Kaufmann; 1999.
[4] Heer J, Bostock M, Ogievetsky V. A tour through the visualization zoo.
Commun ACM 2010;53(6):5967.
[5] Patterson R. Cognitive engineering, cognitive augmentation, and information
display. J Soc Inf Disp 2012;20(4):20813.
[6] Nielsen J. Heuristic evaluation. Usability Insp Methods 1994;17:2562.
[7] Norman D. The design of everyday things. New York: Basic Books; 1988.
[8] Shneiderman B, Plaisant C. Designing the user interface: strategies for
effective humancomputer interaction, 4th ed. (New Edition). India: Pearson
Education; 2003.
[9] Huang W, Eades P, Hong SH. Measuring effectiveness of graph visualizations:
a cognitive load perspective. Inf Vis 2009;8(3):13952.
[10] Tory M, Moller T. Human factors in visualization research. IEEE Trans Vis
Comput Graph 2004;10(1):7284.
[11] Bertin J. Semiology of graphics: diagrams, networks, maps (WJ Berg, Trans.).
Madison: University of Wisconsin Press; 1983 (Original work published
1973).
[12] Evans JSBT. Dual-processing accounts of reasoning, judgment, and social
cognition. Annu Rev Psychol 2008;59:25578.
[13] Mackinlay J. Automating the design of graphical presentations of relational
information. ACM Trans Graph (TOG) 1986;5(2):11041.
[14] Wehrend S, Lewis C. A problem-oriented classication of visualization
techniques. In: Proceedings of the 1st conference on visualization'90. 1990.
p. 13943.
[15] Roth SF, Lucas P, Senn JA, Gomberg CC, Burks MB, Stroffolino PJ, et al. Visage:
a user interface environment for exploring information. In: Proceedings of
the IEEE symposium on information visualization '96. IEEE; 1996. p. 312,
116.
[16] Keller PR, Keller MM. Visual cues: practical data visualization. Los Alamitos,
CA: IEEE Computer Society Press; 1993.
[17] Shneiderman B. The eyes have it: a task by data type taxonomy for
information visualizations. In: IEEE symposium on visual languages. IEEE;
1996. p. 33643.
[18] Casner SM. Task-analytic approach to the automated design of graphic
presentations. ACM Trans Graph (TOG) 1991;10(2):11151.
[19] Trickett S, Trafton J. Toward a comprehensive model of graph comprehen-
sion: making the case for spatial cognition. Lect Notes Comput Sci: Diagr
Represent Inference 2006;4045:286300.
[20] Zacks J, Levy E, Tversky B, Schiano DJ. Reading bar graphs: effects of extraneous
depth cues and graphical context. J Exp Psychol: Appl 1998;4(2):11938.
[21] Zacks J, Tversky B. Bars and lines: a study of graphic communication. Mem
Cognit 1999;27(6):10739.
[22] Tversky B, Zacks J, Lee P, Heiser J. Lines, blobs, crosses and arrows:
diagrammatic communication with schematic gures. Theory Appl Diagr
2000:389408.
[23] North C. Toward measuring visualization insight. IEEE Comput Graph Appl
2006;26(3):69.
[24] van Wijk JJ. Views on visualization. IEEE Trans Vis Comput Graph 2006;12
(4):1000433.
[25] Chen M, Janicke H. An information-theoretic framework for visualization.
IEEE Trans Vis Comput Graph 2010;16(6):120615.
[26] Elting LS, Martin CG, Cantor SB, Rubenstein EB. Inuence of data
display formats on physician investigators' decisions to stop clinical
trials: prospective trial with repeated measures. Br Med J 1999;318(7197):
152731.
[27] Woods DD. The cognitive engineering of problem representations. In: Weir
GRS, Alty JL, editors. Humancomputer interaction and complex systems.
London: Academic Press; 1991. p. 16988.
[28] Ware C. Information visualization: perception for design. 2nd ed.. San
Francisco, CA: Elsevier; 2004.
[29] Blake R, Sekuler R. Perception. New York: McGraw-Hill; 2005.
[30] Weisstein N, Harris CS. Visual detection of line segments: an object-
superiority effect. Science 1974;186(4165):7525.
[31] Weisstein N, Williams MC, Harris CS. Depth, connectedness, and structural
relevance in the object-superiority effect: line segments are harder to see in
atter patterns. Perception 1982;11(1):517.
[32] Enns JT, Rensink RA. Inuence of scene-based properties on visual search.
Science 1990;247(4943):7213.
[33] Grill-Spector K, Kanwisher N. Visual recognition as soon as you know it is
there, you know what it is. Psychol Sci 2005;16(2):15260.
[34] Oliva A, Torralba A. Building the gist of a scene: the role of global image
features in recognition. Prog Brain Res 2006;155:2336.
[35] Thorpe S, Fize D, Marlot C. Speed of processing in the human visual system.
Nature 1996;381(6582):5202.
[36] Biederman I. Perceiving real-world scenes. Science 1972;177(4043):7780.
[37] Biederman I, Glass AL, Stacy EW. Searching for objects in real-world scenes. J
Exp Psychol 1973;97(1):227.
[38] Biederman I. On the semantics of a glance at a scene. In: Kubovy M,
Pomerantz J, editors. Perceptual organization; 1981. p. 21353.
[39] Mandler JM, Ritchey GH. Long-term memory for pictures. J Exp Psychol:
Hum Learn Mem 1977;3(4):38696.
[40] Gernsbacher MA. Surface information loss in comprehension. Cognit Psychol
1985;17(3):32463.
[41] Hogarth RM. Educating intuition. Chicago: University of Chicago Press; 2001.
[42] Hogarth RM. Deciding analytically or trusting your intuition? The advantages
and disadvantages of analytic and intuitive thought. UPF Economics Working
Paper No. 654 2002.
[43] Patterson R, Pierce B, Bell HH, Andrews D, Winterbottom M. Training robust
decision making in immersive environments. J Cognit Eng Decis Mak 2009;3
(4):33161.
[44] Sloman SA. The empirical case for two systems of reasoning. Psychol Bull
1996;119(1):322.
[45] Evans JSB, Stanovich KE. Dual-process theories of higher cognition: advan-
cing the debate. Perspect Psychol Sci 2013;8(3):22341.
[46] Betsch T, Glockner A. Intuition in judgment and decision making: extensive
thinking without effort. Psychol Inq 2010;21:27994.
[47] Reyna VF, Brainerd CJ. Dual processes in decision making and developmental
neuroscience: a fuzzy-trace model. Dev Rev 2011;31:180206.
[48] Kosslyn SM. Graph design for the eye and mind. Oxford: Oxford University
Press; 2006.
[49] Egeth HE, Yantis S. Visual attention: control, representation, and time course.
Ann Rev Psychol 1997;48(1):26997.
[50] Rosenholtz R. A simple saliency model predicts a number of motion popout
phenomena. Vis Res 1999;39(19):315763.
[51] Simons DJ. Attentional capture and inattentional blindness. Trends Cognit Sci
2000;4(4):14755.
[52] Awh E, Vogel EK, Oh SH. Interactions between attention and working
memory. Neuroscience 2006;139(1):2018.
[53] Duncan J, Humphreys GW. Visual search and stimulus similarity. Psychol Rev
1989;96(3):433.
[54] Awh E, Jonides J, Reuter-Lorenz P. Rehearsal in spatial working memory. J
Exp Psychol: Hum Percept Perform 1998;24(3):78090.
[55] Bisley JW, Goldberg ME. Neuronal activity in the lateral intraparietal area
and spatial attention. Science 2003;299(5603):816.
[56] Kane MJ, Engle RW. Working-memory capacity and the control of attention:
the contributions of goal neglect, response competition, and task set to
stroop interference. J Exp Psychol: Gen 2003;132(1):4770.
[57] Wickens CD. Multiple resources and mental workload. Hum Factors 2008;50
(3):44955.
[58] Baddeley AD. Working memory. Oxford: Oxford University Press; 1986.
[59] Baddeley A. Working memory: looking back and looking forward. Nat Rev
Neurosci 2003;4(10):82939.
[60] Baddeley AD, Logie RH. Working memory: the multi-component model. In:
Miyake A, Shah P, editors. Models of working memory. New York: Cambridge
University Press; 1999. p. 2861.
[61] Miyake A, Shah P. Toward unied theories of working memory: emerging
general consensus, unresolved theoretical issues, and future research direc-
tions. In: Miyake A, Shah P, editors. Models of working memory. New York:
Cambridge University Press; 1999. p. 44281.
[62] Miller GA. The magical number seven, plus or minus two: some limits on our
capacity for processing information. Psychol Rev 1956;63(2):8197.
[63] Cowan N. The magical number 4 in short-term memory: a reconsideration of
mental storage capacity. Behav Brain Sci 2001;24(01):87114.
[64] De Groot AD. Thought and choice in chess. Mouton: The Hague; 1965.
[65] Chase WG, Simon HA. Perception in chess. Cognit Psychol 1973;4(1):5581.
[66] Cowan N. Attention and memory: an integrated framework. Oxford: Oxford
University Press; 1995.
[67] Rose NS, Myerson J, Roediger III HL, Hale S. Similarities and differences
between working memory and long-term memory: evidence from the
levels-of-processing span task. J Exp Psychol: Learn, Mem, Cognit 2010;36
(2):47183.
[68] Goldin SE. Memory for the ordinary: typicality effects in chess memory. J Exp
Psychol: Hum Learn Mem 1978;4(6):60516.
[69] Egan DE, Schwartz BJ. Chunking in recall of symbolic drawings. Mem Cognit
1979;7(2):14958.
[70] Ferrari V, Didierjean A, Marmeche E. Dynamic perception in chess. Q J Exp
Psychol 2006;59(2):397410.
[71] Klein GA. Sources of power: how people make decisions. Cambridge, MA:
The MIT Press; 1999.
 R.E. Patterson et al. / Computers & Graphics 42 (2014) 4258
[72] Klein G. Naturalistic decision making. Hum Factors 2008;50(3):45660.
[73] Lopes LL, Oden GC. The rationality of intelligence. In: Rationality and
reasoning. Amsterdam: Rodopi; 1991. p. 199223.
[74] Patterson RE, Pierce BJ, Bell HH, Klein G. Implicit learning, tacit knowledge,
expertise development, and naturalistic decision making. J Cognit Eng Decis
Mak 2010;4(4):289303.
[75] Squire LR. Memory systems of the brain: a brief history and current
perspective. Neurobiol Learn Mem 2004;82(3):1717.
[76] Squire LR. Memory and brain systems: 19692009. J Neurosci 2009;29
(41):127116.
[77] Knowlton BJ, Squire LR. Articial grammar learning depends on implicit
acquisition of both abstract and exemplar-specic information. J Exp
Psychol: Learn, Mem, Cognit 1996;22(1):16981.
[78] Knowlton BJ, Ramus SJ, Squire LR. Intact articial grammar learning in
amnesia: dissociation of classication learning and explicit memory for
specic instances. Psychol Sci 1992;3(3):1729.
[79] Woodman GF, Carlisle NB, Reinhart RM. Where do we store the memory
representations that guide attention? J Vis 2013;13(3).
[80] Thomas JJ, Cook KA. A visual analytics agenda. IEEE Comput Graph Appl
2006;26(1):103.
[81] Fisher B, Green TM, Arias-Hernndez R. Visual analytics as a translational
cognitive science. Top Cognit Sci 2011;3:60925.
[82] Jenkins MP, Bisantz AM. Identication of human-interaction touch points for
intelligence analysis information fusion systems. In: 2011 Proceedings of the
14th international conference on information fusion (FUSION). IEEE; 2011. p.
18.
[83] Pirolli P, Card S. The sensemaking process and leverage points for analyst
technology as identied through cognitive task analysis. In: Proceedings of
the international conference on intelligence analysis, vol. 2005; 2005. p. 24.
[84] Elmqvist N, Yi JS. Patterns for visualization evaluation. In: Proceedings of the
2012 BELIV workshop: beyond time and errors-novel evaluation methods for
visualization. ACM; 2012. p. 12.
[85] Lam H, Bertini E, Isenberg P, Plaisant C, Carpendale S. Seven guiding
scenarios for information visualization evaluation. Technical Report 2011-
992-04. University of Calgary; 2011.
[86] Sanyal J, Zhang S, Bhattacharya G, Amburn P, Moorhead R. A user study to
compare four uncertainty visualization methods for 1d and 2d datasets. IEEE
Trans Vis Comput Graph 2009;15(6):120918.
[87] Healey CG, Enns JT. Attention and visual memory in visualization and
computer graphics. IEEE Trans Vis Comput Graph 2012;18(7):117088.
[88] Healey CG, Enns JT. Large datasets at a glance: combining textures and
colors in scientic visualization. IEEE Trans Vis Comput Graph 1999;5(2):14567.
[89] Ware C. Color sequences for univariate maps: theory, experiments and
principles. IEEE Comput Graph Appl 1988;8(5):419.
[90] Bartram L, Ware C, Calvert T. Filtering and integrating visual information
with motion. Inf Vis 2002;1(1):6679.
[91] Lum EB, Stompel A, Ma KL. Using motion to illustrate static 3d shape-kinetic
visualization. IEEE Trans Vis Comput Graph 2003;9(2):11526.
[92] Neisser U, Becklen R. Selective looking: attending to visually specied
events. Cognit Psychol 1975;7(4):48094.
[93] Mack A, Rock I. Inattentional blindness. Cambridge, MA: The MIT press;
1998.
[94] Garlandini S, Fabrikant SI. Evaluating the effectiveness and efciency of
visual variables for geographic information visualization. In: Spatial informa-
tion theory. Heidelberg, Germany: Springer; 2009. p. 195211.
[95] Hornof AJ. Cognitive strategies for the visual search of hierarchical computer
displays. HumComput Interact 2004;19(3):183223.
[96] Ericsson KA, Simon HA. Protocol analysis: verbal reports as data. Cambridge,
Mass: MIT Press; 1984.
[97] Havre S, Hetzler B, Nowell L. ThemRiver: visualizing theme changes over
time. In: IEEE symposium on information visualization (InfoVis 2000). IEEE;
2000. p. 11523.
[98] O'Brien D. Human reasoning requires a mental logic. Behav Brain Sci
2009;32: 96-97.
[99] Cosmides L, Tooby J, Fiddick L, Bryant GA. Detecting cheaters. Trends Cognit
Sci 2005;9:5056.
[100] Oaksford M, Chater N. The probabilistic approach to human reasoning.
Trends Cognit Sci 2001;5(8):34957.
[101] Johnson-Laird PN, Byrne RMJ. Conditionals: a theory of meaning, pragmatics,
and inference. Psychol Rev 2002;109(4):64678.
[102] Byrne RMJ, Johnson-Laird P. If and the problems of conditional reasoning.
Trends Cognit Sci 2009;13(7):2827.
[103] Johnson-Laird PN. Mental models and human reasoning. Proc Natl Acad Sci
2010;107(43):1824350.
[104] Klein G, Hoffman RR. Macrocognition, mental models, and cognitive task
analysis methodology. Nat Decis-Mak Macrocognit 2008:5781.
[105] Chi MTH, Feltovich PJ, Glaser R. Categorization and representation of physics
problems by experts and novices. Cognit Sci 1981;5(2):12152.
[106] Feltovich PJ, Coulson RL, Spiro RJ, Dawson-Saunders BK, Knowledge applica-
tion and transfer for complex tasks in 111-structured domains: implications
for instruction and testing in biomedicine. In: Evans DA, Patel VL, editors.
Advanced models of cognition for medical training and practice, vol. 97 of
NATO ASI series F: computer and systems sciences; 1992. p. 21344.
[107] Lesgold A, Rubinson H, Feltovich P, Glaser R, Klopfer D, Wang Y. Expertise in a
complex skill: diagnosing x-ray pictures. In: Chi MT, Glaser R, Farr MJ, editors.
The nature of expertise. Hillsdale, NJ: Erlbaum; 1988. p. 31142.
57
[108] Crandall B, Klein GA, Hoffman RR. Working minds: a practitioner's guide to
cognitive task analysis. Cambridge, MA: The MIT Press; 2006.
[109] Hoffman RR, Militello LG. Perspectives on cognitive task analysis: historical
origins and modern communities of practice. New York: Psychology Press;
2009.
[110] Tversky B, Morrison JB, Betrancourt M. Animation: can it facilitate?. Int J
HumComput Stud 2002;57(4):24762.
[111] Zacks JM, Tversky B, Iyer G. Perceiving, remembering, and communicating
structure in events. J Exp Psychol: Gener 2001;130(1):29.
[112] Lin S, Fortuna J, Kulkarni C, Stone M, Heer J. Selecting semantically resonant
colors for data visualization. In: Preim B, Rheingans P, Theisel H, editors.
Eurographics conference on visualization (EuroVis), vol. 23; 2013.
[113] Borgo R, Abdul-Rahman A, Mohamed F, Grant PW, Reppa I, Floridi L, et al. An
empirical study on using visual embellishments in visualization. IEEE Trans
Vis Comput Graph 2012;18(12):275968.
[114] Plaisant C, Grosjean J, Bederson BB. Spacetree: supporting exploration in
large node link tree, design evolution and empirical evaluation. In: IEEE
symposium on information visualization (INFOVIS 2002). IEEE; 2002. p. 57
64.
[115] Kristensson PO, Dahlback N, Anundi D, Bjornstad M, Gillberg H, Haraldsson J,
et al. An evaluation of space time cube representation of spatiotemporal
patterns. IEEE Trans Vis Comput Graph 2009;15(4):696702.
[116] Henze N, Boll S. Evaluation of an off-screen visualization for magic lens and
dynamic peephole interfaces. In: Proceedings of the 12th international
conference on human computer interaction with mobile devices and ser-
vices. ACM; 2010, p. 1914.
[117] Stasko J, Catrambone R, Guzdial M, McDonald K. An evaluation of space-
lling information visualizations for depicting hierarchical structures. Int J
HumComput Stud 2000;53(5):66394.
[118] Liu Z, Stasko JT. Mental models, visual reasoning and interaction in informa-
tion visualization: a top-down perspective. IEEE Trans Vis Comput Graph
2010;16:9991008.
[119] Tversky B, Agrawala M, Heiser J, Lee P, Hanrahan P, Phan D, et al. Cognitive
design principles for automated generation of visualizations. In: Allen G,
editor. Applied spatial cognition: from research to cognitive technology.
Mahwah, N.J.: Erlbaum; 2007. p. 5375.
[120] Holland JH, Holyoak KJ, Nisbett RE, Thagard PR. Induction: processes of
inference, learning, and discovery. Computational models of cognition and
perception. Cambridge, MA: The MIT Press; 1986.
[121] Holyoak KJ, Gentner D, Kokinov BN. Introduction: the place of analogy in
cognition. In: Gentner KD, Holyoak KJ, Kokinov BN, editors. The analogical
mind. Cambridge, MA: The MIT Press; 2001. p. 119.
[122] Juthe A. Argument by analogy. Argumentation 2005;19(1):127.
[123] Markman AB, Moreau CP. Analogy and analogical comparison in choice. In:
Gentner D, Holyoak KJ, Kokinov BN, editors. The analogical mind. Cambridge,
MA: The MIT Press; 2001. p. 36399.
[124] Hofstadter DR. Epilogue: analogy as the core of cognition. In: Gentner D,
Holyoak KJ, Kokinov BN, editors. The analogical mind. Cambridge, MA: The
MIT Press; 2001. p. 499538.
[125] Klein GA. Applications of analogical reasoning. Metaphor Symb 1987;2
(3):20118.
[126] Pedone R, Hummel JE, Holyoak KJ. The use of diagrams in analogical problem
solving. Mem Cognit 2001;29(2):21421.
[127] Gick ML, Holyoak KJ. Analogical problem solving. Cognit Psychol 1980;12
(3):30655.
[128] Gick ML, Holyoak KJ. Schema induction and analogical transfer. Cognit
Psychol 1983;15(1):138.
[129] Burkhard RA, Meier M. Tube map visualization: evaluation of a novel
knowledge visualization application for the transfer of knowledge in long-
term projects. J Univ Comput Sci 2005;11(4):47394.
[130] Gentner D, Bowdle BF, Wolff P, Boronat C. Metaphor is like analogy. In:
Gentner D, Holyoak K, Kokinov B, editors. The Analogical Mind. Cambridge,
Mass: The MIT Press; 2001.
[131] Cleeremans A, Destrebecqz A, Boyer M. Implicit learning: news from the
front. Trends Cognit Sci 1998;2(10):40616.
[132] Perruchet P, Pacton S. Implicit learning and statistical learning: one phenom-
enon, two approaches. Trends Cognit Sci 2006;10(5):2338.
[133] Aslin RN, Saffran JR, Newport EL. Computation of conditional probability
statistics by 8-month-old infants. Psychol Sci 1998;9(4):3214.
[134] Fiser J, Aslin RN. Unsupervised statistical learning of higher-order spatial
structures from visual scenes. Psychol Sci 2001;12(6):499504.
[135] Fiser J, Aslin RN. Statistical learning of higher-order temporal structure from
visual shape sequences. J Exp Psychol: Learn, Mem, Cognit 2002;28(3):
458.
[136] Saffran JR, Aslin RN, Newport EL. Statistical learning by 8-month-old infants.
Science 1996;274(5294):19268.
[137] Turk-Browne NB, Jung JA, Scholl BJ. The automaticity of visual statistical
learning. J Exp Psychol: Gener 2005;134(4):55264.
[138] Turk-Browne NB, Scholl BJ, Chun MM, Johnson MK. Neural evidence of
statistical learning: efcient detection of visual regularities without aware-
ness. J Cognit Neurosci 2009;21(10):193445.
[139] Patterson RE, Pierce BJ, Boydstun AS, Ramsey LM, Shannan J, Tripp L, et al.
Training intuitive decision making in a simulated real-world environment.
Hum Factors 2013;55:33345.
[140] Howe B, Key A, Perry D, Aragon C. Vizdeck: a card game metaphor for fast
visual data exploration. In: Proceedings of the 2012 ACM annual conference
58 R.E. Patterson et al. / Computers & Graphics 42 (2014) 4258
extended abstracts on human factors in computing systems. ACM; 2012.
p. 166772.
[141] Key A, Howe B, Perry D, Aragon C. Vizdeck: self-organizing dashboards for
visual analytics. In: Proceedings of the 2012 international conference on
management of data. ACM; 2012. p. 6814.
[142] Gotz D, Wen Z. Behavior-driven visualization recommendation. In: Proceed-
ings of the 14th international conference on intelligent user interfaces. ACM;
2009. p. 31524.
[143] Chi EH. A taxonomy of visualization techniques using the data state reference
model. In: IEEE symposium on information visualization. IEEE; 2000. p. 6975.
[144] Zacks JM, Tversky B. Structuring information interfaces for procedural
learning. J Exp Psychol: Appl 2003;9(2):88100.
[145] Grinstein G, Plaisant C, Laskowski S, O'connell T, Scholtz J, Whiting M. Vast
2008 challenge: introducing mini-challenges. In: IEEE symposium on visual
analytics science and technology. IEEE; 2008. p. 1956.
[146] Simeone AL, Paolo B. Evacuation traces mini challenge: user testing to obtain
consensus discovering the terrorist. In: IEEE symposium on visual analytics
science and technology. IEEE; 2008. p. 20910.
[147] Scaife M, Rogers Y. External cognition: how do graphical representations
work?. Int J HumComput Stud 1996;45:185213.
[148] Gray WD. Cognitive modeling for cognitive engineering. In: Sun R, editor.
The Cambridge handbook of computational psychology. New York: Cambridge
University Press; 2008. p. 56588.
[149] Kieras DE, Meyer DE. An overview of the EPIC architecture for cognition and
performance with application to humancomputer interaction. HumComput
Interact 1997;12(4):391438.
[150] Anderson JR, Bothell D, Byrne MD, Douglass S, Lebiere C, Qin Y. An integrated
theory of the mind. Psychol Rev 2004;111(4):103660.