Pictures and Words

Elinor Amit

Behavioral Decision Making: Building Approaches from Laboratory Insights

October 31th 2011

How do we think about things?...

The 3 main approaches:

Words (inner speech) (e.g., Vygotsky, 1934; Oppenheim & Dell, 2010) Pictures (mental imagery) (e.g., Kosslyn et al) Abstract, a-modal representations (e.g., Caramazza, Hillis,
Rapp, & Romani, 1990; Lambon Ralph, Graham, Patterson, & Hodges, 1999; Rapp, Hillis, & Caramazza, 1993; Tyler & Moss, 2001; Mahon & Caramazza, 2008)

How do we think about things?...

The 3 main approaches:

Words (inner speech) (e.g., Vygotsky, 1934; Oppenheim & Dell, 2010) Pictures (mental imagery) (e.g., Kosslyn et al) Abstract, a-modal representations (e.g., Caramazza, Hillis,
Rapp, & Romani, 1990; Lambon Ralph, Graham, Patterson, & Hodges, 1999; Rapp, Hillis, & Caramazza, 1993; Tyler & Moss, 2001; Mahon & Caramazza, 2008)

the two most vivid forms of working memory are mental images... and snatches of inner speech
Pinker, 2008

Apple

Red apple

Apple

Red apple

Differences between words and pictures

Medium words pictures
Dissimilar to the object: Similar to the object:

Red apple

Cognized Arbitrary Abstract Gist Simple

Perceived Not arbitrary Concrete Peripheral Complex

We will discuss today

- The association between medium and psychological distance - The implications of the medium of representation to moral judgment

We will discuss today

- The association between medium and psychological distance - The implications of the medium of representation to moral judgment

The medium/morality hypothesis How do we think about things?...

It depends
Proximal event Visual representation

Distal event

Verbal representation

Amit, Algom, & Trope, 2009

What is psychological distance?

The distance between an individual and a target. Subjective Egocentric

What is psychological distance?

Temporal: How much time separates
between the individuals present and the target event

Spatial: How far in space is the event

from the individual

Social: How different is the social

target from the individual
Bar Anan, Liberman & Trope, 2006

Why is medium associated with distance?

Pictures and words serves different cognitive functions: Words preserve the invariant & essential properties of the item for a distal use

Pictures preserve the stimulus in details for an immediate use.

Construal Level Theory (CLT)

Mental construal processes serve to traverse psychological distances and switch between perspectives.

Distal events not so much information =>

abstract representation, entailing the essence, invariant features of the referent.

Proximal events there is information =>

concrete, subordinate representation.

Trope & Liberman, 2003; 2010

Main hypothesis
Pictures and words are associated with psychological distance: People elect to represent close targets in pictures and distal targets in words. Pictures impart a feeling of proximity, whereas words impart a feeling of distance.

 cognitive processing:
- Identification - Categorization - Selective attention - Memory

Social cognition:
- Interpersonal communication - Moral judgment

Neural correlates

 cognitive processing:
- Identification - Categorization - Selective attention - Memory

Social cognition:
- Interpersonal communication - Moral judgment

Neural correlates

Speeded Identification: Social distance

football Medium: pictures, words Distance: socially near, far Task: speeded identification

soccer Socially proximal

Socially distal

Amit, Algom, & Trope, 2009, Jep: General

Speeded Identification: Social distance

650 proximal distal

Reaction Time (ms)

630

610

590

570

picture

word

F(1,13)=7.63, p < .05

Amit, Algom, & Trope, 2009, Jep: General

Speeded Identification: Spatial distance

Medium: pictures, words Distance: spatially near, far Task: speeded identification

Amit, Algom, & Trope, 2009, Jep: General

Speeded Identification: Spatial distance

700
proximal distal

680

Reaction Time (ms)

660

640

620

600

picture

word

F(1,15)=6.3, p < .05

Amit, Algom, & Trope, 2009, Jep: General

Mental travel
+ Exp.1: event related design, 11 subjects Exp.2: block design, 10 subjects

Tomorrow CHAIR
+ 10 years APPLE 20 sec

+
1 sec

Amit, Rim, Greene, & Trope, in prep

Mental travel (Harvard sample)

far>near Language regions

near>far

Visual regions
precuneus

Fusiform R BA47L calcarine

BA 21 L

Fusiform L

N=10, p =.001

Interim conclusion
Pictures are associated with proximity, whereas words are associated with distance. - This association influence performance in various cognitive tasks (e.g., identification) - People spontaneously elect to represent proximal things visually (embodied cognition), and distal things verbally.

Implications of the medium/distance association for public policy

convenience for costumers

pressure from the government

existing routes
location of main public services

paths

air pollution

cost
location of businesses bus drivers union demands

Walk a lot Wait a little

Walk a little Wait a lot

We will discuss today

- The association between medium and psychological distance - The implications of the medium of representation to moral judgment

We will discuss today

- The association between medium and psychological distance - The implications of the medium of representation to moral judgment

The medium/morality hypothesis hypothesis The medium/distance

Proximal event

Visual representation

Distal event

Verbal representation

The medium/morality hypothesis hypothesis The medium/distance

Visual representation

Proximal event

Verbal representation

Distal event

The medium/morality hypothesis hypothesis The medium/distance

Implications for behavior

Visual representation

Proximal event

Verbal representation

Distal event

If you see something

Do something!

 Pictures emergency reaction

Words not so much

Visual processing

Psychologically proximal

High emotional reaction

Verbal processing

Psychologically distal

Low emotional reaction

5 >1

Visual processing

Psychologically proximal

High emotional Emotionally-driven judgments reaction

Verbal processing

Psychologically distal

Low emotional cognitive-driven judgments reaction

5 >1

Dual-Process Moral Cognition

5 >1

Yes

Dual-Process Moral Cognition

5 >1

Yes

No

X
Greene et al, Cognition, 2009

Predictions

Visual processing deontological judgments (rely on emotional reactions)

Verbal processing utilitarian judgments (rely on controlled reaction)

Another rationale for the medium/moral judgment prediction

Pictures thinking about means (consistent with deontological moral reasoning) Words thinking about end-goals (consistent with utilitarian moral reasoning)

Exp 1: How media preferences are related to moral judgments?

L.A. Times

Wall Street journal

Exp 1: How media preferences are related to moral judgments?

10 9 8

More utilitarian

7 6 5 4 3 2 1 0 0 1 2 3 4 5 6 7

r(108) = 0.233, p = 0.011* After controlling for level of education: r(107) = .22, p = 0.018*

Exp 1: How media preferences are related to moral judgments?

10 9

8
7

More utilitarian 6
5 4 3 2 1 0 0 1 2 3 4 5 6 7

Hours/day of TV watching
R(170) = -0.18, p = 0.017* After controlling for education: r(169) = -0.18, p = 0.01**

Conclusions from Experiment 1: Media consumption preferences are associated with moral judgments tendencies: Visual deontological style Verbal utilitarian style * Cannot be accounted for by education

Exp 2: How cognitive style is related to moral judgments?

Amit & Greene, under revision

Exp 2: How cognitive style is related to moral judgments?

utilitarian
More utilitarian

p < .01 N =R(50)=.346, 50 r = .35 p = .01

Verbal - Visual Accuracy

Conclusions from Experiment 2: Cognitive style is associated with moral judgments tendencies: Visual style deontological Verbal style utilitarian

Amit & Greene, under revision

Exp 3: How visual and verbal interference affect moral judgments? Condition 1
Enemy soldiers have taken over your village.

Condition 2
You are standing near a footbridge

5 sec

Is it appropriate to smother your baby?

Is it appropriate to smother your baby?

5 sec

no..yes

no..yes

2.5 sec

Amit & Greene, under revision

Prediction: pictorial interference prevents representing the dilemma visually, thus leads to more utilitarian judgment. Verbal interference prevents representing the dilemma verbally, thus leads to more deontological judgment.

Amit & Greene, under revision

Exp 3: How visual and verbal interference affect moral judgments?

0.58 0.56 0.54

n.s

More utilitarian 0.52 0.5

0.48

0.46
0.44 0.42 0.4

picture

word Interference type

control

Amit & Greene, under revision

Conclusions from Experiment 3: Visual imagery creates more deontological judgments. No effect for words (compared to control) suggests that the default mode of thinking about moral dilemmas is visual.

Amit & Greene, under revision

Exp. 4
Subjects read mean vs. side dilemma. Than made moral permissibility judgment And reported whether they imagined the one to be killed or the five to be saved.

What did you imagine more?

The 5

5 4 3 2 1

The one

0 side mean

F(1,331) = 15.1, p < .0001

imagery

P = .000***

P = .001**

Dilemma (mean, side)

P = .000***

Moral judgment

Sobel = -2.04, p = .004

Take-home message
Intriguing implications for the differences between pictures and words - People think about proximal things in pictures and about distal things in words - They think about means in pictures and about end-goals in words - Thinking in pictures lead to deontological moral judgments, and thinking in words lead to utilitarian moral judgments.

Thanks
Joshua Greene Yaacov Trope Daniel Algom Galit Yovel Evelina Fedorenko SoYon Rim Georg Halbeisen
Nobuhito Abe Alek Chakroff Fiery Cushman Joe Paxton Steven Frenkland David Rand Regan Bernhard Ryan Halprin Sara Gottlieb Allison Gofman Rebecca Fine Warren Winter Paul Lively

Thanks!