Brain Bases of Speech Discrimination

in Monolinguals & Bilinguals:

Developmental Perspectives
Valerie L. Shafer Yan Yu
Richard G. Schwartz Hia Datta
Arild Hestvik
Nancy Vidal
Karen Garrido-Nag
Marcin Wroblewski
Miwako Hisagi
Carol Tessel
The Graduate Center, CUNY
University of Delaware
 Acknowledgments

Supported by NIH HD 46193

 Developmental Neurolinguistics Lab

Co-investigators:
Students: Valerie Shafer
Sarita Austin Staff:
Richard Schwartz
Hia Datta Bruno Tagliaferri
Winifred Strange
Nechama Eichorn Marcin Wroblewski
Arild Hestvik
Karen Garrido-Nag Arianna Balestrieri
Tatianna Laine Elizabeth Pratt
Lisa Landskroner
Michelle MacRoy-Higgins
Jason A Rosas
Ph.D. Graduates:
Miho Sasaki
Miwako Hisagi
Margaret Shakibai
Yael Neumann
Carol Tessel
Nathan Maxfield
Baila Tropper
Kathy Kessler
Nancy Vidal
Mara Morr
Anthea Vivona
Monica Wagner
Yan H. Yu
 Outline
Specific Aims
Background - Development of speech perception
Background - General ERP methods
Experiment 1: ERP indices of speech perception
under different conditions of attention
Child monolinguals and bilinguals (Spanish-English)
Adult monolinguals and bilinguals (Spanish-English)
Experiment 2: ERP indices of speech perception in
infants and young children exposed to English only or
Spanish and English.
Discussion
 Introduction
This series of studies is part of a larger project
examining the relationship between early
neurophysiological measures of speech
perception and later language development.
Principal Hypotheses
Early input shapes automatic speech perception
Poor speech perception of English phonological
contrasts can result in poor language development
 Introduction
The research examined today will address
the following questions:
What effect does monolingual versus bilingual
input have on brain discriminative measures
in infants, children and adults?
How does attention influence processing in
monolingual versus bilingual groups?
 Background - Speech
Perception
All phonetic features that serve to distinguish
phonological segments can be differentiated
acoustically by multiple parameters that
systematically vary in value along several
spectral and temporal dimensions (Strange &
Shafer, in press).
For example
pet and bet - voice onset time (temporal)
bicker vs. bigger - closure duration (temporal)
big vs. did - F2 formant frequency (spectral)
pig vs. peg - F1 and F2 (spectral)
Background- Speech Perception

The mapping of acoustic values to

phonetic feature values is complex and
language specific.
Speech perception requires decoding the
acoustic signal on the basis of these
mappings to recover meaning (Liberman &
Mattingly, 1985).
 Background - development of
speech perception
Very young infants (under 6 months of
age) show language-general abilities.
That is, they show discrimination of almost
any phonetic contrast.
Between 6 and 12 months of age
(dependent on contrast), they stop
discriminating contrasts not found in their
native language (Werker & Tees, 1999).
E.g., Hindi dental vs. retroflex; English I vs. E
 Background - development of
speech perception
Infants learn to weight cues according to
importance in the ambient language -
(Kuhl; Curtin & Werker, 2005).
Infant learns to automatically attend to
relevant acoustic cues in the ambient
language (e.g., Jusczyk, 1997)
These abilities develop over the first few years
of life (Nittrouer & Miller, 1997)
 Background- Bilingual speech
perception
Infants exposed bilingually to Catalan and
Spanish fail to show discrimination of the e-E
Catalan contrast at 8 months (when
monolingually-exposed Catalan infants show this
ability), but do show it at 12 months of age
(Bosch & Sebastian-Galles, 2003).
Catalan and Spanish are prosodically similar, unlike
English and Spanish
Exposure to two language in infants can lead to
a different time-course of speech perception
development for each language compared to
monolingual development.
Background -Bilingual language
acquisition- adults
Late second-language (L2) learners have difficulty
perceiving L2 speech contrasts not found in their first
language (L1).
Language abilities in early bilinguals can reach
native-like proficiency in both languages (Birdsong,
2004).
-but evidence of subtle differences:
1) Bilinguals will favor one system over the other (e.g. Cutler,
et al., 1987). 2) Bilinguals may make compromises between
the two systems (e.g., Williams, 1980). 3) Bilinguals adjust
their phonemic categories to the appropriate ones based on
contextual cues (Elman, Diel & Buchwald, 1977).
 Summary-development of
speech perception
Input affects speech perception
(Jusczyk, 1997)
Infants show that they recognize
differences between pairs of sounds
that make meaning differences in
their native language.
Bilingual exposure modulates the
development of speech perception.
 Background- neurophysiology

Neurophysiological measures show that

language experience affects speech
perception at the pre-attentive level
Mismatch Negativity (MMN) automatic,
preattentive index of discrimination.
MMN is smaller and later for more difficult discriminations

Smaller and later for non-native than native

listeners (e.g., Winkler, et al. 1999; Shafer et al.,
2004).
 Background- neurophysiology
Garrido-Nag, Hisagi & Shafer, in prep.
Early L2 learners (< age 5) show clear brain
discriminative responses to L2 speech contrasts,
similar to monolinguals, and behavioral
identification with attention is excellent.
But more right hemisphere
Late L2 (> age 18) show poorer brain
discriminative responses to L2 contrasts and
poor behavioral identification. However, they
can discriminate the L2 speech sounds.
Difference is seen in neural circuits indexed by
the Mismatch Negativity.
Automatic Brain
Discrimination to
I in bid vs. E in
bed
 Background- bilingualism and
the brain
L1-dominant versus L2-dominant listeners
of Catalan and Spanish show subtle
differences in processing the /E/ phoneme,
which is found only in Catalan.
Brain responses suggest that the Spanish-
dominant bilinguals allow /E/ as a variant of /
e/, and, thus, show poorer decisions about
lexicality in a word/nonword decision task.
No comparison to monolingual groups
 Background- language
impairment
Poor speech perception is related to poor
language
SLI (Tallal; Bishop)
Early development (e.g., Trehub &
Henderson, 1996; Benasich et al. 2005)
Poor brain discriminative responses are
related to language impairment (Uwer, et
al. 2002; Shafer, et al. 2005)
 Introduction- summary
Input affects speech perception
Poor speech perception is related to poor
language development in monolinguals
What is the relationship between the nature of
input in a bilingual environment and speech
perception?
How does development of speech perception
under bilingual conditions relate to language
development?
 Overview:
Our methods & dependent measure

What are ERPs?

ERPs as a dependent measure of

language processing

General methods for all experiments

 Underlying neurobiology of
language
In the past: studied through disorders
Problem: no guarantee that the disordered
brain is like the typical
To gain a biological understanding of how
language works, we have to look at the
underlying processes in a typical brain
(Shafer & Garrido-Nag, 2007)
Brain imaging methods (e.g. fMRI, MEG,
EEG)
Measuring language processing
Motivation

Attention

Language

Input Output

Motor skills
Memory
 ERPs: Auditory and language
processing

Phonology
Syntactic
Auditory processes Reanalysis
Syntax Semantics

_________________________________________________________________
Time (ms)
0 150 300 450 600
ABR N50 P1 N1 ELAN P2 N2 P300 LRP N400 LAN P600
PN MMR LR

obligatory cognitive
responses responses
Neural Basis t0 t1 t2 t3 t4

of ERPs
t1

0 0

t0 t1 t2 t3 t4
 EEG vs. ERP

Electroencephalogram (EEG)
spontaneous brainwaves

6 Average ERP to CVC W ords CZ

4 P2 229 ms

Event Related Potentials (ERPs)

2

V 0

-2

brainwaves time-locked to stimuli of interest -4

N1 119 ms

0 20 0 40 0 60 0 80 0
milliseconds
Electroencephalogram (EEG)
Cz

Voltage

+ ENVIRONMENT Time
Event Related Potentials (ERPs)
Cz

S S S S S

6 Average ERP to CVC W ords CZ

4 P2 229 ms

V 0
Voltage
-2
N1 119 ms
-4
0 20 0 40 0 60 0 80 0
milliseconds

Time
 Event Related Potentials

Voltage fluctuations (Potentials) recorded

from the human scalp
Result of neuronal activity
Time-locked (Related) to a sensory, motor,
or cognitive stimulus (Event)
What information do we get from ERPs?

ERPs are used to index

the timeline of sensory & cognitive processing
Auditory and language processing - can give us
information on how the brain
Detects sounds (auditory processing)
Discriminates sounds (phonological processing)
Accesses and integrates words (semantic processing)
Comprehends grammar (syntactic processing)
Effect of attention on language processing
 Response attributes

Latency (time)
Amplitude (magnitude)
Morphology (waveform shape)
Topography (electrode site)
 Latency, Amplitude, Morphology

ERP to Words
+ 6
Latency

4 229 ms, 4.8 V

2 Amplitude
Morphology
V
0

Amplitude
-2
- Latency
119 ms, -2.65 V
-4
0 200 400 600 800

Time (ms)
Topography

Latency: 226 ms
 Methods specific to our studies
Goals
Examine speech processing
Investigate how brain maturation affects
speech processing
Explore possible differences between
monolingual and bilingual brains
Neural responses of interest
MMR
LR
Choice of stimuli
English Vowels /I/ - //
e.g. bit vs. bet
Contrast not present
in Spanish
Perception of this
contrast in early
Spanish-English
bilinguals
Stimuli

Re-synthesized edited natural vowel sounds [I] []. F1 increased by 25Hz steps, and F2
decreased by 30Hz steps. Formants 3 and 4 remained constant. Stimuli 3 and 9 were
used in this study.

1 2 3 4 5 6 7 8 9
I I I I I I
F1 +25 Hz 500 550 600 650
F2 -30 Hz 2160 2100 2040 1980
F3 2714 2714 2714 2714
F4 3175 3175 3175 3175
 Procedures

Task:
Listening to auditory stimuli
while attending to a video (or
toys)
EEG data acquisition
Neural activity recorded from
64 channels using Geodesic
Software 4.0
An oddball paradigm
 Mismatch Response (MMR)
IIIII
 MMR/ MMN

An automatic discriminative response to auditory sound

contrasts arranged in a oddball fashion, typically a
negative wave peaking around150-200 ms in adults
 PN, LN
Processing Negativity (PN)
Examined via the processing of the repeated or the
standard stimulus (here //)
Relative negative shift in the voltage response to
these standard sounds in attend condition of over
passive

Late Negativity (LN)

Later discriminitive component, following the MMN
Could be related to attention allocation later in time
Adults & Children
 Language processing in bilingual children

Little, if any, information available on speech processing of bilingual

children.

What we know & dont know:

L2 perception is filtered by knowledge in L1 (Best, 1995, Flege,
2002). Early bilinguals?

Multiple sensitive periods (Long, 1990) may explain variation and or

differences in L2 speech processing. When exactly?

Amount and quality of input in each language matters at least, in

terms of morphosyntax and vocabulary. And speech production/
perception?
 Questions:
How well are L2/ English sounds (vowels) represented
and discriminated by early bilingual children compared
to their monolingual cohorts?
How do monolingual and bilingual children allocate
attention in a speech perception task?
Automaticity vs. Controlled attention
Do they improve their discrimination via paying attention to their
modaility?
Non-natives, late-learners are not automatic (Strange and Shafer,
2007)

Do children show the same pattern as adults?

 Paradigm
Without Attention to targets; sounds 50 ms long

I 500 I I I 2000 I
Hz Hz

I ba I I I da I ba
With Attention to tones

I 500 I I I 2000 I 500

Hz Hz Hz

With Attention to speech

I ba I I I da I ba
 Trial count
In each attention condition:
200 deviants; 952 standards (928 in ATTEND-
Bada)
40-48 target sounds (tones or syllables)

12 blocks, 3 sets followed by break

1-4: passive condition (24 hi/low tones, 24 ba/
das)
5-8: attend to tones (48 hi/low tone targets)
9-12: attend to speech (30 ba/da targets)
Monolingual and Bilingual Childrens
Brain Activation in Vowel Perception
Nancy Vidal, Hia Datta, Valerie L. Shafer
 Participants
15 monolinguals and 15 sequential/
simultaneous early bilinguals
8-10 years old
Language Background Questionnaire (LBQ):
English/Spanish language exposure
Standardized tests results:
between -1/+1 SD in at least one language.
How well are L2/ English sounds
(vowels) represented and
discriminated by early bilingual
children compared to their
monolingual cohorts?
passive condition
presence of MMN indicating
automatic auditory discrimination
Passive (watching a movie)

Monolingual Bilingual
 Results-I

Can bilingually exposed Spanish-English

children process these English Vowels?
Yes !!
Both groups showed robust discriminative
components, although different in latency from the
prototypical adult MMN

Are there any differences in the distribution of

these potentials across the hemispheres?
Passive (watching a movie)

450ms 450ms

Monolinguals Bilinguals
How do both monolingual and
bilingual children allocate
attention in a speech perception
task?

passive s. attention conditions

processing negativity (PN)
 With Attention: Tone

Monolingual Bilingual
 With Attention: Speech

Monolingual Bilingual
 STIM*TIME*GROUP; Unweighted Means
Current effect: F(4, 112)=.18109, p=.94777
Effective hypothesis decomposition
Vertical bars denote +/- standard errors
2.0

1.5

1.0

0.5
Selected superior sites

0.0

-0.5
AMP: V
DV_1

-1.0

-1.5

-2.0

-2.5

-3.0

-3.5
Dev
STIM
1
TIME: 200 2 4 600 TIME: 200 2 4 600 STIM
2Stan
1 3 5 1 3 5
ms ms
GROUP: bilingual GROUP: monolingual
 Results-II (a)
Attention does not appear to affect groups
differently
No enhancement of discriminative component
in attention conditions in either group
So, attention does not help the bilinguals
to process better as predicted
Differences in the distribution of these
potentials across the hemispheres?
 With Attention: Tone

483ms 463ms
Monolinguals
Bilinguals
 With Attention: Speech

567 ms 543 ms
Monolinguals
Bilinguals
 Processing Negativity (PN)
TIME*ATTN*GROUP; Unweighted Means
Current effect: F(6, 168)=.92606, p=.47771 ATTEND-Bada
Effective hypothesis decomposition ATTEND-Tone
Vertical bars denote 0.95 confidence intervals PASSIVE
4.0

3.5

3.0

2.5

2.0

1.5

1.0
mV, Cz

0.5

0.0

-0.5

-1.0

-1.5
TIME: 120 200 TIME: 120 200
80 160 80 160
GROUP: bilingual GROUP: monolingual
 Results-II (b)
PN for both groups

No differences between monolingual and

bilingual children

Overall auditory attention to sounds (tested via

standard //s) more in the attend conditions over
the passive conditions
 Summary
At 8-10 years of age, even with the varied range of input
with English, bilingual children can discriminate English
vowels just as well as their monolingual peers

Both groups are attending more to the attend-auditory

and speech conditions than passive

Since their neurophysiological components reflecting

discrimination look different from adults, in latency and
morphology, it is difficult to say how easily and
automatically they are discriminating these very short
vowels.
ERP data: Mismatch (MMN) or
Late Negativity ? (LN)

Adults Children
Brain indices of speech
discrimination in adults
Hia Datta1, Arild Hestvik2, Nancy Vidal2, Miwako
Hisagi3, Carol Tessel2, Marcin Wroblewski2
1CUNY Graduate Center
2University of Delaware
3MIT
 Participants
26 monolinguals (American English)
Mean age 28.6 (SD 6)
14 women
1 left hander
15 Spanish/English bilinguals
All learned English and Spanish before 5 years of
age
Mean age 30.5 (SD 7.3)
11 women
2 left handers
 RESULTS
Automaticity related:
1. MMN (100-300ms)

Attention related:
2. Processing negativity (N1)
3. Late negativity (300-500ms)
 1. MMN
No group difference in the MMN
 No topography group difference
Main effects: stimulus & electrode positions
No effect of attention or group on dev/std difference
 No time course difference
Did groups differ in the onset of the MMN?
No difference between groups, attention conditions
 Conclusion 1:
There is no group difference in the MMN
Both mono- and bilinguals show the same
brain response to the English vowel difference
There is no effect on the MMN mismatch
response of varying the attention conditions
Shows that the MMN is an automatic and
pre-attentive response to same degree in
both monolingual and bilingual speakers
 2. Processing negativity
PN: A negative shift at fronto-central sites during
N1, indexing attentional resource allocation

Cz:
2. Processing negativity
 Conclusion 2:
This means that bilinguals allocate more
attention resources during the ATTEND
vs. the PASSIVE tasks in the early
auditory processing of the stimuli;
monolinguals show no such effect.
 3. Late Negativity
An anterior negativity 300-500ms post stimulus
Related to higher-level attentional processing of deviant stimuli
after the MMN
Main effect of stimulus in both groups:
 3. Late Negativity
Question: Does Late Negativity interact
with group and attention conditions?
Temporo-spatial PCA:
Divides data into separate underlying factors
based on covariance patterns between
electrodes and time points
Temporal PCA: isolates a virtual time region
for the LN
Spatial PCA: isolates a virtual electrode
region within that temporal PCA
 3. Late Negativity
Temporo-spatial factor corresponding to
late negativity
mean ERP across
subjects/conditions:
 3. Late Negativity

STIM x GROUP:
F(1,39)=4.12,
p=.04

Means that
the difference
between deviants
and standards is
greater for bilinguals
in the LN ERP
 Conclusion 3:
Both groups exhibits a late negativity ERP
differentiating deviants vs. standards
BUT: the deviant-standard difference is
significantly greater in bilinguals than
monolinguals
Bilinguals show increased attentional
processing of deviants
 SUMMARY
(1) English-speaking monolinguals and
Spanish-English bilinguals do not differ in
automatic and pre-attentive processing of
vowel contrast
MMN, 220 ms
(2) However, bilinguals show increased
attentional resource allocation during
processing of the stimuli
PN, 110ms (effect of attending to stimuli)
LN, 300-500ms (deviantstandard difference)
 ERP indices of speech processing in infants,
toddlers and young children: evidence of
maturational changes and language experience
from 3 months to 7 years of age

Yan Yu, Karen Garrido-Nag, Hia Datta,

& Valerie L. Shafer
Background: Speech processing in monolingual
children

Kuhl, P.K. (2004)

 In bilingual speech development?

Do simultaneous bilinguals keep pace with or follow

a different developmental trajectory from their
monolingual peers?
What are the neurophysiological (as measured by
ERPs) changes associated with this?

---Not simple Yes/No answers !

(e.g., Bosch & Sebastian-Galles,2003; Burns, Yoshida, Hill,

& Werker, 2007; Sundara & Polka, 2004; Sundara,
Polka & Molnar,2008)
 ERP studies of speech perception in
bilinguals

Research examining speech processing in

children with early exposure to more than
one language indicates:

Speech processing in bilinguals is different from that

of monolinguals even when bilinguals acquire both
languages before 4-5 years of age.

(Garrido-Nag, Hisagi & Shafer, In Preperation)

ERP studies of speech perception in
monolingual babies
Developmental change 1:
Positive MMR(PMMR) vs negative MMR at superior sites
Children under 3 years of age: positive
At age 4 and above: negative (MMN)

(Garrido-Nag, Yu, Datta, Vidal, & Shafer, 2006; Yu, Garrido-Nag, Datta & Shafer,
2007)

Developmental change 2:
MMR latency
i.e., The negative MMR (MMN) has been found to decrease with latency by 11
msec/year from 4 to 10 years of age with no developmental change in MMN
amplitude. (Shafer, Morr, Kurtzberg & Kraus, 2000).

Other variables:
amplitude, topography, other components, etc.
 ERP studies of bilingual babies

ERP measures reflect changes in speech processing

influenced by language input in infant populations
(Dehaene-Lambertz & Gigla, 2004; Friederici, Friedrich,Christophe,2007; Rivera-
Gaxiola, Silva-Pereyra & Kuhl, 2005).

Very few, if any, ERP studies on development of

bilingual speech perception in child populations.
 Goals of the study:
To investigate the maturational changes of brains
discriminative responses to speech input.
To determine the effect of language experience on
speech discrimination of the English /I-E/ phoneme
contrast in monolingual and bilingual children.

Hypotheses:
Latency of brain responses will be faster with
increasing age.
Language experience will influence brain
responses.
Procedures
Task:
Listening to speech stimuli
while attending to a
childrens video or toys

No attention
manipulation!
Method: stimuli & task
Duration:
250 ms I versus E

Presented in a
modified oddball
paradigm

Lasting for upto half

an hour
 METHODS-Participants
For all children: For bilingual children:
Additional tests
case history language background
hearing screening questionnaire about both parent
oromotor exam and child,
socio-economic status language proficiency
parent language sample questionnaire for parents in both
English and Spanish,
parent I-E list
parent language sample in
Parent PPVT Spanish
Parent behavioral identification of TVIP
I-E (pig-bear, pig-peg and i-e)
PLS-3R in Spanish
child language sample
child language sample in Spanish
Rossetti Infant-Toddler Scales,
PLS-4
Bayley Scales of Infant and
Toddler Development
 Subject groups
Age Monolingual Spanish-English
Bilingual
3-5 months 9 9
6-8 months 18 19

9-16 months 13 9
17-27 months 13 10
28-47 months 8 11
4-5 years old 11 5
6-7 years old 10 9
Total N= 153 N =82 N =71
 Bilinguals
Monolinguals
3-8 months 17-27 months 28-47 months 4-7 years old

P100
 MMR responses

Difference between the response to the

standard and deviant.
Negative at superior sites for 4 years of
age to adults (MMN)
Positive at superior sites for infants and
toddlers (PMMR)
Topographic view of MMR
Latency (msec)
MMR latencies
Topographic differences: 6-month olds

As seen in monolinguals,
there is greater negativity
at the left inferior site,

but in bilinguals, both left

and right inferior sites
show equivalent
negativity here.
Topographic of MMR:4 to 7-year-olds

Differences
?
 Summary and Conclusions
Do the bilinguals and monolinguals show any
differences in detection?
NO. Both groups have comparable P100.

Do bilingual show robust brain discrimination of these

English vowel contrasts?
Yes. Both groups show robust MMR.

Do bilinguals show any differences from monolinguals

in terms of how they discriminate these sounds, as
evidenced by the brain responses?
Possibly mostly hemispheric differences, suggesting
differences in how the right and left hemisphere contributes
to speech processing.
 Further research on early speech
perception development
Examine the relationship between language
input and their ERP reponses.
Connect the ERP responses of vowel
perception with behavioral measures, using
the same stimuli from the same infants.
Compare the above measures in not only
bilingual Spanish-English to monolingual
English, but also to monolingual Spanish
group.
Richard G. Schwartz
Graduate Center
City University of New York
 Perception of Native/Non-native contrast
(I versus )
Adults
Children 8-10
Infants, Toddlers, & Children
Early Versus Late Bilinguals
Attention (Passive, Auditory, Speech)
 Good speech discrimination for monolinguals
and bilinguals at the pre-attentive, automatic
level indexed by MMN.

Early bilinguals showed greater right than left

contribution (Garrido-Nag et al.).

Later processing (LN) is different for

monolingual and bilingual adults. The LN
was enhanced in the Attend-speech
condition only for the monolinguals.
 Attention in Adults
LNBilinguals have increased
processing of ATTEND-Tone versus
ATTEND-Speech targets
Better processing for dual information streams
Or automaticity is limited with two speech
streams
PN effectbilinguals allocate more
attentional resources in ATTEND
condition than PASSIVE condition
compared to monolinguals
 Bilingual Spanish-English children
process these English vowels.
Both groups showed robust discriminative
components
Only subtle differences in the distribution of
these potentials across different electrode
sites.
Enhancement of the discriminative
component from Attend-Tone to Attend-
Speech condition
 Bilinguals and monolinguals have
comparable detection (P100).
Bilingual children show robust brain
discrimination of these English vowel
contrasts.
There are differences in how the right and
left hemispheres contribute to speech
processing. For example, in 6 month olds:
Monolinguals have greater negativity at left
inferior site
Bilinguals have equivalent left-right negativity
 Bilinguals (i.e., early L2 learners) differ from
children with language impairment in that
they show similar robust brain discriminative
responses (MMRs) to TD monolinguals
Late L2 learners show poor MMRs.
How can we tell the difference between
Language Impairment and late L2 learning?
Other brain responses may index language
impairment, but not language experience.
The T-complex indexes processing at an
early level of central auditory processing
(CAP) between 50 and 200 ms
Attenuated Ta component for SLI vs.TLD, especially at right
temporal site (T8) ( Shafer, Schwartz et al., in prep.)
 Future Directions and
Implications
Relating the proportional input of each
language to ERP responses

Early clinical identification: Using early

ERP responses to predict later
language outcomes

Examining the relationship between

attention and speech perception