BIMI SUMMER WORKSHOP 21 JUNE 2018

surveying immigrants in dark times

outline of topics
• Survey ABCs
• Trends in polling on immigration
• The National Asian American Survey
• Challenges and opportunities
survey abcs
 two types of inference

RESPONDENT POPULATION
Answers to questions
CHARACTERISTICS CHARACTERISTICS

Measurement: “How well

do questions capture
respondent Representation: “How well do
characteristics?” survey participants
(Column validity) correspond to the target
population?” (Row validity)
 the survey lifecycle
Target
population
Construct

Sampling
Measure frame
ment

Sample

Responses

Respondents
 the survey process
Research Objectives

Interview mode Sampling frame

Instrument design Sample design

Recruitment and interviewing

Data coding and cleaning

Post-survey adjustments

Data Analysis
measurement
‣ CONSTRUCT: What you want to know about.
‣ e.g., "Issue salience"
‣ MEASUREMENT: How you want to measure it.
‣ e.g., "What is the most important problem facing the nation?"
‣ RESPONSE: How you want to collect / categorize responses.
‣ e.g., "our political system," "the battle of Ragnorok," "I don't know," "I'm not sure," "I
refuse to answer that question,"
‣ EDITED RESPONSE: The data you actually analyze.
‣ e.g., delete "the battle of Ragnorok," coding errors, processing errors, etc.
representation
‣ TARGET POPULATION: Who you want to study.
‣ The U.S. adult population in 2018;
‣ SAMPLING FRAME: The universe of possible cases eligible for your study.
‣ Face-to-face: U.S. households, enumerated through counties, blocks, listed addresses, lists
of members of the household.
‣ Telephone: list of working 10-digit phone numbers.
‣ SAMPLE: The cases selected for measurement in your study.
‣ RESPONDENTS: The cases actually measured in the study.
‣ POST-SURVEY ADJUSTMENTS: Post-measurement weights to get better inference on target
population characteristics.
 total survey error approach
‣ Measurement error: observational gap btw. ideal measurement and observed response.
‣ Processing error: observational gap btw. variable construction and observed response.
‣ Coverage error: non-observational gap btw. target population and sampling frame.
‣ Sampling error: non-observational gap btw. sampling frame and the sample.
‣ Nonresponse error: non-observational gap btw. the sample and the respondent pool.
‣ Adjustment error: non-observational errors from mistakes in assigning post- survey
adjustment (e.g., weights).
polling on immigration
issue salience and events
20%
19%
18% 17%
‣ 1996: immigration law
16% 15%
14% 13% ‣ 2001: Post-9/11 bump
12% ‣ 2006: H.R. 4437 and protest
10% marches
10%
‣ 2007: failed comprehensive
8%
6% reform act.
6% 5%
‣ 2010: Arizona's S.B. 1070
4%
‣ 2014: "Gang of Eight" reform
2% proposal, DACA
0%
‣ 2017: Trump travel ban
Jan-93
Jan-94
Jan-95
Jan-96
Jan-97
Jan-98
Jan-99
Jan-00
Jan-01
Jan-02
Jan-03
Jan-04
Jan-05
Jan-06
Jan-07
Jan-08
Jan-09
Jan-10
Jan-11
Jan-12
Jan-13
Jan-14
Jan-15
Jan-16
Jan-17
issue salience and polling
900 0.20
800 0.18 Gallup issue salience
# Survey Items on Immigration/Immigrants

Most Important Problem: % Immigration

700 0.16 time series closely
600
0.14 tracks a year-to-year tab
0.12 of survey items on
500
0.10 immigrants and
400
0.08 immigration
300
0.06
200
(rho = 0.84).
0.04
100 0.02
0 0.00
1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 2013 2015
 immigration stocks and flows
45 16
Foreign-Born Population (million s)
40 14
Percent of Total Population
35 12
30
10

Percentage
25
Millions

8
20
6
15
10 4

5 2
0 0
1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 2010
Source: Census Bureau
 foreign-born and the gss sample
18% ‣ The % foreign-born in
16% the General Social
14% GSS Census Survey sample and in
12% the U.S. (per Census
10%
data) closely track each
other, except for a
8%
pronounced peak in
6%
2006.
4%
2%
0%
1977 1980 1983 1985 1987 1989 1991 1994 1998 2002 2006 2010 2014
 foreign-born and the anes sample
10% ‣ The ANES sample
9% ANES Census(FBORN) Census (FB-CTZN) tracks poorly with
8% Census figures on %
7% foreign-born adults,
6% but tracks much better
5% with Census figures on
4% % naturalized citizens.
3%
2%
1%
0%
1952 1958 1964 1968 1972 1976 1980 1984 1988 1992
surveying asian americans
21 million and rising
21

17.3

Est. 10% of US pop.

11.9
by 2050, when Asian
Americans will be the
7.3
largest share of
immigrants in the US. 3.5
0.88 1.4
0.26 0.26

19 40 19 50 19 60 19 70 19 80 19 90 20 00 20 10 20 15
 fastest growing (2000-2010)
Asian Ame rican 46%

Hisp an ic 43%

NHPI 40%

Am. Ind ian /A laskan Native 27%

African American 15%

To tal P opulatio n 10%

Wh ite 1%
 continued growth projected
Asian Am alone or in combination (millions) 35.7
30.8
25.9
21.2
19

2015 2020 2030 2040 2050

Census Bureau population projections and estimates as of July 1 2012

 geography of immigration
Latin America Asia Eur ope North America Other Asia La tin Amer ica
100% 50 0K
90%
80% 40 0K
70%
60% 30 0K
50%
40% 20 0K
30%
20% 10 0K
10%
0% 0K
1850 1960 1970 1980 1990 2000 2008- 10 20 01 20 03 20 05 20 07 20 09 20 11 20 13

Source: Census Bureau

growth in eligibility and turnout
AAPI Eligib le Voters AAPI Voter Turnout
• +1.5M newly eligible voters
btw. 2012-16 9.8
8.3
+1.14M new voters btw. 2012-

Number in millions
• 7.1
16. 6.3
4.7 5.1
• 4.7% of eligible electorate 3.4 3.9
2.8
2
• Est. more than 11M eligible
and 6M voters by 2020.
2000 2004 2008 2012 2016
Source: Current Population Survey
 23

national asian american surveys

evolution of aapi surveys
1984 IGS CA 1990s LA 2000-1
2008 NAAS 2012 NAAS 2016-7 NAAS
poll Times polls PNAAPS

LA / Orange Pre: voting

Asian + 5 MSA sample Nationally 6k+
CO, Chinese, and
Latino, (NYC, Chicago, representative completes
Filipino, engagement
surname list + Honolulu, LA, 5k+ 11 languages
Korean, Post: attitudes,
Korean SF), dual completes SE Asian, NHPI
Japanese experiences
oversample frame 6 primary gps Wh, Bk, Ltnx
Vietnamese So. Asian, Wh,
(only English (targeted zip 8 interview Community
(Asian Bk, Latinx ++
+ Spanish) RDD + list) languages partnership
languages) NSF
 25

2008 naas
• N = 5,159, mode = telephone (landline)
• Field dates = 8/18 to 10/29, 2008
• National sample and regional targets (CA, NJ/NY, “new destinations”)
• Interview languages: English + Vietnamese, Korean, Mandarin, Cantonese,
Tagalog, Japanese, Hindi (40% in non-English language)
• Sample by groups: 1,350 Chinese, 1150 Asian Indian, 719 Vietnamese, 614
Korean, 603 Filipino, 541 Japanese, and 182 “Other Asian.”
• Sampling frame: list (with nominal RDD for comparison)
2012 naas
‣ N = 6,257, mode = telephone (82% landline, 12% cell, 6% VOIP)
‣ Field dates: July 31 to October 20, 2012.
‣ Interview languages: English, Vietnamese, Korean, Cantonese, Mandarin, Hmong, Khmer,
Japanese, Tagalog, Thai, Hindi, and Spanish.
‣ National sample with oversamples of SE-Asians, NHPIs and comparison samples of whites,
blacks, Latinos.
‣ Sub-group N: 827 Asian Indians, 743 Chinese, 633 Koreans, 599 Filipinos, 537
Vietnamese, 525 Japanese, 319 Hmong, 305 Cambodians, 251 other Asians, 419 Native
Hawaiians, 152 other Pacific Islanders, 350 Whites, 309 African Americans, 308 Latinos
 27

2016 pre-election naas

‣ N = 3,882 (2,238 AAPI), Mode: telephone (72% landline, 28% cell)
‣ Field dates: August 10 to September 29, 2016
‣ Interview languages: English, Vietnamese, Korean, Cantonese, Mandarin, Hmong, Khmer,
Tagalog, Japanese, Laotian, Hindi, Spanish
‣ National sample with oversamples of SE-Asians + comparison groups
‣ Sub-groups: Cambodian (149), Chinese (352), Filipino (252), Hmong (325), Indian (307),
Japanese (175), Korean (336), Vietnamese (342)
‣ Comparison groups: NHPIs (305), Whites (456), African Americans (392), Latinos (410),
mixed race (54)
 28

2016-7 post-election naas

‣ N = 6,448 (4,393 AAPI), Mode: telephone (63% landline, 37% cell)
‣ Field dates: Nov. 10, 2016 to Mar. 2, 2017
‣ Interview languages: English, Vietnamese, Korean, Cantonese, Mandarin, Khmer, Hmong,
Japanese, Tagalog, Hindi, Urdu, Spanish
‣ National sample with oversamples of SE-Asians and So Asians + comparison groups
‣ Sub-groups: Bangladeshi (320), Cambodian (401), Chinese (475), Filipino (505) , Hmong
(351), Indian (504), Japanese (517), Korean (499), Pakistani (320), Vietnamese (501)
‣ Comparison samples: Latino (1,126), Black (401), White (408), NHPI (120)
 29

properties of the naas

‣ Sampling:
‣ Who: national, regional; "Big Six" + targeted others; NHPIs; other race/eth gps
‣ How: list, phone (dual frame), language
‣ Measurement:
‣ Replication baseline items, with within-group measures
 30

contact by political parties

White 44%
Q: As you know, the political
parties try to talk to as many
people as they can to get Black 42%

them to vote for their

candidate. Did anyone from Asian American 29%
one of the political
parties call you up or come Latino 27%
around and talk to you about
the campaign this year? NHPI 26%

Source: Fall 2016 National Asian American Survey

 31

party contact by ethnic group

ASN AM 29%
Question: As you know, the Japanese 36%
political parties try to talk to as Vietnamese 35%
many people as they can to Paki stani 35%
get them to vote for their Bangladeshi 31%
candidate. Did anyone from Indian 31%
one of the political parties call Korean 30%
you up or come around and Fi lipino 29%
talk to you about the Chi nese 26%
campaign this year?
Cambodian 23%
Hmong 16%

Source: Fall 2016 National Asian American Survey

 32

properties of the naas

‣ Sampling:
‣ Who: national, regional; "Big Six" + targeted others; NHPIs; other race/eth gps
‣ How: list, phone (dual frame), language
‣ Measurement:
‣ Replication baseline items, with within-group measures
‣ AAPI-specific measurement
 major discrimination events
Q: ”Next we would like to Unfai rl y not hired 10.4%
ask you about some
important ways that some Poorly treated by neighbors 10.5%

people have been treated

Denied prom oti on 9.2%
poorly or unfairly. Have you
ever …” Poorly treated by polic e 9.1%

33% of Asian Americans Unfai rl y fired 7.1%

experienced at least one
major discrimination event. Denied home purc hase/rent 5.7%

Source: 2017 Post-election National Asian American Survey

 day to day micro-aggressions
Assum ed good at STEM 67.4%
Q: ”We are interested in the way
you have been treated in day to Name m ispronounced 54.3%
day encounters with strangers in Assum ed to be non-Engli sh… 30.2%
the United States. In an average Recei ve poorer s ervi ce 19.6%
month, do any of the following
Assum ed not creati ve 15.4%
things happen to you?
Called nam es / insulted 14.6%
86% of Asian Americans report at Threatened / harassed 9.8%
least one experience of micro-
Treated with fear 3.8%
aggression monthly.
Treated as if dishonest 3.1%

Source: 2017 Post-election National Asian American Survey

 economic insecurity
Saving for retirement 53%
Q: “Here are some issues that
Health care costs 47%
other people have mentioned as
challenges they face. Please tell Cost of coll ege 40%

me how serious each is for you Paying off college loans 37%
and your family” (% “fairly” or Caring for the elderly 34%
“very” serious reported) Paying off medi cal debt 32%
Qual ity of schools 30%
Among Asian Americans, 80% Getti ng vi sas 30%
experienced at least one “fairly” Getti ng bullied i n schools 27%
or “very” serious challenge. z Paying off mortgages 27%
Chi ld care costs 26%
Paying off credit card debt 23%
Source: 2017 Post-election National Asian American Survey
 36

properties of the naas

‣ Sampling:
‣ Who: national, regional; "Big Six" + targeted others; NHPIs; other race/eth gps
‣ How: list, phone (dual frame), language
‣ Measurement:
‣ Replication baseline items, with within-group measures
‣ AAPI-specific measurement
‣ Comparison across groups and over time
 37

change in immigration preferences

St ron gly a gre e Ag re e somewha t Neit her Disagr ee somewh at St ron gly d isagr ee “Please tell me if you
44% agree strongly, agree
somewhat, neither
33%
agree nor disagree,
28%
disagree somewhat, or
27%
24% 25% disagree strongly …
18% 18% The U.S. should provide
15% 15%
13% 13% a path to citizenship for
8% 8% 9% people in this country
illegally.”

2008 2012 2016 (unwei ghted)

 38

intra-group commonality
Comm on race Comm on cul ture
Comm on economic i nterests Comm on poli tical interests What, if anything do
64% 66%
62%
65% 63% Asians in the United
54% 54% 55%
58% States share with one
50%
46% another? Would you say
they share … a common
36%
race? … a common
culture … common
economic interests? …
common political
interests?
2008 2012 2016 (unwei ghted)
 39

inter-group commonality
57% 59% 56% Thinking about government
54%
46%
49% services, political power and
42% 43%
38%
representation, would you
say Asian Americans have a
lot in common, some, little
in common, or nothing at all
in common with … African
Americans? … Latinos? …
Whites?
2008 2012 2016 (unwei ghted)
wi th Whites wi th Latinos wi th Blacks
challenges to surveying
declining response rates
Contact rate Cooperation rate Respon se rate • Contact rate = % of
90 households in sample where
77 79 someone was reached.
73 72
62
• Cooperation rate = % of
43 40 contacted households where
34 31 yielded an interview.
36 21
28 25 14 • Response rate = % of
21
15
9 households sampled that
yielded an interview.
1997 2000 2003 2006 2009 2012

Rates calculated using AAPOR’s CON2, COOP3 and RR3. Rates are typical for Pew Research surveys conducted in each year.
rr effect: demographics
Pew Research Government
Standard survey surveys
% %
U.S. citizen 95 92
Homeowner 63 62
Lived at current address 5+ yrs 56 59
Married 50 54
Children in household 37 37

Source: Scott Keeter, Pew Research Center.

rr effect: financial indicators
Landline households who…
Responded Did not respond
Net worth % %
$500,000+ 23 22
<$25,000 19 21

Home value
$500,000+ 7 9
<$25,000 29 26
Source: Scott Keeter, Pew Research Center.
rr effect: political indicators
Landline households who…
Responded Did not respond
Party registration % %
Republican 17 17
Democrat 23 22
Other 6 9
No record of party 54 51

Registered to vote 82 79
Voted in 2010 54 44
Source: Scott Keeter, Pew Research Center.
rr effect: civic engagement

Pew Research Government

Standard survey surveys
In the past year… % %
Volunteered for an organization 55 27
Contacted a public official 31 10
Talked with neighbors weekly or more 58 41

Source: Scott Keeter, Pew Research Center.

telephone coverage
45 45 Households with only 43.1%

40 40
wireless phones
35 35

30 Households with no phone 30 Households with no phone

25 25
19.7% 19.7%
20 20

15 15

10 10

5 5

0 0
1963 1970 1975 1980 1985- 1997 2001 Early Lat e Early Early 1963 1970 1975 1980 1985- 1997 2001 Early Lat e Early Early
1986 2003 2005 2011 2014 1986 2003 2005 2011 2014

Source: National Health Interview Survey

twilight of the landline

In 2004, 93% of Americans had a landline and only 5% were "cell-only." By 2017, only 44% have a landline and 53% are cell-only.
twilight of the landline
Race Poverty Status Education
64.8% 66.3% 55.2%
62.2%
59.0%
52.1% 52.4%
46.6% 47.4% 48.5%
50.2%

47.1%

Latinx White Black Asian Mix ed Poor Near-poor Not- poor som e H. S. H. S. / GED post H.S. B.A. or
race or less higher

Source: National Health Interview Survey, July-Dec 2016

trend away from landlines
phone today should include cell
‣ Response rates are comparable
‣ Improve coverage and contact rates
‣ Costs are higher but coming down
‣ Opportunities to improve sampling and measurement
‣ Smart phones and multi-media content
‣ Activity flags
‣ Flags for pre-paid (low income, immigrant samples)
what about web?
‣ Coverage very high now: Pew Research 2014 estimates 89% use the internet; American
Community Survey 2013 estimated 79% of adults had service at home.
‣ Cell phones help here, too: An estimated 7-10% of Americans have internet access only
through their smartphone.
‣ Bias simulations: Non-internet population very different from the rest, but not
numerous enough to create a large bias on most kinds of measures if not included in a
survey.
‣ But still mostly an opt-in panel, non-probability sample world.
 52

one more thing …

citizenship: a sensitive item?
‣ Figure shows
5.9% 6.0%
5.5% percentage of
5.2%
"allocated" responses
to ACS citizenship
question
3.0%
2.5%
2.7% 2.7% ‣ Allocation refers to
1.6% 1.6%
1.8%
2.0% imputation of values
where responses are
0.4% 0.4% 0.3% missing or otherwise
invalid.
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Source: https://www.census.gov/acs/www/methodology/sample-size-and-data-quality/item-allocation-rates/
and place of birth?
9.2% 9.1% ‣ Figure shows
8.6% 8.8%
percentage of
7.0% 7.0% "allocated" responses
6.5%
6.2% 6.2% to ACS item on US-
5.5% 5.5% 5.7%
born or foreign-born.
4.7% 4.8%
4.4%
‣ Allocation refers to
imputation of values
where responses are
missing or otherwise
invalid.
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Source: https://www.census.gov/acs/www/methodology/sample-size-and-data-quality/item-allocation-rates/
refusal rates in the Trump Era
14% ‣ Figure refusal rates for
12% the monthly Current
Population Survey
10%
‣ Overall, non-
8%
completion rates are
6% slightly higher (non-
4%
contact rates have
slightly declined in
2% this period)
0%
Jul-15 Sep-15 Nov-15 Jan- 16 Mar-16 May -16 Jul-16 Sep-16 Nov-16 Jan- 17 Mar-17 May -17
Source: https://www.census.gov/programs-surveys/cps/technical-documentation/methodology/non-response-rates.html
evidence from 2020 census pre-tests
‣ Qualitative, non-representative samples (Feb – Sept 2017)
‣ Paid incentives, recruitment through community-based organizations
‣ Pilots designed to test other 2020 content, not confidentiality concerns
‣ Researchers and field staff spontaneously reported "unusual" participant behavior
‣ Intentionally incomplete or incorrect information during household enumeration
‣ Visibly nervous and added time to explain data access and confidentiality
‣ Break-off attempts

Source: Mikelyn Meyers (2017)

from respondents
"The possibility that the Census
could give my information to
internal security and immigration
could come and arrest me terrifies
me." (Spanish language interview)
"In light of the current political situation,
the immigrants, especially the Arabs and
Mexicans would be so scared when they
see a government interviewer at their
doorsteps." (Arabic focus group)

"They say, 'Never open the door!' This

alert has been spread everywhere
now" (Korean focus group)
Source: Mikelyn Meyers (2017)
from field staff
"This may just be a sign of the times, but in
"The politics have changed everything. the recent several months … I'm being
Recently" (interviewer) asked [many] times over, does it make a
difference if I'm not a citizen? " (interviewer)

"There was a cluster of mobile homes, all Hispanic. I went

to one and I left information on the door. I could hear
them inside. I did two more interviews, and when I came
back, they were moving … It's because they were afraid of
being deported." (field staff)

Source: Mikelyn Meyers (2017)

more explicit research
‣ 29 cognitive interviews and 42 focus groups in summer 2017 and spring 2018
‣ Cognitive interviews in Spanish
‣ Focus groups in English, Spanish, Chinese, Vietnamese, Korean, Russian, Arabic
‣ Looked for 6 coded behaviors:
‣ Non-response (incl. citing worries about confidentiality)
‣ Fear (concerns about confidentiality, government)
‣ Data sharing (inter-agency sharing)
‣ Law enforcement (use by other federal agencies)
‣ Targeted groups (specific groups-at-risk mentioned)
Source: Mikelyn Meyers and Patricia Goerman (2018)
focus group analysis
Non- Data Targeted
Study Sample n Fear Law Enf.
Response Sharing Groups
English 6 2 4 3 2 0
Spanish 6 4 6 3 5 0
Chinese 6 3 1 0 1 0
Vietnamese 6 3 3 0 1 0
Korean 6 5 3 0 1 1
Russian 6 4 4 0 2 0
Arabic 6 4 2 0 0 3
TOTAL 42 25 (60%) 23 (55%) 6 (14%) 12 (29%) 4 (10%)

Source: Mikelyn Meyers and Patricia Goerman (2018)

cognitive interview analysis
Non- Data Targeted
Study Sample n Fear Law Enf.
Response Sharing Groups
CBAMS 10 0 1 4 0 0
Privacy #1 9 0 6 1 5 1
Privacy #2 10 1 7 5 5 1
TOTAL 29 1 (3%) 14 (48%) 10 (34%) 10 (34%) 2 (7%)

Source: Mikelyn Meyers and Patricia Goerman (2018)

 62

hand-picked take-aways
 63

take-aways: sampling
‣ RDD is cost-prohibitive (nationally)
‣ Coverage issues with list samples
‣ Coverage issues with language
‣ Weights are tricky (complex design effects, post-stratification due to non-response + due to
vendor list coverage)
 64

take-aways: measurement
‣ More accurate measures of key indicators (vote choice, mobilization).
‣ Context-specific measures re AAPI experiences and attitudes (discrimination, affirmative
action).
‣ Within-group (AAPI sub-groups), between-group (AAPIs to others), and over-time
comparisons.
‣ Some evidence suggests greater rates of acquiescence bias and social desirability effects
among foreign-born respondents.
‣ Language, a key to sample coverage, is also key to measurement bias.
practical take-aways
1. Cast a wide net, and let respondents self-identify their race or ethnicity. Do not give respondents
any hint of which race or ethnicity you are looking for.
2. Have a multilingual interview(er) available at first contact. Allow respondents to select which
language they are most comfortable in to complete the survey.
3. With listed samples, flag respondents by first, middle, or last name.
4. Targeting potential racial/ethnic respondents by geographic density increases your potential
subject pool and is better than a “name-only” approach.
5. If needed, stratify your sample by key age, socioeconomic, and ethnic demographic markers
to ensure your data are balanced. Within each stratum, randomly select respondents.
6. Weight your final data to the best universe estimates within each racial group.