Updated 11.2.

15

Including Social
Emotional Skills
and CultureClimate Surveys
in the School
Quality
Improvement
Index
Recommendations for minimum
participation rates, how to
translate responses into metric
results, and how to include in the
Index

Updated 11.5.15

Purpose of the SEL & CC Field Test

Provide actionable information to participating
districts and schools
Test drive the administration process
Inform decisions about the final instrument for SY
2015-16
Inform decisions about the method of measurement
within the Index & expectations with respect to
participation rates
Engage in analytics to inform the CORE Districts
and the field
Utilize data to inform School Quality Improvement
System Program Evaluation efforts (PACE)
Build a stronger professional network amongst
CORE District staff to engage in this work

Updated 11.5.15

Purpose of the SEL & CC Field Test

Provide actionable information to participating
districts and schools
Test drive the administration process
Inform decisions about the final instrument for SY
2015-16
TODAY
TODAY

Inform decisions about the method of measurement

within the Index & expectations with respect to
participation rates
Engage in analytics to inform the CORE Districts
and the field

TODAY

Utilize data to inform School Quality Improvement

System Program Evaluation efforts (PACE)
Build a stronger professional network amongst
CORE District staff to engage in this work

Updated 11.2.15

Context: CORE, the School Quality

Improvement Index, and our
Measures of Social-Emotional Skills
and Culture-Climate Perceptions.

Updated 11.2.15

CORE is a collaboration among 10 California

school districts.
Were working together to significantly improve
student outcomes.
Sanger Unified
Clovis Unified
Garden Grove
Unified City
Sacramento
Unified
Oakland
Unified
Santa Ana Unified
San Francisco
Unified
Fresno
Unified
Long Beach Unified

Other districts in
CA

Los Angeles Unified

Participating CORE
districts

Over
1.1
million
student
s in
CORE

Updated 11.5.15

Our School Quality

Improvement
Index
College & Career Ready

Making all
students visible:
N size of 20
resulting in over
150,000
additional
students
counted!

Graduates
Social-Emotional &
Culture-Climate
Domain
Elimination of Disparity and
Disproportionality

Academic Domain

Achievement and
Growth*
Graduation Rate*
High School Readiness
Rate (Gr. 8)*

All Students
Group &
Subgroups

Chronic Absenteeism
Student/Staff/Parent
Culture-Climate Surveys
Suspension/Expulsion
Rate*
Social Emotional Skills
ELL Re-Designation Rate*
Special Education
Disproportionality

Guiding principles:
Information as flashlight (and not a hammer)
From a narrow focus to a holistic approach
Making all students visible
From just achievement to achievement and growth
Developed through collaboration and partnership:
Led by the CORE Superintendents
Guided by the experts in our districts
With input from hundreds of educators across the CORE districts
With support from our key partners (e.g. Stanford University, Harvard
University)
With guidance from our Oversight Panel (e.g. ACSA, CSBA, Ed Trust
West, PACE, PTA)

Updated 11.5.15

Each indicator has been carefully developed, refined,

and analyzed before inclusion in the Index
Measurable
Evidence of validity, reliability and stability through the
examination of baseline and/or field test data.
Actionable
Evidence from research that schools can influence and
impact the outcome in question.
Evidence from baseline data that schools serving similar
youth demonstrate notably different outcomes (such that
there is evidence that schools play a substantive role in the
outcome).
Meaningful
Clearly connected (e.g., through research) to college and
career readiness, and the elimination of disparity and
disproportionality (e.g., based upon the current presence of
substantive gaps in performance).

Updated 11.5.15

In 2015, CORE
Districts will
release the 1st
version of the
School Quality
Improvement
Index

Reports
support
continual
improvement
for school
leaders and
teachers.

Updated 11.5.15

Results will
include
performance by
the all
students
group and
subgroups

Social
Emotional
Skills and
Climate
Surveys will
become part
of the Index in
Fall 2016

Updated 11.5.15

CORE is part of the national dialogue on including Social

Emotional Skills in Multiple Measure approaches to school
quality

O
T
S
L
L
I
K
S
L
A
I
TEACHING SOC
VE d emotional
IMPRonO
e social an
th
t
h
lig
sh
a
fl
a
We think
S
E
lay.
p
V
Were putting
I
ey
th
L
le
ro
D
e
N
th
t
ou
k abA
EinS
ls th
AoD
scho
elpR
ess but also
cc
su
skillsto hG
ic
m
e
d
ca
a
ut
o

ot only ab
school quality is n
whole child.
e
th
g
n
pi
o
el
ev
d
about

With almost half a million students participating, our Spring

2015 Field Test of measures of social-emotional skills lets us
explore how to measure these essential skills at scale.

Updated 11.5.15

CORE Field Test of Measures of Social Emotional

Learning and School Culture-Climate
More than 450,000 students participated in the Spring 2015 field test of
SEL measures
District Name
Number of Students
Fresno

34,583

Long Beach

45,342

Los Angeles

308,602

Oakland
San Francisco
Santa Ana
Total

8,386
23,249
34,136
454,298

Two districts collected teacher reports on students SE competencies from

more than 2,700 teachers, covering approximately 71,000 students
District Name
Fresno
Santa Ana
Total

Number of
Number of Students Covered by
Teachers
Teacher Reports
2,436
63,767
301
7,293
2,737
71,060

Updated 11.5.15

Social Emotional Skills Cover Four Topics Including

Inter-Personal and Intra-Personal Skills

Updated 11.5.15

Sample SEL Items

To assess social-emotional skills, we ask students about
their beliefs and behaviors. Below, for instance, are some
self-management items.
Please answer how often you did the following during the past
30 days. During the past 30 days
I came to class prepared.
I remembered and followed directions.
I got my work done right away instead of waiting until the last minute.
I paid attention, even when there were distractions.
I worked independently with focus.
I stayed calm even when others bothered or criticized me.
I allowed others to speak without interruption.
I was polite to adults and peers.
I kept my temper in check.
(Almost Never, Once in a While, Sometimes, Often, Almost All the Time)

Updated 11.5.15

Student, Staff and Family Culture-Climate Surveys cover

four topics as well.
CLIMATE OF SUPPORT FOR ACADEMIC LEARNING
Students and teachers feel that there is a climate conducive to learning and that teachers
use supportive practices, such as encouragement and constructive feedback; varied
opportunities to demonstrate knowledge and skills; support for risk-taking and
independent thinking; atmosphere conducive to dialog and questioning; academic
challenge; and individual attention to support differentiated learning.
KNOWLEDGE AND FAIRNESS OF DISCIPLINE, RULES AND NORMS
Clearly communicated rules and expectations about student and adult behavior, especially
regarding physical violence, verbal abuse or harassment, and teasing; clear and consistent
enforcement and norms for adult intervention.
SAFETY
Students and adults report feeling safe at school and around school, including feeling safe
from verbal abuse, teasing, or exclusion by others in the school.
SENSE OF BELONGING (SCHOOL CONNECTEDNESS)
A positive sense of being accepted, valued, and included, by others (teacher and peers) in
all school settings. Students and parents report feeling welcome at the school.

Updated 11.5.15

Sample Culture-Climate Items

To assess culture-climate, we ask students, staff and
families about their experiences with and perceptions of
their school. Below are sample items from the student
culture-climate survey.

How strongly do you agree or disagree with the following

statements?
Adults at this school encourage me to work hard so I can be
successful in college or at the job I choose.
My teachers work hard to help me with my schoolwork when I need it.
Teachers give students a chance to take part in classroom discussions
or activities.
This school promotes academic success for all students.
This school is a supportive and inviting place for students to learn.
Teachers go out of their way to help students.
I feel close to people at this school.
(Strongly Disagree, Disagree, Neither Disagree Nor Agree, Agree,
Strongly Agree)

Updated 11.5.15

SEL & Culture Climate: A schools culture-climate is related to

social emotional skills reports, and we see a substantive range in
school performance, despite comparable levels of youth in
Correlation
poverty.
between
overall SEL
& overall
cultureclimate is .
47.

The larger
the dot,
the higher
the
percentag
e of youth
in poverty.

Both of these
schools have
close to 90% of
youth in
poverty

Updated 11.5.15

Math & SEL: A schools SEL results are also related to

performance on other indicators, such as math.

The larger
the dot,
the higher
the
percentag
e of youth
in poverty.

In this
graph we
see that
schools with
strong SEL
generally
performed
better on
SBAC math.

Updated 11.5.15

Pause & Reflect:

What do you find
noteworthy or
interesting about what
you have heard and
seen so far? What are
you looking forward to
learning more about?
What opportunities do
you foresee in the use of
this information?

Updated 11.5.15

Recommendations and
Rationale: CORE, the School
Quality Improvement Index, and our
Measures of Social-Emotional Skills
and Culture-Climate Perceptions.

Updated 11.5.15

Minimum Participation Rate Thresholds

Students

Staff

Family

Denominator (already agreed Student enrollment

upon by the CORE Board)
count in the fall census

Number of staff members

as of the administration
date (inclusive of all
certificated and classified
staff invited to respond to
the survey)

Student enrollment count in

the fall census

Elementary School Threshold

75%

75%

40%

Middle School Threshold

75%

75%

30%

High School Threshold

75%

75%

25%

Rationale for Staff and Student Participation Thresholds

Georgia utilizes 75% participation as their threshold for accountability reporting of culture-climate staff and student
surveys, as being both reasonable to accomplish and fairly representative.
Our research partners have advised us to find a threshold that will maximize the likelihood of a representative sample,
and have also suggested that there is not a well established guideline for survey participation rates in this context.
93% of schools in the field study achieved this level of participation on the student surveys. Median participation was
87%.*
85% of schools in the field study achieved this level of participation on the staff surveys. Median participation was 71%.*
Rationale for Family Participation Thresholds
We are unaware of a state benchmark for family survey participation rate expectations. (Georgia reports any case with
fifteen or more responses).
Our research partners have advised us to find a threshold that will maximize the likelihood of a representative sample,
and have also suggested that there is not a well established guideline for survey participation rates in this context.
Median participation rates amongst designation eligible schools were 39% at elementary, 28% at middle and 22% at
high. (75th percentile participation was 54% at elementary, 41% at middle and 32% at high). **
*Based upon participation data from LAUSD, LBUSD, OUSD and SFUSD. **Based upon participation data from all waiver

Updated 11.5.15

Who is eligible to participate in the staff survey?

Recommendation: A staff member is defined as certified or
classified personnel. This includes full-time or part-time but
they must work at least 50 percent of the day in the school.
Itinerant staff may take the survey for more than one school.
Districts will provide a school by school count of the number of
staff members at each school meeting these criterion during
the survey window.
Rationale
This is comparable to the approach in Georgia.
This approach includes the voice of staff members who spend a
substantive amount of time at the school.

Updated 11.5.15

Participation rates below the minimum threshold

Recommendation: If participation is below the prescribed
percentage in the all students group and for any particular
subgroup, then respondents will be added to the denominator to get
to the participation threshold. Any added participants will be
treated as a respondent with the lowest possible scores for each
item on the survey for the purposes of scoring.
Rationale
This is comparable to how participation below the expected
threshold is treated on academic assessments.
This allows us to include results even when the participation
rate minimum has not been met.
This approach incentives schools to meet the participation
threshold.

Updated 11.5.15

Subgroup categories for SEL and Culture-Climate

Students
(SEL & Culture-Climate)

Staff

Family

All Respondents

Yes

Yes

Yes

Lowest Performing
Racial/Ethnic Subgroup

Yes

No

Yes

Students with Disabilities

Yes

No

Yes

English Learners

Yes

No

Yes

Socio-Economically
Disadvantaged Students

Yes

No

Yes

Rationale
Only in the case of staff are subgroups not applicable given that
we are not able to easily pinpoint the subgroups of students that
staff work with.

Updated 11.5.15

Weights for Social Emotional Skills and Culture-Climate Surveys

In the approved CORE Waiver, the weight of Social Emotional Skills is 8%,
and the weight of Culture-Climate Surveys is 8%. We do not recommend a
change to these weights.
If we decide to include each SEL skill as a separate metric on the Index
(e.g., separate ratings for self-management, self-efficacy, growth mindset
and social awareness), we recommend even weighting of these items (e.g.,
2% for each skill).
For Culture-Climate Surveys, we recommend the following breakdown,
keeping to the even weighting concept that has been used throughout the
Index:
-Students: 2.67%
Rationale
-Staff: 2.67%
We-Family:
do not 2.67%
see a reason to move away from the even weighting
scheme utilized for other metrics in the Index.
While one might argue that student and staff participation rates
lead to more representative data, this is the one place in the Index
that we take family input into account, so we recommend
incentivizing participation in the family survey by keeping their
weight even with students and staff.

Updated 11.5.15

Scoring of the Items to Create a Metric Result - CultureClimate (1 of 2)

We recommend including one composite Culture-Climate result on
the Index for students, staff and family, separately and respectively
(and not having a separate result for each topic). We do
recommend
reporting the results both as a composite and as
Rationale
separate
topics
in a supplementary report.
Including all
four topics separately for each of the three survey groups (students,
staff and parents) would create twelve indicators on the Index, which we feel
would be too many.
Three results for culture-climate surveys (one for student results, one for staff
results, and one for family results) is more manageable on the main reports in
the Index. From there, we intend to support users in drilling to the data to see
results by topic.
Culture-climate results by topic are generally strongly related to the composite
rating. For instance, composite ratings of student culture-climate are generally
correlated at .8 or above to topic-specific ratings. The exception is safety, which
is correlated at .54 level with the composite rating, which is still reasonably
strong.
There is a large enough range in composite culture-climate results to generate
ten levels of performance for students, family and staff. The difference between
the 10th and 90th percentile results ranges between 12 and 33 percentage points,
and is typically at least 18 percentage points such that performance thresholds
would generally be at least 2 percentage points wide..

Updated 11.5.15

Scoring of the Items to Create a Metric Result - Culture-Climate (2

of 2)
We recommend the following method for translating survey responses into a metric
result:

The results will be calculated in terms of percentage of favorable responses. Metric

results will be displayed as whole percentages without decimals (e.g., 75% favorable
and not 75.3% favorable).
By item type, favorable responses will be defined as follows (generally referring to the
top two response choices):
-Items on the agreement scale: Agree/Strongly Agree
-Items on the how significant of a problem scale: Insignificant Problem/Mild Problem
-Items on the how many adults scale: Nearly All Adults/Most Adults
-Items on the elementary culture-climate scale: Yes, Most of the Time/Yes, All of the
Time
-Items on the how many times scale (for bullying items): 0 Times
There
will be a two-step process for calculating the percentage of favorable responses
Rationale
to ensure even weighting of the topics in the composite:
-Step
Percent
favorable
a standard
communicating
culture1: Calculate
theispercentage
of method
favorableof
responses
separately
by topic.
-Step
2: Average
together
those topic-specific percentages of favorable responses.
climate
results in
the field.

The two step process for computing composite results creates even
weighting amongst the topics.

Updated 11.5.15

Scoring of the Items to Create a Metric Result SEL (1 of 2)

We recommend including separate skill by skill results for each
Social Emotional Skills result on the Index (rather than a composite
SEL score).
Rationale
Including all four skills separately is manageable from a reporting perspective, allows
us to emphasize SEL.
Like culture-climate topics, SEL skills results by topic are generally strongly related to
the composite rating. For instance, composite ratings of SEL skills are generally
correlated at .7 or above to topic-specific ratings. The exception is growth mindset,
which is correlated at a .53 level with the composite rating, which is still reasonably
strong.
In models that examine how social-emotional skills predict GPA, we looked both at how
each skill taken separately in a model predict GPA and then how the composite result
predicts GPA. The composite is a more consistent predictor on its own than any
individual skill, and there is not a discernable pattern between middle and high school
as to which skill or skills stand out as most predictive. That said, including all four skills
separately in the model explains more of the variation in GPA than looking at the
composite rating does. In other words, the skills separately provide more predictive
power than the composite SEL rating.
The range in performance on the composite measure is smaller (~0.3/5.0 between the
10th and 90th percentile schools) than skill by skill ranges (+/-0.4/5.0 with several cases
closer to 0.5/5.0), so the skill by skill approach provides greater variation in
performance amongst schools. We also generally see greater variation in subgroup
performance than all students performance.

Updated 11.5.15

Scoring of the Items to Create a Metric Result - SEL (2 of 2)

We recommend the following method for translating survey responses into a metric
result:
The results will be calculated in terms of a score from 1 to 5 for each metric result.
We will display two decimal places (e.g., 3.28) to allow the user to see the
differences in school results.
Each item is scored on a scale from 1 to 5, with 1 referring to the worst response
option and 5 to the best option. Every item on the instrument has five response
options.
There will be a two-step process for calculating the ratings to ensure even weighting
of the SEL skills in the composite:
Rationale
-Step 1: Calculate the ratings separately by skill.
-Step
Percent2:
favorable
does
not apply those
to SEL in
the way that itratings.
works for culture-climate.
Average
together
skill-specific
The five point approach keeps the maximum amount of information from the data in the indicator.
The five point approach is consistent with how field test results have been released thus far.
We explored a more complex approach, where each students rating was compared to the average
rating of peers with the same demographics and then we took the difference between each student
and their matched peer. We then tried ranking schools based upon an average of these differences.

We did see that some schools would be ranked differently using this approach, but there isnt strong evidence that this is
a reasonable approach and it creates significant complexity in a metric that is already going to be new to users.
Further, the fact the Index includes results for the all students group and four subgroup categories gives schools with
concentrations of youth in poverty, English Learners, students with disabilities, etc. the opportunity to show strength with
those subgroups relative to other schools.
We will explore a growth measure for SEL after the 2016 administration.

The two step process for computing composite results creates even weighting amongst the topics.

Updated 11.5.15

Translation of Metric Performance into Index Levels

Just as we do with other metrics in the Index, we will translate metric

results into Index Levels (and points) by setting performance thresholds for
ten levels of performance. We will use the same method for these metrics
that we have used for other indicators:
-Base Level 1 on the 10th percentile from the baseline year (Spring 2016).
-For Levels 2 to 5, ensure that spread between performance levels is
consistent
-Base Level 6 on the 50th percentile from the baseline year (Spring 2016)
-For Levels 7 to 9, ensure that spread between performance levels is
consistent
-Base Level 10 on the 90th percentile from the (Spring 2016).
Rationale
Since
the 2015 assessment was a field test, we will use Spring 2016
results
the
baseline
year.

We doas
not
see
a reason
to move
away from the approach utilized for
other metrics on the Index.
On the next page, we provide a set of preliminary thresholds based
upon the field test data.
That said, waiting for the 2016 data to finalize these aligns with the
messaging of the field test year, and will allow us to adjust based upon
minor updates to the items and impacts from the broader set of
decisions contained herein.

Updated 11.5.15

Preliminary Performance Thresholds Based upon Field Test

Data

Coming soon (in development)

Updated 11.5.15

Appendices: Relevant analyses by

our partners at the Harvard Center
for Education Policy Research

Updated 11.5.15

Appendix: Correlations between individual SE competencies

and CC domains
School-level
results for each social-emotional competency or culture/climate domain
were modestly correlated with one another, all in the expected direction
At a school level we find modest correlations between individual social emotional
competencies and culture/climate domains, suggesting that these constructs are related
while capturing different
facets of a school. Composite SEL and Culture Climate
ratings
are strongly related1to each
skill/topic.
Scale
2
3
4
5
6
7
8
9
10
11
12
13
14
1

Self-Management

1.00

Growth Mindset

0.48 1.00

Self-Efficacy, Global

0.62 0.15 1.00

Social Awareness

0.67 0.02 0.76 1.00

Self-Efficacy, Math

0.51 0.13 0.74 0.55 1.00

Self-Efficacy, ELA

0.43 0.23 0.50 0.42 0.21 1.00

Self-Efficacy, Science

0.52 0.11 0.68 0.53 0.36 0.36 1.00

Self-Efficacy, Social Studies

0.47 0.24 0.55 0.45 0.44 0.22 0.32 1.00

Mean Student SEL Measures

Student Climate of Support for
10 Academic Learning

0.93 0.53 0.81 0.76 0.64 0.53 0.63 0.59 1.00

11 Student Sense of Belonging

Student Knowledge and Fairness
12 of Discipline, Rules and Norms

0.32 0.16 0.19 0.26 0.14 0.20 0.29 0.35 0.33 0.83 1.00

13 Student Safety
Standardized Mean of Student
14 School Culture Measures

0.45 0.48 0.11 0.19 0.01 0.04 0.04 0.19 0.39 0.11 0.24 0.22 1.00

0.21 0.05 0.15 0.21 0.09 0.19 0.21 0.24 0.22 1.00

0.24 0.09 0.08 0.18 0.03 0.12 0.17 0.24 0.21 0.86 0.84 1.00

0.41 0.27 0.20 0.29 0.10 0.17 0.23 0.33 0.40 0.85 0.89 0.90 0.54 1.00

Updated 11.5.15

Appendix: Predictive power of individual SEL skills versus the

composite
The
composite is a more consistent predictor on its own than any individual skill, and there is not a
discernable pattern between middle and high school as to which skill or skills stand out as most
predictive. Including all four skills separately in the model explains more of the differences in GPA
than looking at the composite rating does.

Updated 11.5.15

Appendix: Difference between composite SEL ratings and

demographic-adjusted ratings.
The modal school would have the same ranking (on a scale from 1 to 10) with either method.
A relatively small subset of schools would receive a substantively different ranking.
Difference in Rank Between Composite
and Demographic-Adjusted School Student Mean SEL Skills
Elementary Schools

Density
0

.2

.2

Density

.4

.4

.6

.6

Difference in Rank Between Composite

and Demographic-Adjusted School Student Mean SEL Skills
Elementary Schools

-10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0

Difference in Rank

9 10

Spearman's Rho =.66

624 Schools

.5
.4
Density
.3
.2
.1
0
-10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0

Difference in Rank

3 4

5 6

7 8

9 10

Difference in Rank

Spearman's Rho =.66

624 Schools

Difference in Rank Between Composite

and Demographic-Adjusted School Student Mean SEL Skills
High Schools

Spearman's Rho =.72

187 Schools

-10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0

9 10