DTC 101 Session

November 18, 2015

Welcome!
Please sign in and pick up your materials.
There is one binder for each
district/charter. This should go to the
district test coordinator or their designee.
If there is more than one staff member
from a district/charter attending, please
take one binder and then packets for other
staff.

Assessment Inventory Update

Background Information
Delaware Governor Markell launched state-wide
review of all assessments administered by the state,
district, and individual schools (3/12/15)
Goal to decrease testing burden and increase
instructional time
Result of concerns raised by various groups
parents, teachers, community, and other stakeholder
groups

Striving for an assessment system

Showing when students are excelling, when they need
support
Striking a balance and supporting student success

Overview:
Future of Assessment in Delaware
The Office of Assessment has spent approximately the
past two years (focused effort during last 6 months)
reviewing current state assessments
Through district/school feedback, observations,
performance routines, etc., replication has been noted in
the comprehensive use of assessments across the state
(state, district, school assessment redundancy)
Over next five years, the goal is to reduce total testing
time and ensure students are given meaningful
assessments while supporting instruction

State Strategy:
Assessment Inventory Process
All districts required to complete the Assessment
Inventory process across the state (Joint Resolution 2)
Delaware Department of Education (DDOE) released grant
opportunity to help fund district ability to
Complete the assessment inventory process
Include educators throughout the process
Clarification regarding grain size
Review findings and generate recommendations
Develop an action plan to execute on findings
Develop communications and community engagement
process around assessment
Grant award amounts vary based on district/school enrollment

State Strategy:
Assessment Inventory Process
Inventory:
Basic information
Use/purpose
Operation

Recommendations:
No change
Modification
Replacement
Elimination
Impact data collection
7

Assessment Inventory Phases

Phase One: Current Status

Phase Two: Develop Action Plan

Establish an
assessment strategy
Identify and train teams
of teachers to review
current assessments
Conduct
comprehensive review
of local assessments
using Assessment
Inventory

Engage with teachers,

parents, students, and
others to determine
recommendations
Recommendations for each
assessment
No change, modification,
replacement, elimination
Rationale for decision
Authority
Timing of recommendation
Action steps

Develop final plan

Assessment Inventory Phases

Phase Three: Implement Action Plan

Implement each
component of action
plan
Post on website, strive
for input and
transparency of
assessment efforts
Survey teachers
Compile data for
instructional time
gained (12/31/15)

Phase Four: Follow-up and Ongoing

Review success of
implementation and
ongoing status review
Identify successes and
areas requiring additional
focus
Plan forward for ongoing
process

Tools and Resources

DDOE provided resources

Available on the DOE assessment site

Delaware Components of a Balanced Assessment System one-pagers

Delaware Assessment Inventory Process Supplementary Guidelines
Delaware Office of Assessment completed Assessment Inventory
Presentations, templates

Assessment Inventory Tool Kit

Webinar with districts/charter schools

Assessment Inventory and Recommendations
Assessment Inventory professional development ppt
Focus Groups and Surveys

Some additional online resources to support the inventory process

Criteria for Procuring and Evaluating High-Quality Assessments:
http://www.ccsso.org/Documents/2014/CCSSO%20Criteria%20for%20High%20Qualit
y%20Assessments%2003242014.pdf
Assessment Evaluation Tool (Math):
http://achievethecore.org/content/upload/AET_Version2_Math_K-12.pdf
Assessment Evaluation Tool (ELA):
http://achievethecore.org/content/upload/AET_Version2_ELA_3-12.pdf

New resources and toolbox on the DOE assessment inventory webpage

10

Five-Year Plan
Final Feedback Request

11

Five-Year Plan

Requests for feedback at

assessment@doe.k12.de.us by November 25,
2015

12

DeSSA Science
Assessment

Building an Assessment Form

that Does It All
Measure of Student Achievement
State/Federal Accountability for schools
and districts
Adequate sampling of curriculum to
measure progress of standards
implementation over time

Assessment Transitions
Science summative assessments remain in
grades 5, 8, and 10
To adequately measure the NGSS, items must
measure at least two of the three-dimensions
(disciplinary core idea, cross cutting concept
and science and engineering practice) of the
NGSS performance expectation being tested.
Items with a singular focus are not considered
NGSS aligned.

Two phases proposed

Phase I:
Grades 5, 8, 10 summative
Phase II:
Grades 3, 4, 6, 7, 9

http://www.nationsreportcard.gov/science_2009/ict_tasks.asp

PROPOSED development plan

Phase 1
Science

Phase 2
Science

2015-16 SY

2016-17 SY

2017-18 SY

2018-19 SY

2019-20 SY

Test Structure
Research

Operational field
test grades 5, 8,
and 10

Summative

Summative

Summative

Grades 5, 8, and
10

Grades 5, 8, and
10

Grades 5, 8, and
10

Test Structure
Research

Test Development

Operational field
test grades 4, 7,
and 9

Grade-level
tests in 4, 7, and
9

Grade-level
tests in 3, 4, 6,
7, and 9

Operational field
test grades 3
and 6

THIS IS all preliminary

We are currently in the process of investigating test
development for Next Generation Science Standards.
The assessment for 2015-2016 is the same as previous
DCAS science assessment years.
There have been no definitive measures, decisions,
changes, or policies made at this time and final
development is subject to outcomes resulting from the
Assessment Inventory joint resolution signed by the
Governor.

DeSSA Social Studies

Assessment

Building an Assessment Form

that Does It All
Measure of Student Achievement
State/Federal Accountability for schools
and districts
Adequate sampling of curriculum to
measure progress of standards
implementation over time

Assessments
should mirror how social studies content
should be taught and tested in the
classroom
must be structured in a manner that
supports an evidence-based approach to
gathering the information necessary to
support a specific claim about what a
student knows and is able to do

Structure of the DE Social Studies

Standards
Each content area Civics, Geography, Economics, History
has four standards that focus on major concepts that increase in
depth as students move up the grade levels.
Within each content area are benchmarks.
Benchmarks are the assessable units of the standards - what
students should know and be able to do by the time they have
completed the highest grade in a cluster:
K-3
4-5
6-8
9-12
The benchmarks are currently tested in grades 4, 7 and once in
an End-of-Course U.S. History high school test.

Structure of the DE Social Studies

Standards
Each core discipline (Civics, Economics,
Geography, History) gives a rationale for its
inclusion, followed by the four standards for that
discipline, each with an explanation and clusterby-cluster progression of specific expectations.
For example, History Standard One revolves
around the concept of chronology. Beneath the
standard are found the expectations of students at
each grade cluster (benchmarks).
All standards in a content area (e.g. History) are
grouped together to show the skills and content
emphasized and of the progression through which
they will be taught.

How DE Social Studies Standards are

organized
K-12 Summary
Statement

K-12 overarching
standard

Grade-cluster benchmark
to be assessed

Delaware History Standards

demonstrate understanding through explanation,
interpretation, and analysis
go beyond the retention of a body of facts to an
understanding of the why and how of social studies
concepts and big ideas
expect students to analyze historical materials,
explain patterns of historical continuity and change,
develop effective research strategies, differentiate
between historical facts and historical interpretation
assessments focus on applying content, rather than
the memorization of content
students gather information, process information,
and apply information

Science and Social Studies (5 year plan)

Phase I
Social
Studies

Phase II
Social
Studies

2015-16 SY
Item and Test
Development

2016-17 SY
Field test grades 5,
8, and 10

2017-18 SY
Summative

2018-19 SY
Summative

2019-20 SY
Summative

Item Development

Grades 5, 8, and
10

Grades 5, 8, and
10

Grades 5, 8, and
10

Item and Test

Development

Item
Development
Field test grades
4, 7, and 9

Item
Item
Development
Development
Grade-level tests Grade-level tests
in 4, 7, and 9
in 3, 4, 6, 7, and 9

Item
Development

Field test grades

3, 6
Item
Development

Item
Development

DeSSA Calendar
2015-2016 Updates

DeSSA Calendar Extensions

28

Smarter Test Windows Spring 2016

Schools/districts have requested that scores be received earlier during
Spring 2016
DOE has created a process that will break Smarter window into smaller
segments and tests completed by a certain date will receive scores earlier
Student must complete both the CAT and PT portions of a content area in
order to receive a score
Schools/districts must remember the scores in ORS are not final and may
not represent what is used for State Summary and Accountability
IF CAT and PT for a content area
are completed by this date:

The score will be available in the

ORS (DeSSA Portal) by this date:

March 24

April 14

April 29

May 20

May 13

June 3

June 2 (end of test window)

June 23

29

Sub-Scores

About this Presentation

This presentation provides
background, concepts, and examples for a better understanding
sub-scores.
introduces some basic concepts in educational measurement
and testing to support appropriate uses of sub-scores.
is based on the Standards for Educational and Psychological
Testing (1999, 2014), and research studies on sub-scores.

31

Background (1)
Reporting sub-scores is more prevalent in the
educational reform movement, especially in light of the
recent legislations (Skorupski, 2005).
Policy makers, school/district administrators, educators,
college/university admission officers, and test takers are
interested in reporting sub-scores.
scores help make decisions, such as, classroom instruction,
remediation, program assignment, admission, employment, etc.
(Monaghan, 2006).

32

Background (2)
Measurement professionals concerned about reporting
sub-scores
express caution in the appropriate use of sub-scores

The Standards for Educational and Psychological Testing

(AERA, APA & NCME, 1999, 2014) recommend including
documentation of reliability and validity evidence for
reported sub-scores to stakeholders.

33

Background (3)
Sub-score could be useful when it provides an accurate
measure of the construct it purports to measure.
Educational Testing Service suggests three criteria in
reporting sub-scores (2005).
information is trustworthy (reliability/precision)
information is not redundant with more trustworthy information
information from different sources can be compared and
contrasted meaningfully (comparability)

34

Reliability and Precision

What is the concept of accuracy or precision in Classical
Test Theory (CTT)? It refers to the consistency or
reproducibility of test scores, such as test-retest, internal
consistency.
Cronbachs Alpha for
reliability coefficient:

Standard error of
measurement (SEM):

SE
=
(1 xx ' )
m

35

More Concepts (1)

In Item Response Theory (IRT), information function is
an important concept, analogous to the concept of
reliability in CTT. The information function is an indicator
of the precision with which a given ability level is
measured.
The SEM at a given ability is equal to the inverse of the
square root of information, then called conditional
standard error of measurement:
1
SE ( ) =
I ( )

36

More Concepts (2)

Reliability coefficients of 0.85 or above are typically
considered to be good measurement precision. The
magnitude of reliability coefficient could be influenced by
many factors, such as quality of items, testing conditions,
and test length.
The Spearman-brown prophecy formula can be used to
predict the change in the reliability of test scores based
on the extent to which test items are removed from or
added to the current test, assuming that the items are of
comparable quality to the existing test.

37

Spearman-Brown Prophecy
Calculation formula:

rnew

n rold
=
(n 1) rold + 1

With a known reliability coefficient, one could predict

reliability for a sub-score.
Where: rold is the known reliability coefficient
rnew is predicted reliability coefficient
n is the proportion of the original test

38

Example Reliability Coefficient

1.0
rold = 0.95

Estimated Reliability (rnew)

0.9

rold = 0.90
rold = 0.85

0.8
0.7
0.66

0.6
0.5

0.47

0.4

0.36

0.3
0.2
0.1
0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Proportion of Original Test Length

39

Example SEM
1.0

0.8

0.80

Estimated SEm

0.73

0.6

0.59

rold = 0.85

0.4

rold = 0.90
rold = 0.95

0.2

0.0
0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Proportion of Original Test Length

40

Grade 3 ELA/LIT
Stimuli
Claim

Items by Target

Content
Literary

CAT

PT

CAT

7-8

Reading

PT

Informational

Organization

Evidence

Conventions

Listening

Listening

3-4

Research

Research

Total

Total

%*

7-8

16-17

0
0

7-8

7-8

16-17

15-16

Item

%*

14-16

33-34

13

28-30

1
5

Writing

Total by Claim

1
5

13-14

8-9

8-9

19

8-9

19

2-3

8-9

19

8-9

19

43-47

43-47

%* Estimated percentage based on the min. and max number of items.

41

Grade 8 Mathematics
Claim

Content

Stimuli
CAT

1. Concept and
Procedure

Priority Cluster
Supporting
Cluster

CAT

3. Communicating Communicating
Reasoning

PT

12-15
0

2. Problem Solving Problem

4. Modeling & Data Solving
Analysis
Model & Data

Total

PT

Items by Standard
Total

12-15

38

0
4-5

4-5

13

Total by
Claim
Item %

16-20

50

0
6

2-4

8-10

25

8-10

25

0-2

8-10

25

8-10

25

32-40

32-40

42

SEM for Claim Scores

ELA/LIT (GR 3)

N. Items

Mean SEM

Reading

14-16

44.9

Writing
Listening
Research
Total Score

13
8-9
8-9
43-47

44.6
69.1
61.1
23.8

Mathematics (GR 8)

N. Items

Mean SEM

Concepts & Procedures

16-20

43.5

Problem Solving & Modeling

8-10

78.5

Communicating Reasoning

8-10

67.9

Total Score

32-40

31.2

43

Examples - Claim Scores

Grade 3 ELA/LIT

Grade 8 Mathematics

Cut score for Level 3: 2432

Sample 1:
Total score: 2487 (Level 3)
Reading
2523 (41) Above
Writing
2476 (38) At/Near
Listening 2484 (57) At/Near
Research 2458 (45) At/Near
2523 + 1.5 * 41 = 2584
2523 1.5 * 41 = 2461
Score 1.5 SE gives you
86.46 confidence

Cut score for Level 3: 2586

Sample 2:
Total score: 2560 (Level 2)
Concept
2558 (37) At/Near
Modeling 2583 (50) At/Near
Communicating 2535 (61) At/Near
2535 +1.5 * 61 = 2626.5
2535 1.5 * 61 = 2443.5
Score 1.5 SE gives you
86.46 confidence

44

Formative Assessment (1)

In 2015, the U.S. Department of Education released
information related to test durations and the need for
assessments and outcomes to be purposeful.
The result of summative assessments is the primary
indicator about student achievement toward the content
standards by the end of a school year. Test scores are
used for high-stakes accountability.
Many educators and researchers have recognized the
power of formative assessments (FA) to provide
diagnostic information for students and support
information for teachers in classroom instruction.
45

Formative Assessment (2)

The concept of using FA provides feedback to plan future
instruction is not new. However, insufficient content
knowledge and lack of ability in utilizing FA to provide
feedback often leads to inaccurate information and
misses opportunities to help students.
Smarter Balanced and member states are creating
resources for both ELA/LIT and mathematics that
connect test scores on interim assessments to
instructional resources. As part of this effort, item maps
are used to locate a students position and link it to
resources of the Digital Library.

46

Smarter Formative
and Interim Assessment
Updates

State Network of Educators

SNEs work within
Digital Library to create,
post, and review
aligned resources.
Contract to post new
resources and conduct
reviews.
Promote Digital Library
with extension /
activities that support
the sustained use of DL
by educators.

SNE in Delaware:
12-20 Delaware
educators, across
grade and content
areas
November 2015
recruit/select
December 2015
training in Digital
Library

48

Digital Library Resources

Recorded webinar as a
refresher (or for new staff)
Review how to access
the Digital Library
Navigate resources

Forums
Instructional modules
Professional Learning
modules

http://www.dcet.k12.de.us
/DDLCOnline2015/index.s
html

Digital Library Access Requires Registration

(Access for new staff
given by district/school IT)

49

Interim Assessments
Interim Comprehensive Assessments (ICAs)

mimic the administration of the end-of-year

summative Smarter assessments in
ELA/Literacy and Math

include the same item types and test

format, including performance tasks

use allows schools to predict how far along

students are in achieving the level needed
for the summative
Interim Assessment Blocks (IABs)

shorter in length

include the same item types and test

format, including performance tasks, as the
ICAs and end-of-year summative Smarter
assessments

focus on smaller sets of targets and are

more flexible to support instruction

http://www.doe.k12.de.us/Page/2433

50

Grade 3-8 IAB ELA Blocks

51

High School IAB ELA Blocks

52

Grade 3-8 IAB Math Blocks

53

High School IAB Math Blocks

54

Online training modules

55

New DeSSA Portal Resource-AVA

Assessment Viewing Application (AVA) is available on the Smarter
English Language Arts and Mathematics page.
AVA is a component of DeSSA that allows authorized users to view
Smarter interim assessments for administrative or instructional
purposes.
To access AVA, users must first complete DeSSA Entry Training
number 24246.
Once training is completed, up to 2 days is required for the Training
Flag to be set in TIDE.
For more information, please refer to the AVA User Guide.

Important Note: AVA security is the same as for ICAs, IABs. Not
secure, but not public.
http://de.portal.airast.org/smarter/

56

DeSSA Portal Resources

AVA User Guide
Interims Test
Administration Manual
Teacher Hand-Scoring
System User Guide

http://de.portal.airast.org/smarter/
Updated 2015-16

57

Site Visits
Districts and Charters

58

Purpose of the visits

The Office of Assessment Staff will visit each Delaware school during the
2015-2016 school year to offer support, information, and assistance for a
smooth implementation of DeSSA

All districts will receive three visits during the school year and each school
will be visited at least once
- Visits will occur during Nov/Dec, Jan/Feb, April/May

The fall (November/December) and winter (January/February) visits will

provide an opportunity for district and school test coordinators to ask
questions and will also focus on assessment related topics such as: Testing
Security Plans; Test Schedules; Assessment Inventories; Training
Updates; and DeSSA Supports

In the spring (April/May), Assessment team members will visit during the
testing process
Additional information about these visits will be shared at the next District Test
Coordinator meeting

59

Feedback from initial visits

Approximately 12 schools across two districts have been visited thus
far
Additional visits have been scheduled
Each DTC and STC receives a folder with updated documents and
information and meets briefly with the Assessment team member
Visits have been well-received
Thank you to DTCs and STCs for scheduling efforts
Please feel free to share any additional comments/suggestions to
help us support your DeSSA assessment efforts

60

Accessibility and Accommodations

Updates

61

Accessibility Guidelines
Documents Release
Accessibility Guidelines Manual without
Appendices
Appendix A-1: DeSSA Accessibility Form
Appendix A-2: DeSSA Accessibility Form
Definitions
Appendix B: DCAS-Alt1 Accommodations
Form
Released by November 20, 2015:
All other Appendices
Accessibility Guidelines Manual with Appendices
62

Accessibility Guidelines
Training Sessions
Webinar recording
Estimated launch date: early December
Three different modules
SwD and SwD/ELL
ELL and SwD/ELL
General Education students with Supports

Follow-up Live Webinars for Q&A

Estimate launch date: mid December and
again in early February
63

DeSSA Accessibility Guidelines

Significantly revised manual
Change in descriptions of accessibility features:
Level 1: Universal Tools/Features
Level 2: Designated Supports
Level 3: Accommodations

Some Accessibility Features have moved to a

different category to align the Assessments
Manual address four groups of students:

Students with IEPs and/or 504 plans (SwD)

English Language Learners (ELL)
Dually identified students (SwD/ELL)
General Education with supports
64

DeSSA Accessibility Guidelines

Assessments covered by the Guidelines:
Smarter (ELA/Literacy and Mathematics)
DCAS (Science and Social Studies)
High School End-of-Course (Algebra II/
Integrated Math III/ U.S. History)
DCAS-Alt1 (Reading/Math/Science/Social
Studies)
Accessibility Features are not the same for
every test and some students may not use the
same accessibility features across each test
65

Accessibility Features
Level
1

Level
2

Level
3

66

TIDE Layout
Embedded Features
- Left hand side of screen
- All content areas for General
Assessment
- At bottom DCAS-Alt1 and
decision to NOT test in ELA
(for ELL students)
- There are no
accommodation codes (as
from previous years)
- Some slight rewording of
accommodations

67

68

TIDE Layout
Embedded Features

Non-Embedded Features

- Left hand side of screen

- All content areas for General
Assessment
- At bottom DCAS-Alt1 and
decision to NOT test in ELA
(for ELL students)
- There are no
accommodation codes (as
from previous years)
- Some slight rewording of
accommodations

- Right hand side of screen

- Labeled as Accommodation
Codes
- All content areas for General
Assessment included in 4 different
scroll menus
- DCAS-Alt1 embedded in the scroll
down

69

70

TIDE Layout
Embedded Features

Non-Embedded Features

- Left hand side of screen

- All content areas for General
Assessment
- At bottom DCAS-Alt1 and
decision to NOT test in ELA
(for ELL students)
- There are no
accommodation codes (as
from previous years)
- Some slight rewording of
accommodations

- Right hand side of screen

- Labeled as Accommodation
Codes
- All content areas for General
Assessment included in 4 different
scroll menus
- DCAS-Alt1 embedded in the scroll
down

Sections to Ignore
- Non-Embedded Accommodations
- Non-Embedded Designated
Supports
- These sections are duplicative to
Non-Embedded Features above
71

72

Do I Need an IEP/504 or ELL Revision Meeting?

If the change is substantive, YES

If the change is not substantive, answer
may still be YES
Will re-issue Exceptional Children group
memo addressing the decision on whether a
revision meeting is needed

If accommodations are identical to what

student had previously and only the codes
are different, then no revision meeting is
needed
73

Training
Updates

Trainings Page on Portal Updated

http://de.portal.airast.org/resources/?section=6

75

Training Completed as of 11/10/15

Number

Title

# Users
Completed

24246

DeSSA Entry Training

24250

DeSSA Accommodations Manager Training

235

24248

DeSSA District & School Test Coordinator Training

101

24288

Smarter Interim Test Administrator Training

626

24251

DCAS-EOC Test Administration Training

259

1,235

76

Other Trainings Available

Number

Title

Audience

24473

What is a CAT

Educators, Students

24472

Lets Talk Universal tools

Educators, Students

24484

Student Test Interface

Educators, Students

77

Trainings In Production
Number

Title

TBA

Paper/Pencil Test Administrator Training

EOC, DCAS, and Smarter

TBA

How to Use the TIDE Accommodations Upload

Feature

TBA

SysAID Overview

24482

ELL Accessibility Training

24483

SWD Accessibility Training

TBA

Smarter Summative Test Administrator Training

Audience

78

Peer Review
November 2015

Purpose of Peer Review

The purpose of the U.S. Department of Educations peer
review of State assessment systems is to support the
state in meeting statutory and regulatory requirements
under Title I of the Elementary and Secondary Education
Act of 1963 (ESEA) for implementing high technical
quality assessment systems and, where applicable,
provide for ESEA flexibility with an opportunity to
demonstrate that they have met requirements for highquality assessments under Principle 1 of ESEA flexibility.
Under the ESEA, the Department also has an obligation
to conduct peer review of state assessment systems.

80

Background of Peer Review

A key purpose of Title I of the ESEA is to promote
educational excellence and equity so when students
graduate from high school, they master the
knowledge and skills to be successful in college and
the workforce.
The ESEA requires a state to develop and implement
challenging academic content standards and
achievement standards and to apply the same academic
standards to all public schools and public school
students in the state.

81

Background of Peer Review

The first Peer Review started in 2005 with the Standards
and Assessments Peer Review Guidance: Information
and Examples for Meeting Requirements of the No Child
Left Behind Act of 2011 provided by the U.S. DOE.
The current Peer Review is scheduled early next year
with the Peer Review of State Assessment Systems
Non-Regulatory Guidance for States for Meeting
Requirements of the Elementary and Secondary
Education Act of 1965, as amended provided by the
U.S.DOE.

82

An Overview of Process
Six Critical Elements:
Statewide System of Standards and Assessments
Assessment Systems Operations
Technical Quality Validity
Technical Quality Others (e.g., Reliability, Fairness
and Accessibility, Scoring)
Inclusion of All Students
Academic Achievement Standards and Reporting

83

An Overview of Process
Process of Peer Review:
Each state must submit evidence for its assessment
system that addresses each element
Peer Reviewers will evaluate the evidence submitted
by the state
Each state will receive feedback in a form of peer
review notes
A formal decision is made by the Assistant Secretary
of Elementary and Secondary Education
The assessment peer review will not evaluate or
provide recommendations regarding whether or not a
states assessment system complies with Federal civil
rights laws that prohibit discrimination.
84

Wrap-up
Additional items
Questions and Responses
Exit Ticket

85