Designing Writing Assessments and Rubrics

LARC/CALPER Testing & Assessment Webinar

Thursday, November 14, 2013

An Assignment:
Before the webinar on Thursday, please read the essay below and then rate it using the
holistic rubric on page 2. During the webinar, well poll the audience to see what scores
everyone gave the essay. Although I generally teach the rubric and norm raters before I ask raters
to use the rubric, this exercise will provide some experience with using rubrics for those who
have not used them in the past.
The essay is in response to a no-longer-used prompt on a timed-writing placement test. Students
taking the test would be placed into one of three second language writing options. The
assignment read as follows:
Directions: You have forty-five (45) minutes to write an essay. Imagine that you have a friend
from your home country who is preparing to visit the U.S. for the first time. In a well-developed
essay, describe the advice that you would offer him/her. What would you tell your friend to
expect? What would you share from your own experience that you think your friend would find
helpful? What do you think your friend will find most surprising?

The Essay:
When my friend preparing to visit the U.S for the first time I will tell he/she to expect that the
people in the U.S. are very busy, and the school in the U.S are long. Most of the school and work
time are spend on the week day and most of the free time are spend on weekend to cook or play
around the park. From my experience in the U.S. that younger children are very helpful stay in
the U.S. because he/she can learn English faster and new culture to know about. The teacher in
the U.S. are so friendly and have a lot of specials program for non-speak English, for that was
very helpful. Also he/she can experience that a lot of difference people from out of the country
and learn difference culture he/she didnt know about it. Well I think the most surprising the
he/she are very happy to know the special program in the U.S. and a lot of difference people and
culture he/she never meet and know. And finally I hope he/she like what ever he/she think most
helpful for he/she in the future.

Deborah Crusan ~ Wright State University ~ Dayton, OH

deborah.crusan@wright.edu

Designing Writing Assessments and Rubrics

LARC/CALPER Testing & Assessment Webinar
Thursday, November 14, 2013

The Rubric:
This is a holistic rubric. For in-class assessment, an analytic rubric is more useful, but since the
essay you have been given to assess is an essay written for placement, the best way to assess it is
with a holistic rubric.
6-- Demonstrates clear competence in writing on both the rhetorical and syntactic levels, though it
may have occasional errors.
A paper in this category:
* effectively addresses the writing task
* is well organized and well developed
* uses clearly appropriate details to support a thesis or illustrate ideas
* displays consistent facility in the use of language
* demonstrates syntactic variety and appropriate word choice
5-- Demonstrates competence in writing on both the rhetorical and syntactic levels, though it will
probably have occasional errors.
A paper in this category:
* may address some parts of the task more effectively than others
* is generally well organized and developed
* uses details to support a thesis or illustrate an idea
* displays facility in the use of language
* demonstrates some syntactic variety and range of vocabulary
4-- Demonstrates minimal competence in writing on both the rhetorical and syntactic levels.
A paper in this category may reveal one or more of the following weaknesses:
* addresses the writing topic adequately but may slight parts of the task
* is adequately organized and developed
* uses some details to support a thesis or illustrate an idea
* demonstrates adequate but possibly inconsistent facility with syntax and usage
* may contain some errors that occasionally obscure meaning
3-- Demonstrates some developing competence in writing, but it remains flawed on either the
rhetorical or syntactic level, or both.
A paper in this category may reveal one or more of the following weaknesses:
* inadequate organization or development
* inappropriate or insufficient details to support or illustrate generalizations
* a noticeably inappropriate choice of words or word forms
* an accumulation of errors in sentence structure and/or usage
2-- Suggests incompetence in writing.
A paper in this category is seriously flawed by one or more of the following weaknesses:
* serious disorganization or underdevelopment
* little or no detail, or irrelevant specifics
* serious and frequent errors in sentence structure or usage
* serious problems with focus
1-- Demonstrates incompetence in writing.
A paper in this category:
* may be incoherent
* may be undeveloped
Deborah Crusan ~ Wright State University ~ Dayton, OH
deborah.crusan@wright.edu

Designing Writing Assessments and Rubrics

LARC/CALPER Testing & Assessment Webinar
Thursday, November 14, 2013
* may contain severe and persistent writing errors
http://www.etstechnologies.com/html/TOEFLscoringguide.htm

The Advantages of Using Rubrics for the Assessment of Writing

Rubrics allow for more objective and consistent evaluation.

Rubrics necessitate the clarification of teacher criteria.

Rubrics clearly show students ways in which their work will be evaluated and what is
expected of them.

Rubrics raise student awareness regarding peer assessment criteria.

Rubrics provide feedback to teachers regarding instruction effectiveness.

Rubrics provide benchmarks upon which to measure and document progress.

Rubrics provide all students with an opportunity to succeed at some level.

Common Features of Rubrics

Rubrics focus on measuring stated objectives or outcomes.

Rubrics rate student performance using a range.

Rubrics include explicit performance characteristics arranged in levels indicating the

degree to which a standard has been met.

Rubrics should be easy to understand and use.

How to Create Rubrics

Develop the goals for your course and daily class meetings.

Select assessment tasks that provide data aligned with your goals.

Develop performance standards for each of the goals.

Deborah Crusan ~ Wright State University ~ Dayton, OH

deborah.crusan@wright.edu

Designing Writing Assessments and Rubrics

LARC/CALPER Testing & Assessment Webinar
Thursday, November 14, 2013

Differentiate performances (categories) based on well-described criteria.

Rate (assign weight or value to) the categories.

Holistic rubric:
The holistic rubric is generally a five or six point scale which involves ranking a sample of a
students writing by matching it with another piece in a benchmark series, scoring it for the
prominence of certain features (coherence, development) important to that kind of writing, and
assigning it a grade. Holistic rating is an impressionistic rating of essays for overall quality and is
quite often used for placing students in composition classes and for assessing writing in largescale assessments such as the TOEFL and the SAT. Interestingly, even numbered rubrics are
preferred because they force the evaluator to place the work at above or below the middle.
Crusan, 2010a
Advantages:
fast evaluation
relatively high inter-rater reliability
scores represent standards easily interpreted by lay persons
scores tend to emphasize writers strengths
applicable to writing across disciplines
Disadvantages:

One score masks differences across the subskills within each score
No diagnostic information available (no washback potential)
Scale may not apply to all genres of writing equally
Raters need to be extensively trained to use scale accurately
Brown, 2004, p. 242

The rubric you used to score the essay is a classic example of a holistic rubric.

Deborah Crusan ~ Wright State University ~ Dayton, OH

deborah.crusan@wright.edu

Designing Writing Assessments and Rubrics

LARC/CALPER Testing & Assessment Webinar
Thursday, November 14, 2013

Analytic Rubric:
An analytic rubric includes a more detailed analysis, usually based on a scale or checklist of
prominent features or characteristics of a piece of writing. The features selected for evaluation
vary according to the context of the specific writing assignment, the audience, and the purpose
for writing. One of the most prominent analytic rubrics is the ESL Composition Profile (Jacobs,
Hartfiel, Hughey, & Wormuth, 1981). Another equally excellent analytic rubric, a more up-todate adaptation of the ESL Composition Profile, is available in Ferris & Hedgcock (2013).
Crusan, 2010a
Advantages:
Offers more washback
More in-depth
Offers feedback on more aspects of student writing
Disadvantages
Numerical scores alone not sufficient for helping students become better
writers
More time consuming to create than holistic
More difficult to create an analytic rubric
Genre dictates variation in scoring

Deborah Crusan ~ Wright State University ~ Dayton, OH

deborah.crusan@wright.edu

Designing Writing Assessments and Rubrics

LARC/CALPER Testing & Assessment Webinar
Thursday, November 14, 2013

Deborah Crusan ~ Wright State University ~ Dayton, OH

deborah.crusan@wright.edu

Designing Writing Assessments and Rubrics

LARC/CALPER Testing & Assessment Webinar
Thursday, November 14, 2013

Primary Trait Rubric:

A primary trait rubric analyzes certain language use and scores for one trait (e.g. organization, the
correct use of the past tense). Useful for assessing L2 learners proficiency ad growth in writing; they
are potentially more informative than the other types of rubrics because they are more focused and
are particularly helpful in the classroom for diagnosing student writing ability.
Crusan, 2010a
Advantages:
Focus on one trait or skill at a time
May be more informative for writers, especially at the beginning level
Helpful in the classroom
Disadvantages:
Probably not fair to base grades on these rubrics
Cannot be applied to other tasks without revision
PRIMARY TRAIT RUBRIC EXAMPLE

0
1
2
3

PrimaryTrait:VocabularyUseinWriting
Showsnocontrolovervocabularytaught;notenoughwritingtoevaluate
Showslimitedcontrolofvocabularytaughtinclass;appropriateuseis
infrequent
Showscontrolofanadequaterangeofthevocabularytaughtinclassandmost
oftenusesthisvocabularyappropriately
Showscontrolofawiderangeofthevocabularytaughtinclassandalwaysuses
vocabularyappropriately

Adaptedfrom
http://www.carla.umn.edu/assessment/vac/Evaluation/rubrics/types/interpersonal.html

InAssessmentinthesecondlanguagewritingclassroom(Crusan,2010a),youwillfindexamplesof
assignmentsandtheiraccompanyingrubrics.

Deborah Crusan ~ Wright State University ~ Dayton, OH

deborah.crusan@wright.edu

Designing Writing Assessments and Rubrics

LARC/CALPER Testing & Assessment Webinar
Thursday, November 14, 2013

Some Helpful Websites

Rubric Builder http://landmark-project.com/classweb/tools/rubric_builder.php3
This builder allows you to enter your own criteria.
Rubric, Rubrics, Teacher Rubric Makers http://www.teach-nology.com/web_tools/rubrics/
Here you will find a paragraph writing rubric maker, a persuasive writing rubric maker, and a
writing rubric maker along with dozens of other rubric generators. However, you cannot plug in
your own criteria. But its a start.
Teaching Writing http://humanities.byu.edu/elc/teacher/teaching_writing/Teaching_Writing.html
ContainslinkstoaWritingScoringGuideandanexerciseforstudentsinwhichtheyreadbad
essaysandgradethemusingateacherdesignedrubric.
IMPORTANT: While these websites offer helpful tips and ease of creation, there are best used as a
starting point. The best rubrics are those that have been created for and tailored to one assignment.
After all, we are asking our students to do different things when they write a literacy narrative or a
summary or an evaluation.

Deborah Crusan ~ Wright State University ~ Dayton, OH

deborah.crusan@wright.edu

Designing Writing Assessments and Rubrics

LARC/CALPER Testing & Assessment Webinar
Thursday, November 14, 2013

SOME References
Broad, B. (2003). What we really value: Beyond rubrics in teaching and assessing writing.
Logan, UT: Utah State University Press.
Brown, H. D. (2004). Language assessment: Principles and classroom practices. White Plains,
NY: Pearson Education, Inc. (Chapter 9)
CCCC Committee on Assessment. (2006). Writing Assessment: A Position Statement. Retrieved
from http://www.ncte.org/cccc/announcements/123784.htm.
CCCC Committee on Second Language Writing. (2009). CCCC statement on second language
writers and writing. Retrieved from
http://www.ncte.org/cccc/resources/positions/secondlangwriting.
Crusan, D. (2010a). Assessment in the second language writing classroom. Ann Arbor, MI:
University of Michigan Press.
Crusan, D. (2010b). Assess thyself lest others assess thee. In P. K. Matsuda, & T. Silva (Eds.),
Practicing theory in second language writing (pp. 245-262). West Lafayette, IN: Parlor Press.
Crusan, D. (2011). The promise of directed self-placement for second language writers. TESOL
Quarterly, 45(4), 774780.
Crusan, D. (2013). Assessing writing. In A. J. Kunnan (Ed.), The companion to language
assessment. Volume 1: Abilities, Contexts, Learners (pp. 201-215). Hoboken, NJ:
Wiley/Blackwell.
Ferris, D. (2011). Treatment of error in second language student writing, 2nd ed. Ann Arbor, MI:
University of Michigan Press.
Ferris, D., and Hedgcock, J. (2013). Teaching L2 composition: Purposes, process, and practice,
3nd ed. New York: Routledge.
Hamp-Lyons, L. & Kroll, B. (1996/2001). Issues in ESL writing assessment: An overview. In
T. Silva & P. K. Matsuda (Eds.), Landmark essays on ESL writing (pp. 225-240).
Mahwah, NJ: Lawrence Erlbaum.
Huot, B. (2002). (Re)Articulating writing assessment for teaching and learning.
Logan, UT: Utah State University Press.
Jacobs, H. L., Zingraf, S. A., Wormuth, D. R., Hartfiel, V. F., & Hughey, J. B. (1981).
Testing ESL composition: A practical approach. Rowley, MA: Newbury House.
OMalley, J. M. & Pierce, L. V. (1996). Authentic assessment for English language learners:
Practical approaches for teachers. New York: Addison-Wesley.

Deborah Crusan ~ Wright State University ~ Dayton, OH

deborah.crusan@wright.edu

Designing Writing Assessments and Rubrics

LARC/CALPER Testing & Assessment Webinar
Thursday, November 14, 2013

Weigle, S. C. (2002). Assessing writing. Cambridge: Cambridge University Press.

Weigle, S. C. (2007). Teaching writing teachers about assessment. Journal of Second Language
Writing 16, 194-209.
White, E. M. (1999). Using Scoring Guides to Assess Writing. In Richard Straub (Ed.),
A Sourcebook for Responding to Student Writing (pp. 203-212). New Jersey:
Hampton Press.

Deborah Crusan ~ Wright State University ~ Dayton, OH

deborah.crusan@wright.edu

10