EDU 385

Session 10

Writing Supply Items

Short Answer and Essay
 Bell Work

Sister Thorne will review the prior

information from the last session
 Content Objectives
Understand the strengths and
limitations of short-answer test items
 Language Objectives
1. Describe the strengths and limitations of short-answer test
items

2. Distinguish well-stated and poorly stated short-answer test

items

3. Identify and correct faults in poorly stated short-answer test

items

4. Match short-answer test items to intended learner outcomes

5. Construct short-answer test items that are well stated and

extended-response

6. Describe the strengths and limitations of essay questions

7. Distinguish between restricted-response and extended-

response essay questions

8. Describe the strengths and limitations of essay questions

9. Write essay questions that present a clear task, are relevant

to important learning outcomes and provide guidelines for
scoring
 Academic Vocabulary

Short Answer Items

Restricted-Response Essay
Extended-Response Essay
Alternative-Response Items
Complex Learning Outcomes
Student Bluffing
 Selection vs. Supply Type
Items
Selection-type test items can be designed to
measure a variety of learning outcomes from
simple to complex

Tend to be favored in achievement testing

Greater control of type of response students
can make

Broader sampling of achievement

Quicker and more objective scoring
Despite these advantages Supply-type test items
can play an important role in measuring
achievement
 Supply-Type Items
Supply-type test items require students to produce
the answer

May be single word or several page response

Length of response ranges along a continuum, but
usually divided into:

Short answer
Restricted-response essay
Extended-response essay
 Supply-Type Item Continuum
Highly
High
Controlled
Freedom of
Response
Response Restricted-Response Extended-Response
Short-Answer Item
Item Item
Outcomes Measured Outcomes Measured Outcomes
Measured

Knowledge Knowledge
Understanding
Computational Understanding Application
Skill Application Analysis
Analysis Evaluation
Creating
Short-Answer Items
Strengths
1. It is easy to write test items
2. Guessing is less likely than in selection-type test items
3. This test item type is well-suited to computational problems and other
learning outcomes where supplying the answer is important
4. A broad range of knowledge outcomes can be measured

Limitations
1. It is difficult to phrase statements so that only one answer is correct
2. Scoring is contaminated by spelling ability except when responses are
verbal
3. Scoring is tedious and time-consuming
4. This item is not very adaptable to measuring complex learning
outcomes
 Short-Answer Items
Require the examinees to supply appropriate words,
numbers, or symbols to answer a question or complete
a statement

Example
What are the incorrect responses in a multiple-
choice item called? (distractors)

The incorrect responses in a multiple-choice item

are called (distractors)

Can include computational problems and any simple

item form that requires the answer rather than
selecting it

Primarily used to measure simple recall of knowledge

 Short-Answer Items
Short-answer appears easy to write but may be
difficult to construct so only one answer is correct

several other defendable answers could be given

spelling may be a problem
does word have to be spelled correctly?
if so may penalize poor speller who knows answer
if not may have difficulty determining if misspelled
word is the correct response (requires
subjectivity)

Due to these difficulties short-answer items should be

reserved to recall specific information where the
answer is obvious
 Rules for Writing Short-Answer Items
1. State the item so that only a single, brief answer is
possible
*Requires skill in phrasing and precise terms
*What appears as simple and clear questions can often
be answered in different ways
*Review the item with this in mind
 Rules for Writing Short-Answer Items
2. Start with a direct question and switch to an
incomplete statement only when greater
conciseness is possible by doing so
-The use of direct questions increases likelihood that
problem will be stated and that only one answer will
be appropriate
-Incomplete sentences tend to be less ambiguous
when based on problems first stated as sentences

Example

What is another name for true-false items? (alternative-response

items)

True-false items are also called (alternative-response items)

3. It is best to leave only one blank, and it should
relate to the main point of the statement
Leaving several blanks is often confusing and the
answer to one blank may depend on the answer in
another
Examples (Poor and Better)
In terms of response, the (matching item) is most like the (multiple-
choice items)
In terms of type of responses, which item is most like the matching
item? (multiple-choice)

It is important to avoid asking students to respond to

unimportant aspects of a statement
4. Place the blanks at the end of the
statement

-This allows student to read the complete

problem before coming to the blank to be filled
(avoids having to re-read the sentence

-Easier to construct incomplete statements

Examples (Poor and Better)

The supply-type item used to measure the ability to organize and
integrate material is called an (essay item)

Supply-type items used to measure the ability to organize and

integrate material are call (essay items)
5. Avoid extraneous clues to the answer
(i.e., extra unnecessary information)
-A Common clue is length of the blank (e.g., long blank
for long answer, short blank for short answer)
-All blanks should be uniform in length
-Another common clue is use of the indefinite article
a or an just before the blank
Examples (Poor and Better)
The supply-type item used to measure the ability to organize and
integrate material is called an (essay item)
Supply-type items used to measure the ability to organize and
integrate material are called (essay items)

-Poor version (the article an) eliminates the only

other supply-type item choice short-answer
-Solution - eliminate the article an or include both
articles a(an)
6. The numerical answers indicate the
degree of precision expected and the
units in which they are expressed
-Example nearest whole number
-Specify the degree of precision prevents the student
from spending too much time on the problem
-Provides more uniform responses
Checklist for Evaluating Short-Answer
Items
1. Is this type of item appropriate for measuring the
intended learning outcome?
2. Does the item task match the learning task to be
measured?
3. Does the item call for a single, brief answer?
4. Has the item been written as a direct question or a
well-stated incomplete sentence?
5. Does the desired response relate to the main point
of the item?
6. Is the blank placed at the end of the statement?
7. Have clues to the answer been avoided (e.g., a or
an, length of blank)
8. Are the units and degree of precision indicated for
 Essay Questions
Provide greater freedom of response for students as
Choice on how to approach problem
What factual information to use
The degree of emphasis to place on each aspect of
the response

Essay questions especially useful for measuring the

ability to organize, integrate, and express ideas

Areas where selection-type items fall are inadequa

Essays and selection-type items complement each
other
Summary Comparison of Selection-Type Items and Essay
Selection-Type Items Essay Questions
Learning Good for measuring the Ineffective for measuring
Outcomes recall of knowledge under- the recall of knowledge;
Measured standing and application best for the ability to
levels of learning, organize, integrate and
inadequate for organizing express ideas
and expressing ideas
Sampling of The use of a large number of The use of a small
content items results in broad number of items limits
coverage, which makes coverage, which makes
representation sampling of representative sampling
content feasible of content infeasible
Preparation of Preparation of good items is Preparation of good items
Items difficult and time consuming is difficult but easier than
selection-type items
Scoring Objective, simple and highly Subjective, difficult, and
reliable less reliable
Factors Reading ability and guessing Writing ability and
Distorting Scores bluffing
Probable Effect Encourages students to Encourages students to
on Learning remember, interpret, and organize, integrate, and
use the ideas of others express their own ideas
 Types of Essay Questions
Restricted Response Questions
-Places strict limits on the answers to be given as
defined
-Boundaries of subject matter narrowly defined by the
problem and specific form of the answer
-Indicated by words as list, define, and give
reasons
Example
Describe the relative merits of selection-type items and essay
questions for measuring learning outcomes at the understanding
level. Confine your answer to one page

Mr. Rogers a ninth-grade science teacher, wants to measure his

students ability to interpret scientific data with a paper-and
pencil test
1. Describe the steps that Mr. Rogers would follow,
2. Give reasons to justify each step
 Types of Essay Questions
Extended Response
-Provide students almost unlimited freedom to
determine the form and scope of their response
-Some practical limits may be imposed such as time
and page limits
-Students given sufficient freedom to demonstrate
skills of evaluating and creating

Example (Evaluation Outcome):

(The student is given a complete achievement test that includes
errors or flaws in the directions, in the test itms and in the
arrangement of the items)
Write a critical evaluation of this test using as evaluative criteria the
rules and standards for test construction described in your textbook.
Include a detailed analysis of the tests strengths and weaknesses
and an evaluation of its overall quality and probable effectiveness
Essay Questions
Strengths
1. The highest level learning outcomes (analyzing,
evaluating, creating) can be measured
2. Preparation time is less than for selection-type
items
3. The integration and application of ideas is
emphasized

Limitations
1. There is an inadequate sampling of achievement in
time needed for answering each question
2. It is difficult to related to intended learning
outcomes because of freedom to select, organize, and
express ideas
3. Scores are raised by writing skill and bluffing and
 Rules for Writing Essay Questions
1. Use essay questions to measure
complex learning outcomes only
-Preferred use of Essay tests: to give reasons, explain
relationships, describe details, formulate conclusions
-At Evaluating and Creating levels objective tests and
restricted response tests have limited value
-Evaluation of a novel or an experiment requires
extended-response questions
-The Essay Question contribute most uniquely at
this level
 Rules for Writing Essay Questions
2. Relate the question as directly as
possible to the learning outcome being
measured
-Each question should be specifically designed to
measure one or more well-defined outcomes
-Start with precise description of performance to be
measured
-To help student focus on the outcomes to be measured
tell the students the criteria on which their response
will be evaluated
Example:
Your answer will be evaluated in terms of its comprehensiveness, the
relevance of its arguments, the appropriateness of its examples, and
the skill with which it is organized
 Rules for Writing Essay Questions
3. Formulate questions that present a
clear task to be performed
-Avoid starting a sentence with words such as: who,
what, when, where, name, and list - they limit the
response to recall of knowledge outcomes
-Instead use words as why, describe, explain, compare,
relate, contrast, interpret, analyze, criticize, and
evaluate
-A good check for phrasing a restricted response
question is to write a model answer or a mental answer
to the question
-Helps identify ambiguity and approximate time
need
-For extended-response ask one or more colleagues to
read the question to see if it is understood and clear
 Rules for Writing Essay Questions
4. Do not permit a choice of questions
unless the learning outcomes require it
-Makes student comparisons more difficult
-With optional choices students will select the question
to which they are best prepared to answer (limits
ability to sample their overall knowledge of the
subject)
-May be appropriate if only measuring writing skills or
creativity
 Rules for Writing Essay Questions
5. Provide ample time for answering and
suggest a time limit on each question
-Time must be allowed students for thinking as well as
writing, thus generous time limits should be provided
-In measuring complex achievement it is better to use
a few questions and to improve sampling test more
frequently
-Inform students of approximate time they should
spend on each question also may want to set page
limits for the answers
Types of Complex Outcomes and Related Terms for
Writing Essay Questions
Outcome Sample Terms
Comparing compare, classify, describe, distinguish between, explain,
outline, summarize
Interpreting convert, draw, estimate, illustrate, interpret, restate,
summarize, translate
Inferring derive, draw, estimate, extend, extrapolate, predict,
propose, relate
Applying arrange, compute, describe, demonstrate, illustrate,
rearrange, relate, summarize
Analyzing break down, describe, diagram, differentiate, divide, list
outline, separate
Creating compose, design, devise, draw, formulate, make up,
present, propose
Synthesizing arrange, combine, construct, design, rearrange, regroup,
relate, write
Generalizing Construct, develop, explain, formulate, generate, make,
propose, state
Evaluating appraise, criticize, defend, describe, evaluate, explain,
judge, write
 Rules for Scoring Essay Answers
1. Evaluate answers to essay questions in terms of the
learning outcomes being measured
2. Score restricted-response answers by the point
method, using a model answer as a guide
3. Score extended-response answers by the rating
method, using defined criteria as a guide
4. Evaluate all the students answers to one question
before proceeding to the next question
5. Evaluate answers to essay questions without
knowing the identity of the writer
6. Whenever possible, have two or more persons grade
each answer
Student Bluffing and Scoring Essays
Students can obtain higher scores on essay questions by clever bluffing.
Although this requires skill in writing and some knowledge of the topic,
credit should not be given unless the question is specifically answered.
Some common types of bluffing are listed below
1. Student repeats the question in statement form (slightly paraphrased)
and tells how important the topic is (e.g., the role of assessment in
teaching is extremely important. It is hard to imagine effective
instruction without it, etc.)
2. Student writes on a well-known topic and fits it to the question (e.g., a
student who knows testing, well but knows little about performance
assessment and is asked to compare testing and performance assessment
might describe testing in considerable detain and frequently state that
performance assessment is much superior to evaluating the type of
leaning assessment by the tests)

3. Student liberally sprinkles the answer with basic concepts regardless

of whether they are understood (e.g., when asked to write about any
assessment technique, the importance of validity and reliability is
mentioned frequently)

4. Student include the teachers basic beliefs whenever possible (e.g.,

Checklist for Evaluating Essay Questions
1. Is this type of item appropriate for measuring the
intended leaning outcome?
2. Does the item task match the learning task to be
measured?
3. Is the question designed to measure complex
learning outcomes?
4. Does the question make clear what is being
measured and how the answer will be evaluated?
5. Has terminology been used that clarifies and limits
the task (e.g., describe not discuss)?
6. Are all students required to answer the same
questions?
7. Has an ample time limit been indicted for each
question?
Summary of Points
1. Use supply-type items whenever producing the
answer to an essential element in the learning
outcome (e.g., defines terms, instead of identifies
meaning of terms
2. Supply-type items include short-answer items,
restricted-response essay, and extended-response
essay
3. The short-answer item can be answered by a word,
number, symbol, or brief phrase
4. The short-answer item is limited primarily to
measuring simple recall or knowledge outcomes
5. Each short-answer item should be so carefully
written that there is only one possible answer, the
entire item can be read before coming to the answer
space, and there is no extraneous clues to the answer
In scoring short-answer items, give cred for all correct
Summary of Points
7. Essay questions are most useful to measuring the
ability to organize, integrate, and express ideas
8. Essay question are inefficient for measuring recall of
knowledge because they provided limited sampling,
are influenced by extraneous factors (e.g., writing
skills, bluffing, grammar, spelling, handwriting), and
scoring is subjective and unreliable
9. Restricted-response essay questions can be more
easily written and scored, but due to limitations on the
responses they are less useful for measuring the
higher-level outcomes (e.g., integration of diverse
materials)
10. Extended-response essay questions provide the
freedom to select, organize, and express ideas in the
manner that seems most appropriated; therefore, they
are especially useful for measuring such outcomes
Summary of Points
11. Essay questions should be written to measure
complex learning outcomes, to present a clear task,
and to contain only those restrictions need to call forth
the intended response an provide for adequate scoring
12. Essay questions should be scored by focusing on
the intended response, bu using a model answer or set
of criteria as a guide, by scoring question by question,
and by ignoring the writers identity. Be wry of student
bluffing. If an important decision is to be based on the
results, tow or more competent scorers should be used
 Now Go Forth
and
Do Good Things