http://www.psc-cfp.gc.

ca/ppc-cpp/acs-cmptnc-evl-cmptnc/pp-
instrmnt-pc-eng.htm

Paper-and-pencil instruments
An efficient method of assessment
Paper-and-pencil instruments refer to a general group of assessment tools in which candidates
read questions and respond in writing. This includes tests, such as knowledge and ability tests,
and inventories, such as personality and interest inventories.

Paper-and-pencil tests can be used to assess job-related knowledge and ability or skill
qualifications. The possible range of qualifications which can be assessed using paper-and-
pencil tests is quite broad. For example, such tests can assess anything from knowledge of
office procedures to knowledge of federal legislation, and from the ability to follow directions
to the ability to solve numerical problems.

Because many candidates can be assessed at the same time with a paper-and-pencil test, such
tests are an efficient method of assessment.

Developing paper-and-pencil tests: 4 basic steps.

All assessment methods must provide information that is relevant to the qualification(s) being
assessed. The following 4 steps ensure that paper-and-pencil tests provide this information.

Step #1

Listing topic areas/tasks

Step #2

Specifying the response format, number of questions, the time limit and difficulty level

Step #3

Writing the questions and developing the scoring guide

Step #4

Reviewing the questions and scoring guide

Step #1

Listing topic areas/tasks

For each knowledge/ability qualification that will be assessed by the test, list the topic
areas/tasks to be covered. Check off any critical topic areas/tasks that are particularly
important to the job.

For example, the topic areas that will be covered for the qualification «knowledge of office
procedures» might be knowledge of correspondence, knowledge of filing and knowledge of
making travel arrangements. Or, for example, the tasks to be assessed for the qualification
«ability to solve numerical problems» might be the ability to add, subtract, multiply and
divide.

Step #2

Specifying the response format, number of questions, the time limit and difficulty level

Prior to writing the questions for your test, you should decide on such things as the response
format, the number of questions, the time limit and the difficulty level.

What type of response format should I choose?

The three most common response formats are:

 (a) multiple-choice (see Example 1);

 (b) short answer (see Example 2); and
 (c) essay.

(a) Multiple-choice

With a multiple-choice response format, a large number of different topic areas/tasks can be
covered within the same test and the questions are easy to score. However, because all
potential answers must be chosen by some candidates, it is time-consuming to write good
questions.

With a short-answer response format, as in multiple choice, a large number of different topic
areas/tasks can be covered within the same test and these questions are easy to score. In
addition, less time is required to write these questions compared to multiple-choice ones.

(b) Short-answer

With a short-answer response format, a large number of different topic areas/tasks can be
covered within the same test and these questions are easy to score. In addition, less time ie
required to write these questions compared to multiple-choice ones.

(c) Essay

With an essay response format, only a few topic areas/tasks can be covered due to the amount
of time it takes to answer questions; however, the content can be covered in greater detail.
Essay questions require little time to write but they are very time-consuming to score.
Although at first glance a multiple-choice format may seem a relatively easy and logical
choice if breadth of coverage is emphasized, don't be fooled. It is hard to write good multiple-
choice questions and you should only choose this type of response format if you are willing to
devote a lot of time to editing, reviewing, and revising the questions. If depth of coverage is
emphasized, use an essay response format.

How many questions do I need?

The number of questions you need depends on the breadth and depth of coverage required and
the importance of each topic area/task. Generally, the more important a topic area/task, the
more questions you should have. You should initially write several questions from each topic
area or task so that you can choose the best ones for the final version of the test.

With a multiple-choice response format, 30 questions are often sufficient. If you are using a
short-answer or essay format, fewer questions will be required.

How much time should I allow?

Unless speed of work is being assessed, the time limit should be set to allow the majority of
candidates to finish within the allotted time. A general guide for multiple-choice questions is
to allow about a minute per question. The time needed for short-answer and essay questions
depends on the complexity of the questions. If you are in doubt, try your test and time limit
out in advance.

How difficult should the questions be?

You should consider both the number of possible candidates and the level of job performance
required for the position. The questions should have a range of difficulty levels. You should
try to make all questions differentiate between poorly-qualified and well-qualified candidates.

Step #3

Writing the questions and developing the scoring guide.

All questions should tap meaningful information. Also, the level of language used for the
questions should be appropriate for the requirements of the position. The questions do not
have to always be expressed verbally. Diagrams, graphs, or tables may be incorporated into a
question where useful.

When you are writing each question, you should prepare the answers, designate the marks to
be allotted to each item, and decide on the rules for scoring. This ensures that there is a clear-
cut answer for each question. It also allows you to indicate the value of each question so that
candidates can decide for themselves the amount of time they should spend on each question.
The marks assigned to each question should reflect the relative importance of the question.

(a) Multiple-choice

The scoring guide for multiple-choice questions must include a scoring key indicating the
correct answer and it may also include a rationale for or explanation of the correct answer. If
marks are to be deducted for guessing, this must be determined and stated in the instructions
to candidates.

(b) Short-answer

The scoring guide for short-answer questions should include predetermined scoring
procedures and mark allocations. Each required point in the answer should be listed with its
relative mark allocation.

(c) Essay

The scoring guide for essay questions should include predetermined scoring procedures and
mark allocations. The major points of the answer should be listed with their relative mark
allocation. If marks are to be deducted for incorrect grammar, spelling and punctuation, this
must be stated in the instructions to candidates.

The scoring guides for short-answer and essay questions should be clear enough so that
scorers can judge whether or not marks should be given to a variation of the answer.

Step #4

Reviewing questions and scoring guide.

Have the questions and scoring guide reviewed by individuals familiar with the position in
question. Typically, this would involve other managers and/or job incumbents. Have them
ensure that each question is answerable, relevant to the job, technically and grammatically
correct and clear and understandable. Any symbols, words, phrases or content found to be
offensive, ambiguous, inappropriate, or misleading should be revised or replaced. The
reviewers should also assess the appropriateness of the time limit and difficulty level of the
questions. It may be helpful to have the reviewers actually take the test under testing
conditions. You should only include in the final version of the paper-and-pencil test those
questions that your reviewers endorsed.

The development of your own paper-and-pencil tests can be difficult and time consuming.
However, if these 4 steps are followed, the end result should be a quality assessment
instrument.

Example 1:

Below is an example of a multiple-choice question and scoring guide (with accompanying

rationale) designed to assess the qualification, ability to solve numerical problems.

Question:

1. You are responsible for purchasing equipment and furniture. How many filing cabinets can
you purchase with a budget of $6,000, if 50 filing cabinets cost $10,000? Choose the correct
answer from among the four choices provided. No marks will be deducted for guessing.
(Please note that the use of calculators is not permitted.)

1. 15
 2. 20
3. 25
4. 30

Scoring guide rationale:

d.30

$10,000/50=$6,000/X
X=$300,000/10,000
X=30

Example 2:

Below is an example of a short answer question and scoring guide designed to assess the
qualification which is called "knowledge concerning immigration".

Question:

1. What are 5 major factors that are considered in a request to immigrate to Canada? (5 marks)

 circumstances in home country (i.e., is persecution likely?)

 presence of support in Canada (i.e., relatives)
 skills and experiences that are required in or advantageous to Canada
 health
 financial status

Scoring guide: 1 point each

 Test Construction
an essay by
Delano P Wegener, Ph.D.

http://www.delweg.com/dpwessay/tests.htm

Rational for Tests:

Measures of student performance (testing) may have as many as five purposes:
 Student Placement,
 Diagnosis of Difficulties,

 Checking Student Progress,

 Reports to Student and Superiors,

 Evaluation of Instruction.

Unfortunately the most common perception is that tests are designed to statistically rank all students
according to a sampling of their knowledge of a subject and to report that ranking to superiors or
anyone else interested in using that information to adversely influence the student's feeling of self-
worth. It is even more unfortunate that the perception matches reality in the majority of testing
situations. Consequently tests are highly stressful anxiety producing events for most persons.

All too often tests are constructed to determine how much a student knows rather than determining
what he/she must learn. Frequently tests are designed to "trap" the student and in still other situations
tests are designed to insure a "bell curve" distribution of results. Most of the other numerous testing
designs and strategies fail to help the student in his learning process and in many cases are quite
detrimental to that process.

In a Mastery Based system of instruction the two main reasons for testing are to determine mastery
and to diagnose difficulties. When tests are constructed for these purposes, the other four purposes
will also be satisfied. For example, consider a test which requires the student to demonstrate mastery
and at the same time rigorously diagnoses learning difficulties. If no difficulties are indicated, it may be
safely assumed that the learner has mastered the concept. That information may then be used to
record student progress and to make reports to the student and superiors. Examining student
performance collectively for a group of students provides information about the quality of instruction.
Examining a single student's performance collectively for a group of learning objectives may be used
to determine proper placement within that group of learning objectives.

It is therefore important that the instructional developer construct each question so that a correct
response indicates mastery of the learning objective and any incorrect response provides information
about the nature of the student's lack of mastery. Furthermore, each student should have ample
opportunity to "inform" the instructor of any form of lack of mastery. Unfortunately the mere presence
of a test question influences the student's response to the question. The developer should minimize
that influence by constructing questions which permit the student to make any error he would make in
the absence of such influence. For example, a multiple choice question should have all the wrong
answers the student might want to select and should also have as many correct answers as the
student might want to provide.

True/False Questions:
True/false questions should be written without ambiguity. That is, the statement of the question should
be clear and the decision whether the statement is true or false should not depend on an obscure
interpretation of the statement. A true/false question may easily be used, and most commonly is used,
to determine if the student recalls facts. However, a true/false question may also be used to determine
if the learner has mastered the learning objective well enough to correctly analyze a statement.
It is important to be aware that only two choices are available to the student and therefore the nature
of the question gives the student a 50% chance of being correct. A single True/False question
therefore is helpful only if the student answers the question incorrectly and the incorrect response
indicates a specific misunderstanding of the learning objective. A collection of true/false questions,
about a single learning objective, all answered correctly by a student is a much stronger indication of
mastery. It is therefore important that the instructional developer construct a "test bank" containing a
large number of true/false questions. It is also important to include numerous true/false questions on
any test which utilizes true/false questions.

Ideally a true/false question should be constructed so that an incorrect response indicates something
about the student's misunderstanding of the learning objective. This may be a difficult task, especially
when constructing a true statement. The instructional developer should try to accomplish the ideal, but
should recognize that in some instances he/she will not reach that goal.

Multiple Choice Questions:

Multiple choice questions should be written without ambiguity. That is, the statement of the question
stem should be clear and should leave no doubt about how to select choices. Additionally the choices
should be written without ambiguity and should contain all information required to make a decision
whether or not to choose it. The decision whether to select or not select a choice should not depend
on an obscure interpretation of either the stem or the choice. A multiple choice question may easily be
used to determine if the student recalls facts. However, a multiple choice question may also be used to
determine if the student has mastered the learning objective well enough to correctly analyze a
statement.

The instructional developer should not construct multiple choice questions with a uniform number of
choices, a uniform number of valid choices, or any other recognizable pattern for construction of
choices. Instead the instructional developer should include as many valid and invalid choices as is
required to determine the student's deficiencies with respect to the learning objective. Moreover, each
choice should appear to be a valid choice to some student.

Multiple choice questions should therefore contain any number of choices with one or more valid
choices. The student is of course required to select all valid choices and failure to select any one of
the valid choices will provide information about the student's misunderstanding of the learning
objective in the same way that selection of an invalid choice reveals the nature of his/her
misunderstanding.

The nature of the choices provided in a multiple choice question may be of two types: those which
require merely recall of facts and those which require additionally activity such as synthesis, analysis,
computation, comparison, or diagramming. The instructional developer who is seriously concerned
with the student's success will use both types extensively.

Fill-in-the-Blank Questions:
The temptation, when constructing fillintheblank questions, is to construct traps for the student. The
instructional developer should avoid this problem. Ensure that there is only one acceptable word for
the student to provide and that the word (or words) is significant. Avoid asking the student to supply
"minor" words. Avoid fillintheblank question with so many blanks that the student is unable to
determine what is to be completed.

Sometime/Always/Never Questions:
The collection of Sometime/Always/Never (referred to as SAN) statements are statements which are:
true sometimes, always true, and never true. The statements used in these questions must be stated
carefully and should contain enough information to permit the student to decide whether the statement
is true sometimes, always, or never.

SAN questions (especially the sometimes statements) are the most difficult to construct but can be the
most significant part of a test. SAN questions should be constructed to force the student to engage in
some critical thinking about the learning objective. When used properly, SAN questions force the
student to consider important details about the learning objective. Careful use of this type of question
and careful analysis of student's response will provide detailed information about some of the student's
deficiencies.

SAN questions are especially appropriate, and easy to construct, for learning objectives addressing
concepts which are "black" or "white" except in a few cases. The true statements in a collection of
true/false questions are of course always true statements while the set of false statement may be
further subdivided into those which are true sometimes and those which are never true.