Running head: FORMATIVE EVALUATION

Capstone Project- Week Five

By Lynda S Strickland
EIDT 4001-1, Walden University
Professor Susan Schnorr
August 16, 2015

FORMATIVE EVALUATION

Instructional Material Evaluation

Pilot testing was incorporated in order to synthesize the real training and delivery aspect for
BeaconLBS. While training has occurred in person and via WebEx, mass implementation is
planned for the PDS-Q add-in product. The learning consists of a blended learning system
including self-instruction and technology-based training. Material of instruction include: User
Guide, video demo, and hands on application in a live system. The learning element was
delivered via email which included an introduction, log-in and password, and the BeaconLBS
PDS-Q User Guide.
Pilot testing was conducted with testers matching the learner qualifications as defined in the
needs analysis. Testers ranged in age from 30 to 63 years old, male and female, with education
varying from high school to master degrees. Two Testers were familiar with the program and
two testers had no experience. The instructional material was delivered via a welcoming email
with proceeding instructions which included their user id and a temporary password to the
BeaconLBS portal. The only deviance from a real training application is that the staging site was
used rather than the production site.
The use of the staging site did cause a few delays as IT decided to run testing without
notification and entrance to the portal was spotty for a few days, delaying the hands on training.
Training proceeded well otherwise. Testers where asked to evaluate in three areas, 1) PDS-Q
User Guide, 2) video demonstration, and 3) learner experience.

Results (Appendix A and B)

Testers were asked to rate answers on a Likert Scale of 1 to 5 ranging from very ineffective to
very effective. The Likert Scale measures peoples attitude in terms of the extent they agree
tapping into the cognitive and affective components (McLeod, 2008). I also included 9 open
ended questions for testers to further define positive, negatives or changes they would like to see.

FORMATIVE EVALUATION

PDS-Q User Guide

The questions for the user guide had a possible 40 point scale equation. Between 4 testers the
points averaged at 34.25 with an 86% likability rate (34.25/40). Comments were very similar that
the User guide was visually simple and clean, and easy to follow. Only one tester had a
negative comment that the media needed to be reformatted for equal vertical/horizontal ratio.
This same tester who has experience with the system also stated, there was too much
information and it should be specific to just opening and submitting the lab order. This leads
me, as a trainer in the program, to assume he is chunking material when demonstrating rather
than the whole package.
Video Demonstration

The video demonstration had possible 35 points equation. The average points came in at
29.75 with an 85% likability factor. The testers felt it was a positive overview to see the system
in action but felt the screen needed to be larger and break the sections down.
Learner Experience

The learner experience only had 3 of the 4 testers complete this section. Using the scoring
from the three testers who completed with a possible point range of 45, the average came in at
39.3 with an 87.4% likability rate. Overall, the learners stated the system was easy to use, learn,
and accomplish the goals set forth.

Revisions
According to Piskurich (2006) the decision to revise pieces or to revise the whole training
program is particularly important with self-instructional materials.
It is my opinion the User Guide can be used as-is with a possible addition of a short two page
job aid breaking down the key points of click and submit. It is my opinion however; BeaconLBS
should have a Train-the-Trainer workshop so all trainers are consistent in their training,

FORMATIVE EVALUATION

understanding, and messaging of the PDS-Q add-in product. The only change needed would be
to insert a new link for a revised video demonstration.
Changes to the video are definite; due to time constraints the original video had not yet been
edited. The current video is 7 minutes long and could be shortened if it is created with sections
rather than 1 long piece as one tester suggested. This would allow viewers to watch specific
sections of their choosing. Piskurich (2006) describes this as chunking, which is breaking down
mass content into pieces for easier digestion by trainees.
Another revision for the video would be to bring the screen forward for a larger view.
Adding the website onto the screen to emphasize the web address would be another addition I
would make during the log in phase. These are all minor changes and would not be cost
inhibitive.

Summary
The Pilot test would be considered successful with an overall 86% likability score. The results
showed a slight elevation in scores from testers with no knowledge versus the testers who
worked for BeaconLBS. A slight bias could be a contributing factor.
Slight modifications are required for the video more so in smoothing the information and
ensuring a larger field of view. The User Guide was effective and the learners were able to
successfully meet the objectives of logging in, accessing the lab order, submitting a lab order,
printing a lab order, and emptying their queue.

FORMATIVE EVALUATION

Appendix A
PDS-Q Training Material Evaluation
points
available
Part 1 User Guide
Tester 1
tester 2
Tester 3
Tester 4

Written responses:

average
points
40 34.25
31
33
36
37

*visually simple and clean

*Reformat for equal vertical/horizontal
ratio
points
available

Part II Video

average
points
35 29.75

Tester 1
tester 2
Tester 3
Tester 4

29
29
31
30

Written responses:

%likability
85.00%

*larger video screen

*break up the sections
*great to see in action
points
available

average
points

Part III Learner

Experience

45 39.33

Tester 1
tester 2
Tester 3
Tester 4

0
31
42
45

Written responses:

%likability
86%

%likability
87.40%

*was easy to accomplish each goal

FORMATIVE EVALUATION

Appendix B

likability
88.00%
87.00%

87.40%
86.00%

86.00%

85.00%

85.00%

likability

84.00%
83.00%
Part 1 User Guide

Part II Video

Part III Learner

Experience

Part III Learner Experience

average points

Part II Video

points available
Part 1 User Guide
0

10

20

30

40

50

FORMATIVE EVALUATION

Reference
McLeod, S. A. (2008). Likert Scale. Retrieved from www.simplypsychology.org/likertscale.html
Piskurich, G. M. (2006). Rapid instructional design: Learning ID fast and right (2nd ed.). San
Francisco, CA: Pfeiffer.

FORMATIVE EVALUATION

RWR CoEL Approved Rubric

Formative
Evaluation
(AECT 5.3)
Graded in Week 5

Evidence shows that:

Evidence shows that:

Evidence shows that:

Formative evaluation
report is incomplete or
not included.

Report includes a
general analysis of beta
test and provides
satisfactory rationale for
recommended
revisions.

Report includes a
complete analysis of
beta test and
provides rationale for
recommended revisions
supported by Learning
Resources.