Running head: FORMATIVE EVALUATION 1

Brandy Milson

Formative Evaluation

University of Houston-Clear Lake

Dr. Rice

November 2, 2010
Running head: FORMATIVE EVALUATION 2

There are many educational tools and programs available to educators regardless of the level

of instruction needed and the readiness level of students. When asked to choose one to evaluate,

the choices can be overwhelming. "A formative evaluation is the process of evaluating the

instructional product during its design and development. The purpose of formative evaluation is

to review the instruction for weaknesses and make the necessary revisions to correct errors and

enhance the effectiveness before implementation" (Gayle & Rasmussen, 2006, p. 133).

Formative evaluations are often done by expert reviewers and a sample of the target audience

(tryout) such as a selection of students in your classroom. For this formative evaluation, I have

chosen to analyze and critique is Shopping Spree offered by e-Learning for Kids and it can be

found at http://e-learningforkids.org/Courses/EN/ Shopping_Spree/launch.html. Shopping free is

designed to provide age 12 math students a lesson on calculating the percentage of a number.

According to Gayle & Rasmussen (2006), "Evaluation is the means for determining the value,

or worth of an instructional product or program" (p. 128). Therefore, the goal of this formative

evaluation is to determine the worth of Shopping Spree and determine if it is a quality

educational resource. "WBI's value is often communicated to its stakeholders in terms of

instructional effectiveness, efficiency, and appeal" (Gayle & Rasmussen, 2006, p. 128). Gayle

and Rasmussen argue that effectiveness, efficiency and appeal are highly influential on learner

mastery of goals and objectives.

Three Evaluation Criteria for Formative Evaluation

Effectiveness

Gayle and Rasmussen (2006) contend that," effectiveness of WBI is measured by determining

whether learners achieved the instructional goals (p. 128)". The effectiveness criterion evaluates
Running head: FORMATIVE EVALUATION 3

the mastery of goals and success of instruction. The instructional goal of Shopping Spree is for

students to learn how to calculate the percent of a number. Content is introduced through an

interactive visual and audio lesson. After each concept is introduced you have to calculate a

percentage which earns you money to spend on a Shopping Spree. There are five venues

(restaurant, clothing store, bookstore, sports store, and shoe store) for practicing calculating

percentages (as sales prices). I played the game several times and there are different shopping

scenarios but there is a lack of increase in difficulty.

Efficiency

Where there is a forced choice or tradeoff required for time, monies and other resources

efficiency is the quality that you must examine closely. When examining the efficiency of a

program, stakeholders primarily look at the financial benefits as opposed to the costs. Does the

educational tool or program provide a savings of either time, money, or other resources. Is the

instruction delivered in a manner that saves money and time for stakeholders? Shopping Spree is

a free educational program offered by e-Learning for Kids which is a private non-profit

organization for kids . For many primary stakeholders, there is a savings of time and money.

Appeal

"Appeal involves gaining and maintaining learner's attention on and interest in the

instructional task. Appeal usually focuses on the motivational aspects of instructional content

and WBI features (activities, message design, etc.)" (Gayle & Rasmussen, 2006, p. 128).

Appeal is evaluated mainly by its ability to gain and maintain learner interest and the usability of

the overall design and navigational elements. Shopping Spree has appeal for learners as it places

them in real life situations of having to figure out sales prices or tips for restaurant servers and

in a colorful and storytelling manner. The lesson is presented by the Math Wizard who floats on
Running head: FORMATIVE EVALUATION 4

the screen and explains through text and sound the ways that users can calculate percentages.

Throughout instruction the wizard asks them to perform simple calculations. After answers are

entered the wizard shows the steps to working out the problem and provides a detailed

explanation of the solution. Then as you practice the method the Math Wizard just taught you

make a basket into the basketball net each time you get an answer right or you pop a balloon on a

carnival board. You get immediate effect that is reinforcing and entertaining.

The Formative Evaluation Plan

To conduct a formative evaluation there are several steps which must be taken to ensure a

comprehensive evaluation. The first thing that an evaluator needs to do when constructing a

formative evaluation is come up with a plan.

Instructional Goals

This evaluative area assesses the accuracy, clarity, completeness and congruence of

instructional goals and objectives. (Gayle & Rasmussen, 2006, p.136) In other words, for the

instructional program Shopping Spree by e-Learning for Kids, users are asked to learn how to

calculate percentages. An evaluator would check to see if the content is accurate, determine if

the instructional goal and objective of calculating percentages was clear for users, decide if the

examples and information covered is sufficient for mastering objectives, and finally they would

check to see if all exercises and games associated with the program match objectives and goals.

Instructional Content

After a program or tools instructional goals have been analyzed, the next step is to look closely

at the instructional content. First, determine if it is accurate and presented in the appropriate

sequence. In the case of Shopping Spree, all information was accurate and the assessment

questions and games where presented after the instructional information which is a logical
Running head: FORMATIVE EVALUATION 5

sequence for most learners. Next, an evaluator should check practice exercises and directions for

clarity and completeness to ensure that learners working on their own are not confused or

frustrated. Once completeness and accuracy has been checked, it is important to determine the

amount and quality of instructional detail. Is it enough to support the concepts or skills being

taught. In the case of Shopping Spree, there is sufficient level of content if it is meant to be used

as a review or enrichment exercise but there is not enough detail if it is meant to be a standalone

instructional unit. Finally, a designer has to ask if the instructional content is appealing. It

doesn’t matter how effective your instructional program is, if it is not appealing and will not

maintain user interest then it is not going to have much of an impact.

Technology

Once instructional goals and content are evaluated you must examine the technological

elements of the program or instructional tool. According to Gayle and Rasmussen (2006),

"Technology is evaluated for functional errors such as typographical, spelling, grammar,

punctuation, and word usage" (p.136). Once the functional elements are tested it is time to

analyze technology applications such as hypermedia links. You have to examine all of the

website elements and makes sure that they are functioning properly and are instinctive. Finally,

when looking at technological aspects it is important to assess the availability of support and

assistance from the product which helps prevent or solve technical problems that may arise.

When looking at Shopping Spree I located no functionality errors and there are no hypermedia

links just links within the instructional program which all is functioning properly. The one

major downfall I found was a lack of instructional support but the company does give an address,

email, and website where questions and concerns can be addressed. Depending on who

maintains the program response time could vary widely.

Running head: FORMATIVE EVALUATION 6

Message Design

Once all other elements are in place it is time to analyze the message design of the program or

instructional tool. Is the media pleasing? Does it fit with an overall theme? What are the

graphics like? Are they emphasizing, distracting, or entertaining? Are graphical visual and

audio elements working properly and do they present a cohesive message? Are the navigational

elements clear and relevant to goals and navigation? When examining the message design of

Shopping Spree, I found that it was well designed and the overall theme and instructional goal

was clear throughout the instructional lesson and practice exercises. The graphics were visually

pleasing and fit within the overall shopping theme and they were both emphasizing and

entertaining but were not distracting at all. All navigational elements are consistently located in

a bottom bar with unified icons and identifying text rollovers in case there was any confusion.

The Main Steps

Identifying Stakeholders

The first step is to identify the Stakeholders are who evaluative information is reported to.

There are two types of stakeholders which include primary and secondary stakeholders. Primary

stakeholders are more directly involved in production and delivery. Often, they have ultimate

decision making authority. For Shopping Spree the Primary Stakeholders are the global

nonprofit organization called e-Learning for Kids or EFK. They designed the program and are

the only ones with editing or implementation privileges. The learners that use the program are

also primary stakeholders as they provide feedback (or their parents do) on the success of the

program. There are also some secondary stakeholders who are indirectly affected or loosely

responsible for or interested in the instructional program or tool. In Shopping Spree there could

be a number of secondary stakeholders such as Curriculum and Instruction specialist that are
Running head: FORMATIVE EVALUATION 7

using the program in their schools with students, parents, teachers, or possibly even school

boards.

Evaluative Product

Once the stakeholders have been identified it is time to determine the evaluative product(s).

"What is being evaluated?, focuses on the specific instructional materials, processes, or products

that will be examined during formative evaluation" (Gayle & Rasmussen, 2006, p. 144). There

are many types of design materials such as "design plans, storyboards, instructional content,

instructional strategies, interface, navigational features, message and visual design features,

WBI prototypes, assessment tools and scores, practice exercises and more" (Gayle & Rasmussen,

2006, p. 144). In a pre-produced educational product it may be impossible to examine

storyboards, design plans, and other preplanning manipulatives because that information is often

not made public. Shopping spree was analyzed using the interface which is easy to get around

and interpret, the navigational features which are clearly located in an uniform location

throughout the lesson or practice exercises, the message and visual design which were of high

quality, and the practice exercises which fit perfectly with the theme and were an accurate

assessment lesson content presented.

Evaluators and Reviewers

The next step in the formative evaluation process is to identify your evaluations and

reviewers. According to the Joint Committee on Standards for Educational Evaluation, "An

evaluator should be someone who is competent and trustworthy, who possesses substantive

knowledge, technical competence, integrity, experience, public relation skills, and other

characteristics considered necessary by stakeholders"(Gayle & Rasmussen, 2006, p. 145). For

the purposes of this assignment, I am the reviewer and evaluator. I have a number of the
Running head: FORMATIVE EVALUATION 8

recommended requirements set forth by Gayle and Rasmussen. Once an evaluator is established,

it is important that their roles are based in a purpose and the tasks requirement to fulfill that

purpose. The main purpose of the evaluator is to determine strengths and weaknesses of the

evaluation. In shopping Spree there are a lot of educational strengths, many of which I have

already discussed, but there are also a few weaknesses which must be revised. I think that there

should be levels of difficulty which can increase with user correct responses. This would allow

for some growth with the game which will increase use. I also think that it would be more

appealing if there were more locations added to the menu so that there would be an increase real

world effectiveness of practice which will increase knowledge retention. The final element to

consider is evaluator credentials. These must be identified and deemed appropriate by

stakeholders. I am an instructional technology graduate student that is in my final year of

coursework. I also have a good in-depth understanding of calculating percentages which is the

primary instructional task provided by Shopping Spree.

Evaluation Methods

Once evaluators have been determined focus must be redirected to methods of

evaluation which best fit the instructional product and need. One must determine how they are

going to gather data. The method is usually determined by the program or tools themselves or the

evaluative purpose itself. "Practical matters such as time, budget, and logistics influence the

selection of methods and tools" (Gayle & Rasmussen, 2006, p. 147). Some of the more common

methods include surveys, observations, pre or post testing, expert or end user reviews, and extant

data such as organizational reports, emails and discussions. (Gayle & Rasmussen, 2006, p. 147)

The best thing to keep in mind when selecting a method is to keep it simple. For a product like

Shopping Spree, it would most likely be best to gather data through observations, surveys, or
Running head: FORMATIVE EVALUATION 9

other factors which can be located online or created locally. Since instruction will take place

online it would be ideal if evaluation also took place online.

Evaluation Location and Timing

One thing that distinguishes WBID evaluations and traditional formative evaluations is that

they are ongoing until initial implementation. There is no traditional waiting period and as

mistakes are found they are rapidly corrected. The first evaluation occurs on the design plans.

"The design team, experts, and/or end users evaluate the WBI design plan for the clarity of its

goals, meaningful examples, and relevant instructional activities. They may review the

instructional and motivational strategies for their effectiveness, appropriateness, and potential

learner motivation" (Gayle & Rasmussen, 2006, p. 153). Although traditional models allow for

one to one, small group and field test tryouts WBID often relies on some form of tryout with end

users. An evaluator can create their own sample population to tryout material with or often the

initial implementation is the tryout for the program. It is impossible to determine the Shopping

Spree evaluation location and timing without access to their pre-planning material and design

documents.

Decisions

Once evaluations have been completed they need to be analyzed which can be accomplished

by descriptive statistics, inferential statistical tests such as the T-test or ANOVA, or

nonparametric tests such as a chi-square. You can also use qualitative analyzational tools such

as an interview with end users. Once data has been analyzed, modifications and

implementation decisions can be made.

Running head: FORMATIVE EVALUATION 10

Communicating Results

Forms of Communication

There are two ways of communicating evaluation processes and results. Verbal reports are

easy to share and can be done quickly if immediate access to stakeholders is possible. If I

wanted to report my findings to EFK, I would have to generate a written report which is more

formal and permanent. If the option is available a combination or written and verbal reports is

the best strategy but this is not often the case. Regardless of the form consistency is important.

"Other critical features of reporting practices include timeliness, brevity, clarity, and

responsiveness to the audience" (Gayle & Rasmussen, 2006, p. 159). Evaluators must always

keep stakeholders in mind when communicating evaluation results but be careful not to engage

in communication overload.

Verbal reports. These are primarily used to report project progress, problems, and technical

issues. You can deliver verbal messages on the phone, face to face or through online webcasting

or communication. It is important to remember these types of reports are temporary. If you want

anything permanent or set it in stone it is important to document as much as possible and get

signatures on revisions, additions, and omissions. Unfortunately it is impossible to determine if

verbal reports were made on Shopping Spree without access to formative documents.

Written reports. These reports can serve as progress or final reports. They can take the

form of memos, letters, emails, or other forms of written communication. Some designers and

evaluators use templates which make the communication process easier. It is also impossible to

report on the written reports generated for Shopping Spree without inside access to the

foundations planning documents and development process.

Running head: FORMATIVE EVALUATION 11

Final Report. The final report is often presented as an executive summary with full

documentation. This report discusses evaluation activities, findings, and recommendations for

stakeholders. Once you have your final report you can now publish your executive summary

through conference presentations at conventions such as ISTE and professional publications

associated with the field.

Running head: FORMATIVE EVALUATION 12

Evaluative Questions Sample

Authors: e-Learning for Kids

Your name: Brandy Milson

Date Reviewed: November 2, 2010

User Profile: Elementary and middle school students studying percentages in math class

Title of Program: Shopping Spree

Publisher of Program: e-Learning for Kids

Format (MAC or PC OR BOTH): Both

Cost of program: It is provided freely by an educational non- profit foundation

Subject Area: Math

Technical Requirements RAM: None

Hard drive Space: None

Additional Hardware Needed (If any): Speakers

Types of Sample Question Actions Reporting to General Comments

Questions Students

Is the goal of the A close This is often It is extremely

educational tool or examination reported to important that these
program accurate? of the lesson students is tools correlate with
Is Effectiveness

objectives lessons the instructional

Questions

and goals is overviews and needed. There is no

needed. on course stated goals for
Goals

Does it syllabi Shopping Spree.

match up
with course
objectives
and scope
and
sequence
Running head: FORMATIVE EVALUATION 13

documents
Are the goals and Examine This is often The implied goals
objectives of the goals and reported to within Shopping
educational tool or objectives students is Spree are clear and
program clear? for clarity. lessons narrowly focused on
Think about overviews and calculating
smart goals on course percentages.
syllabi
Are the goals or Think about Reported to If goals are not
objectives of the SMART students achievable then they
educational tool or goals? through are incomplete. You
program achievable? assignments can achieve an
and on the understanding of
course syllabi how to compute
percentages through
Shopping Spree.
Are the goals and Does the This is often Online delivery is the
content appropriate for delivery not reported to only way that you
the method of delivery? system students can deliver the
match the Shopping Spree
content program
What educational need(s) Examine the The Shopping Spree
does the content address? educational educational focuses on the need
need closely. need is for students to apply
If there is no outlines in the percentage
need there is syllabus and calculations to real
no need to scope and life scenarios
use the sequence
program or documents
tool
Are the educational Examine the Objectives will Shopping Spree does
objectives clearly stated? product or be stated not explicitly state
Content

tool to see if through verbal goals

objectives and written
are stated communicatio
n
Is the content presented in Look closely Sequence is Appropriate
a logical sequence? at the scope often presented sequence is
and in an important. Shopping
sequence for assignment spree is an excellent
logical errors schedule. example.
Are there clear instructions Re-read There are Shopping Spree has
throughout the program directions reported to excellent directions.
carefully for students both
errors verbally and
through text
Running head: FORMATIVE EVALUATION 14

using
Shopping
Spree.
Is the content appropriate Determine This is often Shopping spree is
to the stated audience? appropriate not reported to designed for
age range students. elementary and
middle school math
students
appropriately.
Is the information Make sure If any Shopping Spree
complete, covering the there are no omissions are seems complete.
content properly? accidental discovered you
omissions can report to
for content students
verbally or
through written
directive.
Are reference citations Check for a Report Shopping Spree has
provided? references references to no reference
page students citations.
through written
directive so
that they may
look up
primary
sources if they
need to
Do the instructional Look at You can report Shopping Spree is
activities promote Blooms to students more of a drill and
thoughtful and reflective taxonomy. verbally or practice program.
responses or Determine through written
discussions? the level of directive.
instruction
Do the activities Look at the You can report The Shopping Spree
promote learning? activities to to students activities promote
determine if verbally or learning.
they are through written
educational directive.
in nature.
Does the site contain any Look closely You wouldn’t This is very
extraneous and
for possible report this to important to
unsuitable vocabulary,
language or concepts, inappropriate students if stakeholders and
bias, or stereotyping? language, these elements there are no loaded
bias, and exist, you language present in
stereotyping. simply Shopping Spree.
wouldn’t use
Running head: FORMATIVE EVALUATION 15

the program or
tool.
Does the program project Look for You can report There were no biases
any obvious bias? Are bias and to students or misleading
there an errors or errors verbally or omissions in
misleading omissions? through written Shopping Spree,
directive.
Does the content Analyze the You can report Shopping Spree
emphasize depth or primary to students emphasized
repetition? instructional verbally or repetition
task through written
directive.
Is the content subject Look for a You would not Shopping spree had
matter interesting, lively, story, look report this to an excellent story
or compelling? for elements students but line and was
of interest they will interesting and
surely report engaging.
this to you
Is there a difficulty range Look for a You can report Shopping spree did
available? difficulty to students not have a difficulty
range for verbally or range but it was
enrichment through written designed for age 12.
or recurring directive.
use
Look for You can report
Are references,
supporting to students
bibliographies, or other
material through written
supporting evidence
directive.
provided?

Do the technology Look closely You can report When designing

applications function at links, to students online instruction it
properly? audio files, verbally or is important to
video files through written update links and
Technology

and all other directive. material often. All of

technology Shopping Spree's
associated audio, video, and
with the other files are
program or working properly.
tool.
Where materials easy to Access is You can report Shopping Spree is
access by the students? crucial make to students easily accessible
sure there verbally or from any computer.
Running head: FORMATIVE EVALUATION 16

are no access through written

issues for directive.
students
Are copyright and Look for You can report If copyright
intellectual property not copyright to students violations exist don’t
violated? violations. verbally or use the product. The
Know the through written Shopping Spree
law! directive. created EFK owns
the copyrights fir the
program.
Is the program easy to Instillation You can report If you can't install the
access or install? problems are to students program then it will
a nightmare verbally or never be used. There
make sure through written is no instillation
that the directive. required for
program is Shopping Spree
compatible
with student
hardware.
Are messages an Look for an You can report Theme integration is
integrated whole? overall to students important and in
theme. verbally or Shopping Spree is
through written the shopping theme.
directive. You have to calculate
percentages that you
would have to
perform on an
average shopping
Message Design

trip.
Is there quality student Is there Is there quality Is there quality student
interaction with content? quality student interaction with
student interaction with content?
interaction content?
with content?
Once students have Test this with You can report Shopping spree is easy
learned the basics of the students and to students to use and has on-
software is it easy to use? aids verbally or going instruction
through written prompts
directive.
Do students require Determine the You can report Shopping Spree
additional training amount of to students requires no training.
training verbally or
needed to use through written
the program directive.
or tool
Running head: FORMATIVE EVALUATION 17

Is the program easy to Test this with You can report Shopping Spree is very
learn to use for first time students and to students simple to use on the
students? Have students aids verbally or first time because of
learned how to use similar through written on-going directions.
programs? directive. First time use is
important because end
users will never come
back if they are
confused or frustrated
by their initial
experience
Do supporting graphics Examine You can report Shopping Spree has
and features enhance each graphic to students excellent enhancing
learning without for useful verbally or graphics that
distracting? ness and through written illustrate the
purpose directive. Most concepts well.
often students
will report it to
you
Is the appropriate voice Analyze the You can report Shopping Spree uses
used in expressing the tone of to students a Math Wizard
content to the learners? voice and verbally or character that is
determine if through written appropriately
it is the right directive. Most matched to the target
fit often students age group
will report it to
you
Was humor used Look for You can report Shopping Spree
appropriately? uses of to students doesn’t really use
humor and verbally or any humor.
determine if through written
appropriate/ directive. Most
inappropriate often students
will report it to
you.
Are directions clear? Examine You can report Shopping Spree
directions to students directions are clearly
throughout verbally or and concisely
the entire through written presented.
educational directive. Most
program for often students
clarity will report it to
you.
Does the text stand- Look for You often do Shopping spree
alone if graphics are cross not report this doesn’t have much
unavailable? compatibility to students. stand-alone text.
and analyze
Running head: FORMATIVE EVALUATION 18

graphical
elements
Are the goals stated Examine This is often
clearly and concisely? goals and reported to
objectives students is
for clarity. lessons
Think about overviews and
smart goals on course
syllabi
Is there congruence Check for This is often Shopping Spree is
between the unity and mapped out for unified.
instructional goals and correlation students
content? through an
assignment
Goals

breakdown in a
course syllabus
Is the content Examine if This is often Shopping Spree is
information clearly and information reported to clear and concise
Efficiency Questions

concisely presented? time students is

consuming? lessons
overviews and
on course
syllabi
Is it appropriate to the Examine the This is often Shopping Spree is
discipline? course reported to designed for math
objectives to students is courses.
determine lessons
appropriaten overviews and
ess on course
syllabi
Is it timely, up to date? Look for This is often Percentage
Content

outdate not reported to calculations are static

information students and do not change
with time.
Is access to the Look for You can report No access to
instructor or other outside to students instructors or others
learners provided? access verbally or provided by
possibilities through written Shopping Spree
directive.
Is the website structured Examine You don’t Shopping spree is
Technology

appropriately? navigational usually report structured

hierarchies this to students appropriately.
and site unless it is not
maps
Running head: FORMATIVE EVALUATION 19

Do the technology Check links, You don’t Shopping Spree

applications function and all other usually report functions
easily and efficiently aspects this to students appropriately
unless it is not
Is the organization and This is best You can report The organization and
structure of the message analyzed to students structure of Shopping
coherent? through a verbally or Spree is coherent
scope and through written
sequence directive.
document
Are the titles and Check the You can report Shopping Spree
subtitles used to site to students doesn't use titles and
organize the content? navigation verbally or subtitles
map through written
directive.
Message Design

Are there synchronous Examine the You can report Shopping Spree
and asynchronous types types of to students doesn't use
of activities for students activities. verbally or synchronous and
and instructor? Remember through written asynchronous types
variety is directive. of activities
important
Are the goals relevant to Ask learners They usually Shopping Spree's
learners? what they inform you goals are relevant to
hope to get first. learners as they are
out of the often consumers of
instructional products which go on
program sale and they eat at
restaurants and must
Goals
Appeal (usability)

learn to calculate
tips.
Is the content Ask student The usually Shopping Spree is
interesting/challenging? aids and inform you. interesting and
others if the challenging.
program is
Content

interesting or
challenging
to them.
Running head: FORMATIVE EVALUATION 20

Are typographical, Have You can report Shopping Spree has

spelling, grammar, another to students no functional errors
punctuation errors person check verbally or
distracting? for through written
functional directive.
errors that
you have
overlooked
in the editing
process
Is navigation easy? Have a first Students will Shopping Spree's
Technology

time user report to you navigational

test it out when the elements are easy to
navigation is identify and
difficult. sequence is clear.
Are the message and the Review the Students will Shopping Spree is
media pleasing? media for the report to you pleasing and fun.
program and when they are
determine if displeased
the effect is
pleasing
Do supporting graphics Analyze Students will Shopping Spree is
and features enhance graphics for report to you graphics are not
learning without clarity to when they are distracting
distractions? identify distracted
distractions.
Are the graphics, Analyze Students will Shopping spree
animations, and sound multimedia report to you multimedia is clear
clear? elements when things
with an aid. are unclear
Do any tutorial packages Examine the You can direct Shopping Spree
come with the educational program or students to does not have help
program or tool? tool for tutorials and files or tutorials
Message Design

Are help files or a user additional help files if

manual available? Are help needed through
they written in an easy to
written or
read manner?
verbal
communicatio
n
Running head: FORMATIVE EVALUATION 21

Are the screen layouts Analyze Students will Shopping Spree

appropriate to the screen report to you screen layouts are
content and goals? layouts to when screen appropriate
see if they fit layouts are
the theme distracting or
and confusing
educational
goals
Does the program offer Look for You can Shopping Spree
appropriate feedback? sources of provide gives feedback after
feedback students each practice game
used in the information on or question
program or feedback
tool through verbal
or written
directives
Would this program Analyze You can Shopping spree could
operate smoothly in a your communicate be implemented in a
classroom with multiple students, operation classroom if students
learners? number of procedures and were given
computers, requirements headphones for their
and through verbal workstations.
classroom and written
temperament directives
Can this program be used Look for You can Shopping Spree can
as enrichment or enrichment communicate be used as
independent practice? or enrichment of independent practice
independent independent but isn’t very useful
practice by practice for enrichment since
assessing through verbal it focuses on drill and
access, and written practice
difficulty, directives
and potential
higher level
thinking
possibilities
Are transitions smooth? Are Students will Shopping Spree has
transitions often tell you if smooth transitions
between they are not
instructional
content
smooth
Running head: FORMATIVE EVALUATION 22

Are graphics Analyze the You normally Shopping spree

informational, skill purpose for wouldn’t includes graphics
building, or edutainment? each graphic. report this to which do all three
Graphics students purposes.
that are
designed for
entertainmen
t should be
used
sparingly
Can users personalize Determine ifYou can There is no
content? content can communicate personalization with
be edited and
through verbal Shopping Spree.
personalizedand written
directives
Is printing easy? Test pages You can You cannot print
for printing communicate Shopping Spree
procedures exercises
through verbal
and written
directives
Are prompts and menus Go through Your students Shopping Spree
clear? the program will often Prompts and Menus
closely communicate are clear.
looking at with you if
prompts and prompts or
menus, menus are
unclear
Is "how to quit" easy to Look for the You can Shopping Spree has a
find? navigational communicate quit icon located in a
program procedures uniform location at
which allows through verbal the bottom of each
you to exit and written page.
the tool or directives
program
Can a session be saved and Check to see You can Shopping Spree
re-entered? if the communicate cannot be saved.
program or procedures
tool allows through verbal
you to save and written
progress directives
Running head: FORMATIVE EVALUATION 23

Does the software track Check to see You can Shopping Spree has
student progress? if there is a communicate no tracking tools
monitoring through verbal
tool which and written
tracks directives
student
progress and
performance
Does the program have Look for You can Shopping Spree has
speech capabilities accessibility communicate verbal and written
tools through through verbal text directives
the program and written written throughout
directives but no explicit
accessibility tools
*Does the site have the Ask student Students will Shopping Spree
potential for developing aids and often helps students gain
confidence and
satisfaction as a result of other is communicate confidence in their
learner effort? completion this to you. percentage
of the conversion skills.
program was
satisfying
and why or
why not?

Many of the questions in this table are from

Herring, D. F., et. al., (2005). Multimedia software evaluation form for teachers. Education,
126(1), 100-11.
Those questions highlighted in green have been obtained from
Branch, R., Kim, D., Koenecke, L. Evaluating Online Educational Materials for Use in
Instruction. Retrieved from http://www.libraryinstruction.com/evaluating.html
Running head: FORMATIVE EVALUATION 24

Assessment for Shopping Spree

Instructions for assessment: After testing all five located provided by the online educational program
Shopping Spree, evaluate the instructional program by circling the point Y or N in the boxes on the check
list. Evaluations will be collected and used for revisions.

Checklists
Program Evaluation
Quality Item Mandatory Possible Points
Y/N (only if the
Answer is yes
Are points marked
Shopping Spree includes clear directions for their instructional topic of choice Y 20
Shopping Spree has met the criteria for effectiveness, efficiency, and appeal Y 20
Multimedia elements of Shopping Spree are appealing and functional Y 10
Shopping Spree is age appropriate for its target audience Y 10
Shopping Spree provides appropriate feedback Y 10
Shopping Spree content is closely aligned with learning objectives Y 20
Shopping Spree is engaging and challenging for learners Y 10
Total Points 100
Final Grade
If a score of less than 60 points is obtained then instructional objectives have not been met and revisions must be
made to ensure quality

FeedbackComments:___________________________________________________________________________
_____________________________________________________________________________________________
_____________________________________________________________________________________________
_____________________________________________________________________________________________

Instructional Objective Evaluation (Post Test Survey)

Please complete the post

Instructional Objective Yes No
Were the entry skills for this program effectively communicated before the
workshop?
Did the program adequately teach about calculating percentages
Do you feel that this program adequately prepared leaners to convert percentages
in an authentic scenario?
Were the instructions and navigational elements of Shopping Spree helpful?
Do you think you would use Shopping Spree to teach percentages in the future if
given the appropriate learning task?
Running head: FORMATIVE EVALUATION 25

Evaluator Feedback Form

What would you change to improve Shopping Spree?
_____________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________

What did you like least/most about this Shopping Spree?

________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________

What do you feel you need further assistance on?

___________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________

Do you feel that that the instructional approaches used in this Shopping Spree met learner needs
and instructional objectives?-
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
Running head: FORMATIVE EVALUATION 26

Needed Actions

The chart above documents needed actions through several evaluation phases. Mostly it

is important to closely analyze individual elements and make revisions or clarify information as

needed. If you are using a program revised by others, often revisions are not possible. However,

you could clarify vague elements in students if they are small and do not interfere with the

engagement and skill acquisition.

Reporting to Students

There are many ways to report information to students. Often the most effective method is

simple written and verbal directives. When transmitting information to students it is important to

keep it simple and not cause cognitive overload or frustration. Sometimes reporting is as simple

as writing the directions for an assignment or constructing a well-defined course syllabus.

Regardless of the method on some elements such as clarity, goals, and task difficulty

communication is important.

Conclusion

Formative evaluations are important to WBID because they assess the ongoing progress of a

program or tool. Programs must be analyzed for effectiveness, efficiency and appeal because

they are often predictors of successful programs and tools. You also must look at the four parts

to a formative plan and the six main steps involved in the design plan. Finally, you must

determine actions that you will take as well as how you will communicate evaluative

information to primary stakeholders such as designers, investors, administrators and teachers.

Hopefully this paper explains how to conduct a formative evaluation.

Running head: FORMATIVE EVALUATION 27

References

Branch, R., Kim, D., Koenecke, L. Evaluating Online Educational Materials for Use in

Instruction. Retrieved from http://www.libraryinstruction.com/evaluating.html

Gayle, V. D., & Rasmussen, K. L. (2006). Web-based learning: Design,

implementation, and evaluation. Upper Saddle River, NJ: Pearson

Education, Inc.

Herring, D. F., et. al., (2005). Multimedia software evaluation form for

teachers. Education, 126(1), 100-11.