Sweeten, Diversity Project, Advanced Instructional Design EDPS 6431

User Testing Summary (Initial)

Five graduate students were tested following the designers cognitive

walkthrough, in order to clarify strengths and weaknesses of the system.
Testing methodology
A PowerPoint presentation was developed as a low-fi prototype with
hyperlinks, to enable click-navigation, like a website.
Screen-cast-o-matic software was used to record screencasts of the
user testing mouse clicks, audio, and timing.
User comments were captured in the audio portion of the recording,
supplemented with manual note-taking of comments for later review
and compilation.
Data Gathered
Data was gathered for 5 user sessions, using a Likert scale for 10 questions, with 3
additional questions inviting user-written comment. Additional key comments are
compiled from notes and recordings.

The Likert Scale Ratings:

1-Strongly Disagree, 2-Disagree, 3-Slightly Disagree, 4-Slightly Agree, 5Agree, 6-Strongly agree.

Questions
(mean/average)

Use
r1

1. Overall, the site is easy to use.

(5.6)
2. Overall, the site has clear features.
(5.4)
3. The site provided me new information.
(5.2)
4. The website was easy to navigate.
(5.4)
5. I was able to keep track of where I was on
the website.
(4.2)
6. The feedback information provided helpful
clarifications.
(4.4)
7. I would participate in future research on this
topic. (4.0)
8. The design of the website was clear.
(5.6)
9. The questions were clear.
(4.4)
10. I was able to complete the task in a

Page 1 of 5

Use
r3

Use
r4

Use
r5

Us
er
2
6

Sweeten, Diversity Project, Advanced Instructional Design EDPS 6431

timely manner.(4.8)
Time to completion
Minutes: seconds

10:
41

8:2
1

14:
58

10:
23

13:
23

Written feedback comments were provided in response to the following

3 prompt questions:
1. I found_________ to be confusing.
2. I wish the website has more_________.
3. My favorite feature of the website was__________.
Analysis of the users written comments revealed the following themes:
o 4 out of 5 users specifically mentioned the clean, simplicity of
the sites design as their favorite feature of the prototype.
o 2 out of 5 users requested more pictures or animations.
o 3 out of 5 users expressed confusion and guessing about what
the correct answers might be.
o 1 user was more specific in requesting that the videos or
animations demonstrate scenario application of the concepts
prior to test questions.
Observations and notes from the Think Aloud sessions revealed the
following themes of positive and negative experiences with the test
session:
o Users value navigation ease and feel calmer when a clean,
simple interface prevents them from getting lost.
o 5 out of 5 users commented that the repetition of the question &
feedback pattern was helpful.
While this point was interesting to note, the final design
will focus on less repetition and more interesting
engagement with scenarios.
o Several users commented that they learned more from feedback
when they were wrong, and were less likely to read the dense
text when they were correct.
o Strategies for multiple choice demonstrated that there was more
interest in getting the questions correct and moving-on quickly,
rather than the question system creating significant curiosity
about the information.

Major Findings for Usability

The system at this initial testing phase was primarily comprised of
dense text.
The information and assessments were focused on vocabulary first,
with the intention of providing a foundation fro upcoming components.
The 6 vocabulary questions in a row were still too much
wording to cover, and the user testing suggests that in this case

Page 2 of 5

Sweeten, Diversity Project, Advanced Instructional Design EDPS 6431

with sensitive and complex material involved, media animations

or video assists with creating an accessible, relaxed attitude for
the information to be absorbed.
The information contained in questions and feedback will have greater
relevance and effectiveness when presented in a more entertaining
and less test-oriented format.

The system provided opportunities for deep learning and application of

knowledge, yet without first providing sufficient visual variety or
interest-gathering scenarios, prior to testing.
Feedback was the most effective portion of the learning.
Requiring learners to test and learn from right or wrong answer
feedback is stressful. This risks alienating adult learners who
gather information more willingly when they: 1.) see it as
immediately relevant, and 2.) feel thoroughly prepared and
confident before entering any testing scenarios.
There was frequent disbelief that some of the statements or scenarios
could occur.
This reaction validates the projects Challenge statement.
Unfortunately, the disbelief reaction also underscores the
weakness of reading text for presentation method versus
compelling video or animated scenarios to engage learners.
Visual design and animations will also allow users to watch
situations unfolding, rather than asking them to imagine.
The system was not particularly fun or engaging.

Page 3 of 5

Sweeten, Diversity Project, Advanced Instructional Design EDPS 6431

Since the subject matter is already challenging and potentially

offensive, efforts should be made to make it as entertaining and
interesting as possible.
Visual design and animations will facilitate a visual branding
style, setting a friendly tone for the project.

Major Findings for Learning

The initial system supported:
o Academic knowledge of foundational vocabulary.
o Navigation ease and clarity.
The initial system did not support:
o Interesting, non-threatening exposure to visual scenarios
(animations or video.)
o Immediate learner opportunities for scenario applicationknowledge of the priority compassion & empathy goal-states.
o Lighthearted, non-shaming, learning presented in a way that
calms learners and overcomes possible psychological resistance
to experimentation and examination of the microaggression
topics.
Priorities for Revision and Future Development
Major priorities for developing the system to completion next semester:
Incorporating story/scenarios immediately as a key to generating
interest and overcoming resistance to this information.
o Based on user testing, the highest priorities for revising the
current system prototype will be to add engaging stories and
visual variety (videos, animations, humor) to the examples and

Page 4 of 5

Sweeten, Diversity Project, Advanced Instructional Design EDPS 6431

training portion prior to the introduction of emotionally-charged

vocabulary.
o The initial system focused on academic approaches to
vocabulary strength first, elaborating and applying in progressive
steps.
o The heaviness of the topic, and the need for immediate interest
and engagement indicates a much different approach.
The project goal will not be to create experts on the topic of
Microaggressions, but rather to create a friendly introduction for those
who might otherwise remain unaware or resistance.
Revisions to the Instructional Analysis will prioritize: reducing amount
of text vocabulary, increasing scenario applications, increasing
immediate user-engagement with scenarios from the beginning of the
first module, and emphasizing follow-up strategies for after
communication errors and microaggressions have already occurred.
System Maps require complete revision to incorporate new content.

Page 5 of 5