Usability Test Report for KSER.

org
Date of Report: March 17, 2009
Date of Test: March 10-11, 2009
Location of Test: Edmonds, WA

Executive summary

Three participants were selected to perform five tasks on the KSER.org website. The tasks ranged from
simply finding the site, to more complex actions like making a donation, finding information about a
program, listening to the live audio stream and locating/viewing an archived playlist. All the users were
able to complete their tasks, but all said the site was confusing, hard to use, incomplete, and visually
lacking. One person said the site seemed like an empty shell – a pretty interface with no real content
inside.

Methodology

Who we tested
Three participants, having the following characteristics, evaluated KSER.org.

Age Gender

18-25 2 Women 1
26-39 0 Men 2
40-59 1 TOTAL (participants) 3
60-74 0
TOTAL (participants) 3

What participants did

Participants spent between 30-60 minutes start to finish. They were first asked a series of questions,
then guided to complete 5 tasks on the KSER.org website. There were asked a series of post-test
questions and then completed a questionnaire.

What data we collected

We observed the participants as they used the site, collected verbal comments, noted task completion
success and completion times, and collected a ratings questionnaire.

Source: Usability.gov 1
Major findings and recommendations

Major Issues
 Program information is wrong or missing.
 “Listen now” link requires clicking donation link. Screen locks up.
 Archived playlists are hard to search. Hyperlinks to playlists don’t look like links.
 Placement of “Listen now” link is not obvious.
 Donation level names seem arbitrary.

Solutions
 Dedicate time and resources to clean up and complete program info.
 Review need for donation link as part of streaming process.
 Determine if another playlist search tool is available, or the cost of modifying the existing tool to
improve usability.
 Move “Listen now” link above “Live playlist” link. Consider using a button instead of a text link.
 Review donation level names. Not critically important because users ultimately select based on
dollar amounts.

Detailed findings and recommendations

Introductory Questions

Summary of introductory questions

Sample Questions
What do you do for a living?
Have you been to KSER.org before?
Responses
Product tester – video games. Retail clerk @
Macy’s. IT Tech support.
All participants had been to the site at least a few
times. Reasons for going included “browsing”
and “looking for recently played songs.”

What other radio station websites are you
familiar with?
Why did you choose that site?
Do you listen to the radio? Do you listen
online?
Sites mentioned included KEXP, KPLU,
NPR.org.
A lot more content for users/listeners. More
info beyond radio – blogs, podcasts, in depth
playlists. Looking for news.
All participants listen to radio and had experience
listening online.

Source: Usability.gov 2
Task 1 - Finding the Website
Participants were asked to fine the site www.kser.org and to give their impressions.

Number of participants 3
Percent successful 100%

Findings Recommendations
All participants completed the task with
ease by finding the site in < 1 minute.
All participants said the purpose of the site
was to tell people about the radio station
and give information about the program
schedule.
All participants said the options on the
home page were clear.
Participants said they would start in
different areas: About section; Schedule
section; pull up an audio stream.
2 participants had issues with the artwork. Review the color scheme.
Colors bland, artwork muddy.
2 participants had browser issues – Firefox Debug the site on multiple platforms (Mac, PC)
and Safari. and multiple browsers (I.E., Firefox, Safari etc.)
1 participant thought the Community section Clarify the purpose of the Community section.
was a discussion forum. Also consider adding a discussion forum as time
and resources permit.
1 participant observed a lot of content on Consider reducing the amount of peripheral
the sides of the page, particularly on the content on the sides of the home page.
lower right.

Source: Usability.gov 3
Task 2 – Making A Donation
Participants were asked to attempt to make a donation using the Donate Now! button.

Number of participants 3
Percent successful 100%

Findings Recommendations
All participants were able to navigate to the
donation page and make selections within
a couple of clicks.
1 participant remarked on the high number Consider reducing the number of donation levels
of options – more than 10. or consolidating levels.
2 participants wondered about the Better define donation levels and provide
membership levels. It wasn’t clear to them descriptions. Review the hierarchy logic.
how the levels were defined and the labels
seemed arbitrary e.g. Professional is listed
below Family.
1 participant checked “in memory of” under Test the Gift Subscription field and fix if
Gift Subscription, but could not uncheck it necessary.
later.
1 participant was reluctant to input credit Consider adding a PayPal or Virtual Check option.
card information. Would have preferred a
PayPal option.
2 participants said they would donate small Structure the order form to make it easier for small
amounts - $35 or select Other and write in donors to quickly contribute.
$10 or $20,

Source: Usability.gov 4
Task 3 – Finding Program Information
Participants were asked to determine what time the Reggae program is on.

Number of participants 3
Percent successful 33%

Findings Recommendations
1 participant searched via the program
grid. 2 participants search via the
Programs pulldown menu.
All participants ran into difficulty Organize the Programs list by music genre, or
accomplishing the task. 1 participant create a key that maps program names to music
surmised the correct program from the genres. This would overcome the biggest
word “dread” in the title House of Dread. 2 perceived obstacle to accomplishing the task.
participants failed to locate the program
without help.
All participants remarked on the lack of Build out any program pages that do not already
descriptive program information on the have some basic descriptive content. Gather
House of Dread page. Looks like just the content by having program hosts complete a
default info is being displayed. template.
2 participants said the program grid was Consider revising the program grid for greater
hard to read. 1 participant said the grid readability. Perhaps add a link to a landscape
was being squished by content along the .PDF view.
sides. A landscape view would be more
user friendly. 1 participant said color
bands between hours might help improve
readability.
1 participant said defaulting the list of Review whether defaulting the Regional
“Regional Information Links” to every page Information Links to every page is advisable.
is confusing. Becomes the focus of the
page when there is no program
information.
All participants said the program pages
should include a program description, info
on what the show is about and some
content e.g. sample artists, photos etc.

Source: Usability.gov 5
Task 4 – Listening To The Live Stream
Participants were asked to listen to KSER live over their computer speakers.

Number of participants 3
Percent successful 66%

Findings Recommendations
2 Mac users were able to start the live
stream within 1-2 clicks. 1 participant .
clicked on the “Listen now” link and was
prompted to download an applet. 1
participant clicked the “64k” link and
listened through iTunes.
When 1 PC user clicked the “Listen now” Check to make sure the donation link is not
link, then clicked through the “Please freezing the site.
consider donating” link, the site locked up.
Participant restarted the site and was able
to start the music.
1 participant said finding the “Listen now”
link was a “medium difficulty” task because
it is surrounded by other links. 2
participants said finding the “Listen now”
link was easy.
All participants thought the placement of
the “Listen now” link in the upper left corner
was good.
1 participant thought the “Listen now” link Consider converting the “Listen now” link to a
was too subtle. Would have preferred a button and placing it above the “Live Playlist” link.
button placed above the “Live playlist” link.
1 participant was annoyed at having to Reconsider placing a donation link in the audio
click through a donation link to activate the stream click path. Consider adding donation
stream. buttons at other non-critical places on the site.
1 participant thought there were too many Consider adding a Help link to explain the
audio stream links with no explanation of differences in the audio stream links. If “Listen
which one to choose or why. live” is the preferred method, indicate so.

Source: Usability.gov 6
Task 5 – Viewing An Archived Playlist
Participants were asked to find and view an archived playlist for the program One More Saturday Night.

Number of participants 3
Percent successful 33%

Findings Recommendations
All participants struggled with this task.
The greatest difficulty was in understanding
how to use the archived playlist search
tool. It was described as “clunky”, “not
intuitive”, “broken” and “frustrating”. 1
participant said it seemed “10 years old.”
All participants found the Archives section
within 2-3 clicks, but none understood how
to search. 1 participant knew the show
must be on Saturday, so they clicked the
“previous” link until the Saturday playlists
were displayed. 1 participant jumped to
the Program Grid to find the day and time,
then jumped back to Archives. All
participants thought it necessary to key in
the exact date and time.
No participants found or utilized the “Show”
and “Host” checkboxes without prompting
from the moderator. Once identified for
them, the participants were able to use the
“Show” checkbox to search for playlists.
When playlists were displayed, 1 Display links in commonly understood format e.g.
participant didn’t understand the headers underlined blue text.
were hyperlinks to full playlists because
there were no visual cues such as
underlining.
1 participant clicked the link, but “wasn’t Consider opening the playlist as a pop-up or in
sure what was happening for a second.” another screen. Make it clear that a result is
The playlist opens below the fold if there being displayed.
are numerous headers, so it’s hard to tell if
opening the playlist was successful.
1 participant wondered why “Mic Breaks” Define “Mic Breaks” and/or leave them out. They
were included in playlists. are probably of little interest to listeners.
1 participant wondered why the search tool Consider replacing the search tool with something
displays options all on one line. Suggested more elegant or revise the layout to make the
using available blank space to spread search options more apparent.
things out (e.g. the checkboxes) for greater
usability.
1 participant noted errors, duplications and Identify and eradicate menu errors.
inconsistencies in the “Host” pulldown
menu.

Source: Usability.gov 7
 1 participant noted the lack of Help
available and wondered about the
definition of terms. “Host” could mean
server; “Show” could mean display.
1 participant said the biggest barrier to
success is failing to understand the
available search methods. Defaulting to
the date checkbox (which is not labeled)
makes it appear to be the only option.
Source: Usability.gov
Consider creating a Help file or adding some text
to explain the search tool options. Consider
adding help at other points in the site where
people are getting confused.
Label the date search checkbox to be consistent
with the other two checkboxes. Ask the user how
they want to search (Date, Show, Host) and make
it clear how to do each.
8
Exit Questions/User Impressions
NOTE: It may be useful to put responses in a table, especially if you want to list all participant responses
rather than an overall summary. Following are two examples.
Summary of user impressions
Questions
What is your impression of the site? Give it a
grade from A to F.
What adjectives or characteristics would you
use to describe the site?
What things did you like best about the site?
What things did you like least about the site?
Would you use the site on your own in the
future?
Would you recommend the site to others?
How might the site be more useful?
Are there any capabilities or information you
would like added?
Did you notice any capabilities or features
offered that you would NOT use?
Is there anything you think KSER could do
better here?
User impressions by participant
Participant No. Like best?
1 Easy to start listening
2 Program grid
3 Today on KSER
Source: Usability.gov 9
Responses
The general feeling was that the site makes a
good first impression, but that it lacks necessary
content and is hard to use. The grades ranged
from B- to C.
Superficial, clunky, confusing, attractive, clean,
cluttered, functional, not intuitive, sweet, cute,
trying (as in aspiring).
Pretty easy to get in and start listening right
away. Program schedule view. Basic schedule
on left side, “Now playing” link, not too many
requests for donations.
Didn’t like the schedule grid. Did not like the
playlist search function. Had trouble with
“Listen now” (locked up the site.) Program
pages had tons of missing info.
Yes, if there wasn’t a radio available.
Yes, it’s good for listening. Supporting public
radio is a good thing.
Sort out playlist issues. Add some info about
the shows. Fix broken functionality. Right
now there is no reason to go there beyond the
“Listen now” capability.
There is no descriptive info. Add more rich
content. Also, Archives suggests archived
programs that can be listened to. That would
add value.
2 participants said no. 1 said they were not
interested in Community Blogs or Events.
Sort out playlist issues. Add some info about
the shows. Brighten it up, make it more
visually appealing, more definition of things,
make more specific and easier to understand.
Decrease the number of pulldown options on
the front page navigation bar. Merge some of
the options. Put all blogs in one place.
Like least? Improvements
Playlist search Add prog descriptions
Listen now Make easier to use
Missing program info Archived programs