Part 4: Evaluation

Georgia Institute of Technology

CS 3750

Team: Stressless
Sean Breyer
Casey Chae
Chelsea Chong
Katherine Fitzgerald

Part 4: Evaluation Page 2

Team: Stressless

Project Description
The Stressless groups product is an armband that measures the strain exerted on the elbow
joint. The chosen design includes a small screen interface on the armband that includes a
simple visual bar that is color-coded, which ranges from green, meaning the strain does not
put the joint at risk for injury, to yellow, which shows the increased risk of the strain, and to
red, representing a hazardous strain level. When the amount of strenuous activity enters into
states of yellow and red on the bar, the armband increases vibration in relation to the level
of hazardous strain levels.
Additionally, the Stressless product displays a line graph with more detailed pressure readings
with records of the users strenuous activity over time. There is also an accompanying mobile
application, which gives the user another way of looking up his or her activity logs. It also
includes an incident-reporting feature and medical capabilities, such as a WebMD-like
database and a channel to contact the users doctor(s).
The main users of this product are physical laborers, such as construction workers and
assembly line workers. There are many products that focus on recovering from an injury, but
Stressless attempts to prevent such injury altogether.

Evaluation Description
The basic structure of our evaluation involves our volunteer performing a re-determined set
of benchmark tasks with the purpose of allowing them to familiarize themselves with our
designs full range of functionality. The background tasks were chosen as to cover as large a
range of functionality as possible, and to cover chaining tasks together. After performing
these tasks, the volunteer will answer a brief survey reflecting on their experience, the
survey will involving them rating the design on different aspects relating to the ease and
accessibility involved with navigating the interface.
Prior to performing any tests the volunteer is informed that they will be testing a design
consisting of an armband with an embedded interface and an associated app. They are told
that the design is focused on detecting and prevent arm and elbow strain, they are not told
how this is accomplished or given any instructions for manipulating the interface. Next, the
procedure is described and they are told that they will perform a series of tasks and then be
asked to rate the systems performance on a number of criteria, they are told that the whole
process should take around thirty minutes. They are then guided to websites that simulate
the interface of both the armband application and the mobile application. The evaluation
begins by the volunteer being read the first benchmark task and are allowed to complete it at
their own pace, there will be no aid from the the experimenter if the volunteer is stuck on a
task that is noted down and they are told the next task and asked to begin on that.
Our benchmark tasks are as follows:

Part 4: Evaluation Page 3

Team: Stressless

Task 1 - Starting from the bar graph display on the armband interface, navigate to the
volume control screen and change the volume to maximum. Afterwards, change the pressure
display from a bar graph to a line graph.
Task 2 - Starting from the line graph display on the armband interface change the arm
orientation from left to right. After that, activate an emergency call, exit out of the call,
and activate the compression before returning to the line graph screen.
Task 3 - Progress through the signup and login screens on the associated mobile
application. (Inform the volunteer that much of the mobile application functionality is not
yet implemented so some of the buttons and options will be unresponsive). Submit this
previously filled out incident report.
Task 4 - Using the mobile application, open a chat window with Dr. Brown, after that
find and open an article called 10 Tips for Back Pain
After completing the benchmark tasks the volunteer is encouraged to answer a short
questionnaire and then write down any comments they have.

Description of Users
The users chosen for the study center around friends and family of the group members. This is
largely a matter of convenience, but it does establish a population with two distinct age
ranges and a number of ethnicities and backgrounds. Many of the users do not work in the
intensely physical workplaces, where we think our design could bring the most benefit. Still,
this population provides a good metric to determine whether the design appeals to a wide
audience that may not need to use it rigorously.

Design Rationale
The reasoning behind our evaluation approach was to make a comprehensive plan designed to
both address the usefulness and appeal of the design to a general population and the designs
usefulness in more specific rigorous situations. To address the designs appeal to the general
population we devised a series of simple but comprehensive benchmark tasks meant to give
new users a sense of the system with little prodding or instruction. The questionnaire is then
designed to help gauge their attitude towards different aspects of the design without having
to depend on them coming up with high-level useful feedback unaided. We believe the
design could have a lot of appeal to people working in rigorous physical occupations; since we
do not have access to many people in relevant industries, the group figured a cognitive
walkthrough would be an effective method of trying to imagine and explore different
problems that might arise in wide range of scenarios, with which the relatively average users
we are questioning would not be familiar.

Part 4: Evaluation Page 4

Team: Stressless

Cognitive Walkthrough
Task 1 - Starting from the bar graph display on the armband interface, navigate to the
volume control screen and change the volume to maximum. Afterwards, change the pressure
display from a bar graph to a line graph.
Starting from the bar graph there is no icon specifically related to volume. Since the graph
screen (line or bar) is the home-screen of the device adding a quick mute/unmute button in
the bottom right corner could be very useful. Since there is no specific volume icon to
choose, it makes sense for most users to press the gear icon, a common indicator for
settings. At this screen the user can easily see the panel for volume once they click that they
have a slide bar to increase or decrease the volume. One problem is that it takes at least
three clicks to change the volume, and since there is no method to directly return to the bar
graph home-screen it takes another two clicks to return. It makes sense that most users will
choose a volume they like and not vary from it; most of the users interaction with volume is
predicted to deal with muting it for a quiet environment or to have a conversation with
someone and afterward returning the volume to their usual preferred level. Again, a mute
button on the graph screen would greatly reduce the amount of times a user has to drill into
the interface to change the volume.
Task 2 - Starting from the line graph display on the armband interface change the arm
orientation from left to right. After that, activate an emergency call, exit out of the call,
and activate compression before returning to the line graph screen.
This tasks begins by the user moving to the settings page and clicking on the panel labeled
orientation. Everything is clearly labeled and while it takes a couple steps to get to the
orientation this is a task that the user rarely will have to undertake so it is not that much of
an inconvenience. The first issue is that there is no direct way to go from the orientation or
settings page to the medical page. The only route is pressing the back button twice to return
to the graph screen and then pressing the clearly labeled medical button. On the medical
page, an emergency call can be made; one issue was found that there is no confirmation
button for something as serious as a medical call and that could cause some accidental
dialing. Cooling and compression are both clearly labeled and clearly indicate when they are
active or inactive. The lack of a confirmation button on the medical call is a definite design
flaw, and it brings up the point that a sleep button for the whole device might be useful to
disable any interaction with the screen when the user does not require it.
Task 3 - Progress through the signup and login screens on the associated mobile
application. (Inform the volunteer that much of the mobile application functionality is not
yet implemented, so some of the buttons and options will be unresponsive). Submit the
previously filled out incident report.
The signup screen and logon screen are very typical and should not pose many problems to
most users. Currently it allows you to remember a username and a password to save the user
some time, it could be worthwhile even skipping this step and having the system
automatically login. This could be convenient since typically the app will only be used by one
user. The incident report option on the page after the user logs in is clear and remembers a
lot of the information to save time, and the option to see a history of your incident reports.

Part 4: Evaluation Page 5

Team: Stressless

From there, the saved incident report from the users history is submitted by clicking the
Submit button.
Task 4 - Using the mobile application, open a chat window with Dr. Brown. After that, find
and open an article called 10 Tips for Back Pain.
After logging in, the main screen includes an option called Chat with a Doctor; choosing this
option to fulfill the task is quite apparent. It would be logical to put this functionality behind
a small paywall to help prevent spam, it is likely a usefully enough service that people would
be willing to pay a small amount. This monetization could also provide an incentive for
doctors to put time into the service.
To find the specified article, first, the application necessitates the return to the main screen,
where the third option is Web Search. Again, deciding which option to achieve the tasks
goal is obvious. Once Web Search is selected, the next page comes up with a search engine
for specific articles. Since the application is already filled out, the search engine already has
a keyword that has been searched, and the specified article from the task is the only result
from the search. After clicking the result, the article is reached. The web search could use
some featured articles, light curation could make this a service people use instead of just
web searching outside of the app.

Summary of Cognitive Walkthrough Results

First, the group found that a mute/unmute button on the front page would save lengthy trips
to the volume screen. Then, it was advised that a shortcut to the home screen should always
be available. Abiding by Reverse Fitts Law, the medical call button needs to have a
secondary confirmation, without it, it is too easy to accidently press. Along with this, the
application automatically logging in makes sense since only one user would typically use this
device. Monetization on the chat to a doctor feature found on the mobile application would
reduce spam and provide an incentive for doctors. Curation and featured articles would give
the applications search feature more targeted appeal.

Cognitive Walkthrough Discussion

The cognitive walkthrough revealed a lot of important problems particularly with the
armband app. We noticed chaining multiple tasks together was overly difficult; this was in
part due to a lack of a global home button. Also, it appeared to us that users might be having
to do too much work, because important functionality was too deep in the menus. Regarding
the mobile application, we noticed a major flaw: our chat to a doctor feature was not
particularly well thought out. There was no barrier to entry to prevent spam, and doctors
had no real incentive to take part in the service. Also, the walkthrough revealed a weakness
of the mobile application in general; most of the functionality it provides could be found
elsewhere, so we might need to put more work in to make it something more clearly useful.

Part 4: Evaluation Page 6

Team: Stressless

New User Testing with Questionnaire

The questionnaire that we had the users fill out after attempting to perform the tasks is shown below.
A discussion of the results of our testing follows.
Questionnaire
1 Strongly Disagree 2 Disagree 3 Neutral 4 Agree 5 Strongly Agree
The interface is visually appealing
Armband Interface:
Mobile Application

1
1

2
2

3
3

4
4

5
5

2
2

3
3

4
4

5
5

The interface is easy to use

Armband Interface:
Mobile Application

1
1

Navigating to specific commands within the interface is logical and efficient

Armband Interface:
Mobile Application

1
1

2
2

3
3

4
4

5
5

4
4

5
5

Specific commands can be reached quickly

Armband Interface:
Mobile Application

1
1

2
2

3
3

The software is not missing any important functionality

Armband Interface:
Mobile Application

1
1

2
2

3
3

4
4

5
5

This design provides functionality I could see myself personally using

Armband Interface:
Mobile Application

1
1

2
2

3
3

4
4

5
5

Much of the functionality this system provides seems useless

Armband Interface:
Mobile Application

1
1

2
2

3
3

4
4

5
5

I needed to learn a lot of things before I could get going with this system
Armband Interface:
Mobile Application

1
1

2
2

3
3

4
4

5
5

The design provides appropriate medical functionality for easing elbow strain and injury
Armband Interface:
Mobile Application

1
1

2
2

3
3

4
4

5
5

Part 4: Evaluation Page 7

Team: Stressless

The functionality in the mobile app should be integrated into the armband
Armband Interface:
Mobile Application

1
1

2
2

3
3

4
4

5
5

3
3

4
4

5
5

The mobile application is unnecessary

Armband Interface:
Mobile Application

1
1

2
2

If the design indicated I was undergoing a dangerous amount of strain, I would stop what I am doing
Armband Interface:
Mobile Application

1
1

2
2

3
3

4
4

5
5

I could easily see myself accidentally triggering an emergency call on the armband
Armband Interface:
Mobile Application
Comments:

1
1

2
2

3
3

4
4

5
5

Part 4: Evaluation Page 8

Team: Stressless

Questionnaire Results
The results of our testing on other users is shown below. We collected this information from
five different users, generally explained in the previous Description of Users.

Part 4: Evaluation Page 9

Team: Stressless

Discussion of Questionnaire Results

The purpose of the questionnaire was to get a general sense of how the general public would
feel about our design: its usefulness, functionality, and general appeal. Results were very
positive towards the general design of the armband and interface; friends and family formed
most of the user base and this likely skewed the data strongly towards the
positive. Consistently, users strongly agreed that the design was appealing, logically
organized, and easy to learn. Users were consistently less positive towards the mobile app
than the armband; the average also leans toward the claim that the mobile application was
unnecessary. One very important piece of data is that users almost universally strongly
agreed that the functionality the app and the armband provide the users view as useful, but
they are on average about neutral on whether they themselves would ever use the
functionality that either interface provides. One user summarized this in their comment I
feel like it is a good product, but it is very targeted toward a certain part of the population. I
dont really see myself using it in everyday life. This is definitely a big concern for us and we
likely need to test on a much larger and more disparate user base to determine whether our
device has general appeal or is just targeted towards specific groups. Another aspect of the
questionnaire results is that the mobile application consistently scored lower than the
armband under nearly every criteria; one possible reason is much of the mobile applications
functionality is not enabled yet and that might have made it difficult for users to understand
it and connect to it.

Summary of Design Implications

Unfortunately, it is difficult to return to the homepage from within the armband interfaces
other pages. The group has also learned that it is too easy to accidentally activate an
emergency call with the armband application. Along with this, there is no easy mute button
present on the armband interface. Due to the fact that both interfaces are visually appealing,
no significant visual changes should be made. Furthermore, both interfaces are generally easy
to use and can be used with minimal training or instruction. Although the idea of an
accompanying mobile application was well-liked initially, it does not provide functionality
general users find overly necessary, according to the groups evaluation. Mobile application
may be missing too much functionality to be properly tested. Generally, people liked all
functionality available; the mobile application just needs more appeal.

Part 4: Evaluation Page 10

Team: Stressless

Design Changes
Much of the changes to the armband interface are clear-cut. Generally, the feedback from
the questionnaire was very positive towards the armband interface, but there were some
issues discovered in the cognitive walkthrough. One important addition would be a mute
button to the home graph page of the armband interface. Our reasoning is that users
typically find a value they want and stick with it; most of their interaction with the volume
settings will be turning it completely off maybe to have a conversation and then turning it
back to its previous volume. Also, a mute button on the graph page, where the user obtains
visual feedback, the group believes would greatly reduce the number of times a user has to
drill into the somewhat obfuscated volume page. Another aspect learned from our cognitive
walkthrough is that since the armband interface is structured around the graph page as a
home page, it would greatly reduce time navigating the interface if all pages had a button to
return to the graph screen. The last thing we noticed was the call medical help function on
the armband interface is too easy to accidentally trigger; a secondary confirmation could
prevent most accidental triggers.
The mobile application is more complicated because the questionnaire response from a
general user is much more mixed. One clear change we found from the cognitive walkthrough
is that a small charge should be added to access the Chat with a Doctor feature. This
would both prevent spam and would create a cash flow that could be an incentive for doctors
to sign in to the service. One important aspect from the questionnaire is that users were not
seeing a lot of use in the application. A focus on featured articles, suggestions, and targeted
advice might be a method of making the application more appealing. In conclusion moving
forward, the group believes the armband interface is solid, so the changes that will be made
are aimed towards making it more efficient and reliable. We also think the mobile
application has good functionality, moving ahead the group aims to make it more appealing.