Running head: USABILITY OF EDUCATIONAL SOFTWARE 1

Evaluating the Usability of Educational Software

MIT APP Inventor 2

Meecha Jackson

North Carolina State University

USABILITY OF EDUCATIONAL SOFTWARE 2

Abstract

Education software, like all other educational material, should be evaluated before it is used in

the classroom. Educators need to be able to evaluate educational software for usability and

learning so that they can make decisions about what software to purchase and how to use the

software in the classroom. Checklist are a very common approach to evaluating software. They

measure and qualify the software based on the students learning outcome, but provide no data

based on the comprehensive user experience (UX), The aim of this paper is to emphasize the

importance of educational software usability, how it underwrites the user experience, and to

examine methods of testing usability.

A structured evaluation method known as the Empirical Model is described in the study of MIT

App Inventor 2 software. The data collected is unusable in determining App Inventor usability.

Keywords: educational/Instructional software, user experience, usability, and SUS evaluation.

USABILITY OF EDUCATIONAL SOFTWARE 3

Usability of Educational Software

Their Effect on the User Experience

Introduction

One of the most appealing and things about technology is when it’s new. Schools and

teacher often get into trouble when they fall into the snare of “new technology” because they fail

to ask themselves one question, is it right for my classrooms? The fact is the appeal of being new

is a powerful motivation, but for schools this can have disastrous consequences without proper

research and planning. To be successful schools need to follow the right set of strategies when

choosing new classroom software. One crucial factors that affect the suitability of software is its

usability, and whether it yield a positive user experience.

User Experience: defined

User experience, the term was coined by Don Norman, refers to a customer’s total

experience when it comes to interacting with a product, from how the users feel about the

product to, acquiring it, to using it. Good designs result in positive user experiences, while poor

design universally results in negative experience. Thus, the goal of design should be to create

and unforgettable and pleasing user experience (Kramer, 2012).

The interest in user experience design has grown heavily in the last few years. Industries

are realizing the value of the user experience design. But with all the chatter, the term user

experience design and its definition can be ambiguous and challenging to understand. One of the

tools available to help explain user experience is Honeycomb Diagram created by Peter Morville.

The illustration is a modular approach that guides designers in defining priorities that make up a

good user, subsequently constructing a good design.

USABILITY OF EDUCATIONAL SOFTWARE 4

Figure 1User Experience Honeycomb

Marc Hassenzahl (2014) explain UX this way:

I open my eyes. Lush light floods the room, birds chatter. It is only
6:30 o'clock in the morning, but I feel well-rested and alive; time to
get up, to brew some coffee. Are you jealous of my morning
routine? Were you startled out of your sleep by a merciless alarm
clock? Was it dark outside, no birds around, and did you feel
groggy and bleary-eyed?

The experience I described in the beginning was not created by sun

and birds, but by Philips' Wake-Up Light. This is a crossing of an
alarm clock and a bedside lamp. Half an hour before the set alarm,
the lamp starts to brighten gradually, simulating sunrise. It reaches
its maximum at the set wake-up time and then the electronic birds
kick in to make sure that you really get up. Admittedly, it is a
surrogate experience, but so are love stories and travel novels. It is
artificial, but not vulgar. And more importantly, it substantially
changes the way one wakes up. It changes the experience.
USABILITY OF EDUCATIONAL SOFTWARE 5

Usability: the concept

Usability refers to the quality of a user's experience (UX) when interacting with products

or systems, including websites, software, devices, or applications. A system is considered usable

when it provides a service and is of use to a person. According to Rosenzweig, “Usability is

often seen as a subset of the UX. Nielsen links five attributes to measure usability: easy to learn,

efficient to use, easy to remember, few errors, and pleasant use 1. Usability.gov who is the

leading resource for user experience (UX) best practices and guidelines, states that usability is a

combination of factors including: intuitive design, ease of learning, efficiency of use,

memorability, error frequency and severity, and subjective satisfaction (U.S. Dept. of Health and

Human Services, 2006)

Designing education software for usability is not easy, it involves establishing user

requirements for a new system or product, developing design solutions, prototyping the system

and the user interface, and testing it with selected users on multiple platforms. However, before

any usability design or evaluation action can begin, it is necessary to realize the basis of use for

the product, i.e. the goals of the target audience, and environment in which the product will be

used in.

Based on these opinions about usability I may conclude that there are two broad areas to

collect relevant data: system and user performance and user satisfaction in using it. In the

remainder of this paper I focus on methods of and technique of conducting a usability test. Then

I will describe the usability test carried out on MIT App Inventor 2 service.

App Inventor

1
Nielsen uses the following categories: learnability, efficiency, memorability, error, and satisfaction
USABILITY OF EDUCATIONAL SOFTWARE 6

MIT App Inventor for Android devices is a web-based service designed with the goal of

teaching middle and high school student the fundamental of programming. The service uses

Blocky programming language, that represent coding as interlocking graphical puzzle. It is

opened source, so users can freely access, use, change it, and share as they desire. Through freely

available curriculum and tutorials, users can rapidly build fully functional apps for smartphones

and tablets. See images.

Figure 2 App Inventor User Interface Structure

iiFigure 3App Inventor Block Design
USABILITY OF EDUCATIONAL SOFTWARE 7

Figure 4 Sample App I Created

Testing Method

In my study, the learner task method was selected to test the usability of App Inventor.

Learner task evaluation, also known as empirical evaluation, test applications with a user

performing a set of predetermined tasks. This method usually produces the most reliable and

effective assessment of applications usability. It is particularly aimed at non-experts in user-

centered design and evaluation (Dillon, 2001). A good practice when selecting a method, is to

also consider techniques for data collection that can be applied.

Technique

First, I selected three do it yourself app projects that are easily obtainable online as the

contextual framework of the usability test, one more complex than the other. I attempted to
USABILITY OF EDUCATIONAL SOFTWARE 8

complete every step as laid out in the step by step tutorial Second, I used the System Usability

Scale (SUS) tool created by John Brook in 1986 for measuring the usability, and user

satisfaction. SUS has a proven track record as a reliable evaluation tool. It correlates well with

this study, allowing for subjective measures of usability

The System Usability Scale is a Likert Scale 2 which includes 10 questions which users of

your website will answer. Users will rank each question from 1 to 5 based on how much they

agree with the statement they are reading. 5 means they agree completely, 1 means they disagree

vehemently.

Interpreting SUS Scores

The user’s scores for each question are converted to a new number, added together and

then multiplied by 2.5 to convert the original scores of 0-40 to 0-100. Though the scores are 0-

100, these are not percentages and should be considered only in terms of their percentile ranking.

Based on research, a SUS score above a 68 would be considered above average and anything

below 68 is below average,

My overall score was 74, however I am not sure if that is a true indicator of user

satisfaction or usability. I like and enjoy using App Inventor. These biases can result in poor test

performance and flawed data interpretation. It is worth noting that best practices propose using a

2
Likert (1932) developed the principle of measuring attitudes by asking people to respond to a series of statements
about a topic, in terms of the extent to which they agree with them, and so tapping into the cognitive and affective
components of attitude
USABILITY OF EDUCATIONAL SOFTWARE 9

combination of evaluation methods: expert, observational, survey, and experimental will

generate different types of collected data.

The System Usability Scale is shown in the next page. The selected statements cover a

variety of aspects of system usability, such as the need for support, training, and complexity, and

thus have a high level of face validity for measuring usability of a system.
USABILITY OF EDUCATIONAL SOFTWARE 10

System Usability Scale

© Digital Equipment Corporation, 1986.

Strongly Strongly
disagree agree

1. I think that I would like to

use this system frequently 1 2 3 4 5

2. I found the system unnecessarily

complex 1 2 3 4 5

3. I thought the system was easy

to use 1 2 3 4 5

4. I think that I would need the 1 2 3 4 5

support of a technical person to
be able to use this system
1 2 3 4 5

5. I found the various functions in

this system were well integrated
1 2 3 4 5

6. I thought there was too much

inconsistency in this system 1 2 3 4 5

7. I would imagine that most people

1 2 3 4 5
would learn to use this system
very quickly

8. I found the system very 1 2 3 4 5

cumbersome to use

1 2 3 4 5
9. I felt very confident using the
system
USABILITY OF EDUCATIONAL SOFTWARE 11

Findings

I had two goals for doing this usability test: 1. to see if App Inventor is a kind of

instructional tool that I will incorporate in my classroom instructional design. 2. To determine if

App Inventor is usable by novice users or learners with disabilities.

Based on my experience, App Inventor would provide a positive user experience, to

included user performance and user satisfaction. The intuitive design alleviates difficulties

commonly experienced in programming applications, such as learning complex coding syntax.

Also, App Inventor tries to make programming easier by eliminating the traditional lines of

codes, and replacing it with a more graphical and strategic method of coding. Still, novice users

and learner with cognitive issues will struggle with the unfamiliar UI, but will gain confidence as

programming skills develop.

When should you test

You should test usability throughout product production. Here are some guidelines:

1. If possible, usability testing can and should be conducted on the current iteration of a

product before beginning any new design work, this will help you identify any good parts

to keep or emphasize, and remove troubling parts.

2. After you’ve begun the strategy work around a brand-new site or app. This will quickly

identify areas for opportunity, and reduce the amount of assumptions your design team

will make about what the user wants.

3. Test your competitors' designs to get data on a range of alternative interfaces that have

similar features to your own.

4. Conduct a field study to see how users behave in their natural habitat
USABILITY OF EDUCATIONAL SOFTWARE 12

5. Refine your design ideas and test through iteration

6. Once you decide on and implement the final design, test it again. Subtle usability

problems always creep in during implementation (Nielsen, 2012)

Conclusion

The growth of technology in all aspects of our lives has created a need for the UX concept. This

paper discusses the importance of that concept and how it pertains to the design process. It further

discussing how Usability is a key component of overall good user experience, which is the goal of any

good design. I now look at design through a different lens.

Educational Analysis

In the study, the primary investigator was me seeking to test the usability of MIT App Inventor

for education teaching configuration. The objective of the study was not just to evaluate the functionality

and ease of use as claimed by App Inventor, but to determine the usability of the tool as web-based

service in the structured classroom, and potential curriculum integration. The data collected is

unsubstantiated due to testing limitation, thus unusable in determining App Inventor usability.

Through examination, usability testing education software can be rigorous process. Some

constraints to consider: time – educators must have time design, prepare, administer the test, and analyze

the results\; financial-equipment and software must be purchased and budgeted for; space- evaluators

need a dedicated lab to perform the test.

Studies show there are no significant improvements in student achievement in reading,

mathematics or science in countries that have invested heavily in technology (Glance, 2015). Could be

said that usability tests were not performed, consequently rendering the user experience unprofitable. Just

a thought.
USABILITY OF EDUCATIONAL SOFTWARE 13

References
Brooke, J. (2017, July 19). Improving the user experience. Retrieved from Usability.gov:
https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html
Dillon, A. (2001). Usability evaluation. In W. Karwowski, Encyclopedia of Human Factors and
Ergonomics. London: Taylor and Francis. Retrieved from
https://www.ischool.utexas.edu/~adillon/BookChapters/Dillon-
evaluation%20of%20software%20usab.htm
Glance, D. (2015, September 18). Study says technology is ineffective in improving outcomes in schools.
Retrieved from Phys.org|The Conversation: https://phys.org/news/2015-09-technology-
ineffective-outcomes-schools.html
Kramer, K.-L. (2012, May 25). User experience in the age of sustainability: A practitioner’s blueprint.
Burlington, Massachusettes, United States : Morgan Kaufmann . Retrieved from
https://books.google.com/books?id=QknCExTbU4IC&pg=PA103&lpg=PA103&dq=user+experi
ence+ultimate+goal+of+any+design&source=bl&ots=icNbancl7x&sig=XPhMvA_i2QNdmjNkN
QiFbl4LyQs&hl=en&sa=X&ved=0ahUKEwjKpeuolp7VAhUJySYKHYZNDqUQ6AEITDAF#v
=onepage&q=user%20experienc
Marc, H. (2014). User experience and experience design. Retrieved from Interaction Design Foundation:
https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-
interaction-2nd-ed/user-experience-and-experience-design
Nielsen, J. (2012, January 4). Usability 101: Introduction to usability. Retrieved from Nielsen Norman
Group: https://www.nngroup.com/articles/usability-101-introduction-to-usability/
Rosenzweig, E. (2015). Usable technology can change the world. Burlington, Massachusetts : Morgan
Kaufmann. Retrieved from https://moodle-
courses1718.wolfware.ncsu.edu/pluginfile.php/348692/course/section/28518/Chapt_%21_what_i
s_UX.pdf
U.S. Dept. of Health and Human Services. (2006, August). The Research-Based Web Design & Usability
Guidelines. Retrieved from Usability.gov: https://www.usability.gov/what-and-why/usability-
evaluation.html

i
App Inventor User Interface Structure. Digital Image. MIT App Inventor. Web. July 22, 2017.
http://appinventor.mit.edu/explore/ai2/setup.html
ii
App Inventor Block Design. Digital Image. MIT App Inventor. Web. July 22, 2017.
http://appinventor.mit.edu/explore/ai2/setup.html
USABILITY OF EDUCATIONAL SOFTWARE 14