Grammarly Usability

Evaluating an Automated Writing Assistance Tool

Kylie Walker
April 23, 2016

Grammarly Usability - 1

Table of Contents
Introduction................................................................................. 3

User Assumptions...............................................................................3

Methodology................................................................................3

Subject and Testing Setup...................................................................3

System Usability Scale.........................................................................4

Results........................................................................................4

Registration and Login........................................................................4

Uploading and Checking a Document....................................................7
Evaluating Results & Changing a Document..........................................9
Producing a Completed Document........................................................9

References.................................................................................10

Grammarly Usability - 2

Introduction
While individuals can and do vary widely in their level of writing skill, almost every
writer can benefit from a second pair of eyes as documents are developed.
Sometimes, however, no one else is around or prepared to review ones writing
what can be done in these sorts of situations?
Grammarly is an automated system that purports to make you a better writer by
finding and correcting up to 10 times more mistakes than your word processor
(Grammarly, 2016b). Grammarlys products are used by millions of writers
worldwide and licensed by more than 600 leading universities and corporations
(Grammarly, 2016a). Hence, it would seem that academics and students are the
primary target market for Grammarly, though individuals employed in writingintensive fields might also find some use for the product.
As students and professionals alike vary widely in their writing abilities and
technical know-how, it is critical for Grammarly to make its products as accessible
and as easy to use as possible without negatively impacting the value of the
product. While Grammarly offers native apps and plug-ins for Microsoft Office and
various web browsers, it also includes a web-based platform that is free and that
is the focus of this usability test. (Grammarly Premium is also available, though
it costs $11.66 per month).

User Assumptions
This test makes several assumptions about the user:
He or she has attended some college
He or she has some experience using Microsoft Office or other similar word
processors
He or she is moderately technically savvy, familiar with using web-based
search engines, and comfortable using a computer

Methodology
This cognitive walkthrough seeks to understand the potential problems that might
arise as a user attempts to use the free web-based version of Grammarly. In order
to successfully use the product, a typical user will need to perform the following
steps:
1. Register for a new account and sign in
2. Upload a document and allow Grammarly to perform a check of the
document
3. Evaluate the results and change/update the document
4. Produce a completed document
Grammarly Usability - 3

Subject and Testing Setup

The subject for this particular test of the Grammarly website was a 21-year-old
male psychology major enrolled at a small liberal arts college. The test was
performed in the subjects apartment on a 13 MacBook Pro laptop computer over
a weekend; the computer was running the latest version of OS X (El Capitan,
v10.11.4), with Google Chrome (v50.0.2661.86, 64-bit) used as the browser. The
Awesome Screenshot plugin for Chrome was installed and used to take periodic
screenshots.
The test coordinator sat next to the subject at the computer and asked the subject
to perform the tasks listed above, with screenshots periodically obtained at
relevant intervals. The subject was also asked to comment verbally on the system
as he performed the tasks, and was asked to complete the System Usability Scale
(Brooke, 1986) to calculate an overall score for the software. The audio of the test
was recorded on the test coordinators iPhone for reference later.

System Usability Scale

Developed by John Brooke, the System Usability Scale (SUS) is a simple 5-point,
10-item Likert-type scale that produces an overall score of 0 to 100. For each item
on the scale, the user is asked to indicate agreement on a scale from 1 (Strongly
Disagree) to 5 (Strongly Agree). The scale items are listed below:
1.
2.
3.
4.

I think that I would like to use this system frequently

I found the system unnecessarily complex
I thought the system was easy to use
I think that I would need the support of a technical person to be able to use
this system
5. I found the various functions in this system were well integrated
6. I thought there was too much inconsistency in this system
7. I would imagine that most people would learn to use this system very
quickly
8. I found the system very cumbersome to use
9. I felt very confident using the system
10.
I needed to learn a lot of things before I could get going with this
system
To score the SUS:

Subtract 1 from scale position for odd-numbered items, producing a score of

0-4
Subtract scale position from 5 for even-numbered items, producing a score
of 0-4
Sum all scores and multiply by 2.5
Grammarly Usability - 4

Results
Registration and Login
The user was first asked to register for a new account and log into the web-based
Grammarly interface. The user clicked on the Log in link located at the top right
corner of the homepage, as indicated in Figure 1 on the next page. I didnt see
anywhere else to register or log in, so I just assumed that if I clicked there itd give
me an option to sign up.

Grammarly Usability - 5

Figure 1. Grammarly home page with Log in link circled.

This was a fruitless effort; the login screen offered no ability to register (see Figure
2 below).

Figure 2. Login page. There is no registration option present.

I guess I have to find someplace else on the homepage to sign up, the subject
observed; he returned to the homepage and tried the Add to Chrome button.
Grammarly then tried to install an extension, but the subject was reminded that
Grammarly Usability - 6

the web-based interface was of interest, not the extension. He canceled the
extension installation, which led to a new screen (see Figure 3).

Figure 3. Screen that appeared after cancelling installation of an extension.

The subject noticed the small No, thanks link at the bottom of the screen
(circled in Figure 3). He clicked this and was taken to a registration screen (see
Figure 4 below). I dont understand why I would want to install an extension that I
know nothing about. Why couldnt I just sign up first? he asked.

Grammarly Usability - 7

Figure 4. Grammarly signup screen.

The subject was able to very quickly and easily register and log in, which brought
him to the main interface (see Figure 5). The subject expressed confidence that he
would be able to sign in in the future: Now that my email is registered, Im pretty
sure Ill be able to just use that Log in link that I mistakenly clicked at first to get
back into this later.

Figure 5. Main Grammarly interface.

Grammarly Usability - 8

Uploading and Checking a Document

The subject was asked to upload a document to Grammarly to see what kinds of
suggestions it might make. He very quickly noticed the New icon with the
Upload link directly below it (see the middle of Figure 5). After choosing a paper
from a previous class, the paper had been uploaded and analyzed (see Figure 6
below).
Once the process was completed, the subject seemed somewhat unsure of how to
proceed. I guess I should click on the issues, he noted, referring to the bottom
right indicator that said there were 6 critical issues and 28 advanced issues
(see bottom-right of Figure 6).
Unfortunately, clicking on this link simply took the user to an upsell screen. It
seems like every time I try to do something with this system, it wants to sell me a
service, the user observed, looking unhappy. So far I dont think that this has
given me much value at all.
The user was asked to return to the document and promoted to try again, so this
time he scrolled through the document and noticed several marginal notes that
seemed to be attached to certain parts of his writing; see Figure 8.

Figure 6. Uploaded paper in the Grammarly interface.

Grammarly Usability - 9

Figure 7. Upsell encountered during the document check.

Grammarly Usability - 10

Figure 8. Margin note appearing in the Grammarly interface.

The user expressed satisfaction with this interface. This seems intuitive to me,
he noted. I can see here that the program has highlighted stuff that it thinks
might be problematic and then when I click on it, it explains why.

Evaluating Results & Changing a Document

The user spent several minutes scrolling through Grammarlys suggestions. Most
of these seem pretty minor but I might as well fix them. He quickly figured out
that clicking Ignore would dismiss a suggestion, and that he could use the inline
text editor to fix other issues. This is all very straightforward. I can make changes
and it looks like Grammarly is checking my work as I go along.

Producing a Completed Document

To conclude the test, the user was asked to produce a completed document that
he could save as a Microsoft Word document. Unsure of exactly what to do, the
user hovered over the various icons in the black bar along the left side of the
screen, which produced menu fly-outs. The down arrow in the box gives me an
option to download, which is what I think I want. He clicked this option and the
paper was immediately downloaded.
The subject was asked to open the document to make sure that it was
satisfactory. He seemed pleased: All of my formatting is still here, even though
from the look of the Grammarly editor I was a little worried that it might erase all
of my italics and indentations and stuff.

Grammarly Usability - 11

System Usability Scale

Having completed the tasks asked of him, the subject was asked to complete
Brookes System Usability Scale. Results appear below in Table 1.
Strongly
Strongly
Disagree
Agree
1
2 3 4
5
Score
I think that I would like to use this
X
2
system frequently.
I found the system unnecessarily
X
3
complex.
I thought the system was easy to
X
3
use.
I think that I would need the
support of a technical person to be
X
4
able to use this system.
I found the various functions in
X
1
this system were well integrated.
I thought there was too much
X
3
inconsistency in this system.
I would imagine that most people
would learn to use this system
X
3
very quickly.
I found the system very
X
3
cumbersome to use.
I felt very confident using the
X
3
system.
I needed to learn a lot of things
before I could get going with this
X
2
system.
27 x 2.5 =
67.5
Table 1. System Usability Scale results.

Conclusions
Aside from some initial difficulties relating to registration and accessing the
Grammarly system online, this usability test suggested that Grammarlys webbased system is a fairly intuitive application, though it is not without its
drawbacks. Its SUS score of 67.5 out of 100 suggests that it could use some
tweaks.

Advantages
The usability tester praised Grammarlys intuitive and quick upload and download
features, noting that the controls were simple and straightforward, and that
Grammarly preserved the formatting that was already present in his Microsoft
Grammarly Usability - 12

Word document. The user also expressed satisfaction with the in-line marginal
comments presented after Grammarly analyzed his document; he noted that
there was no confusion as to what Grammarly was suggesting, and he felt that
the depth and clarity of the explanations provided by Grammarly were particularly
helpful.

Disadvantages
The tester noted that he felt like the free version of Grammarly was probably
extremely watered down, but he also noted that the price for the full version of
the software seemed quite high and that he wouldnt be willing to pay for it
without a better idea of its capabilities. He also noted that he was not initially sure
if Grammarly would preserve the formatting of his document, since there was no
indication of whether this was the case and because the inline text editor
appeared to have stripped out all formatting. The tester was also turned off by the
constant upsells, nothing that he wasnt sure if or when he might encounter yet
another advertisement. Finally, the tester stated that he felt extremely frustrated
with the registration process; he said that it should not have been so difficult to
sign up, and that Grammarlys constant plugging of its browser add-on was
annoying and difficult to bypass.

Recommendation
The primary usability problems that Grammarly displays seem to mostly pertain
to registration and login; access issues aside, it is straightforward and seems to
offer a lot of promise. That said, because of its prohibitive price tag for individuals,
it is probably most useful as a tool that is licensed by schools, colleges, and
universities on behalf of its faculty, staff, and students. It could presumably be
integrated into school technology packages and library resources, and might also
help professors and teachers detect plagiarism. For individual use, it is probably
not the best or most ideal choice simply because of its price, though those with
deep pockets or for whom writing is a vocation may find that its benefits outweigh
the cost.

Grammarly Usability - 13

References
Brooke, J. (1986). SUS: A quick and dirty usability scale. Retrieved April 23, 2016,
from http://cui.unige.ch/isi/icle-wiki/_media/ipm:test-suschapt.pdf
Grammarly. (2016a). FAQ. Retrieved April 23, 2016, from Grammarly:
https://www.grammarly.com/faq
Grammarly. (2016b). Home. Retrieved April 23, 2016, from Grammarly:
http://www.grammarly.com

Grammarly Usability - 14