Usability Study

Command Line Editor Comparison

1|Page

Introduction
When entering college in a Computer Science or Computer Engineering
department, students are often required to log in to a Linux or Unix system to
complete programming assignments. Doing this requires learning how to use
command line text editors. This usability study tests the learnability and
functionality of Vim and Emacs through the completion of 13 tasks which are
common for students in these fields.

Methods
Tasks:
1. Find documentation
2. Edit the file
3. Save and quit
4. Quit and discard changes
5. Undo/Redo a change
6. Copy and paste
7. Cut and paste
8. Change tab width and tab to spaces
9. Match brackets/parentheses
10. Change the color of keywords
11. Save file under a different name
12. Evaluate documentation
13. Write a Hello World program
Test Environment:
I set up an environment typical for doing homework in college to
conduct this study. I used my laptop at the dining room table in my
apartment, so that I was on a machine that I am familiar with. There
2|Page

were a few distractions throughout the test, which is in line with a

college study environment. I used Putty Secure Shell to log in to the
University of Idahos Linux system. Inside of the Secure Shell, I used
Vim and Emacs to write C programs.
Evaluation Criteria:
Each task was scored on a 5 point Likert scale. The values correspond to the
following statements: 5. Strongly Agree 4. Somewhat Agree 3. Agree 2. Somewhat
Disagree 1. Strongly Disagree.
The criteria statements used for evaluation varied between tasks, but they
include:
My first Google search yielded documentation that looked promising
The command was in the original documentation
The command was found quickly and completed without hassle
I used only one documentation page to complete the entire task list
Documentation was quick to find and commands/explanations were understood well
enough to complete the tasks
I learned the vital commands throughout testing and remembered them well enough
to complete the task

Results
The evaluation results of each task is detailed below.
Task 1: Find documentation
Scores

My first Google search yielded

documentation that looked promising
5
4

Vim
Emacs
Vim Notes: I searched "vim commands" and clicked the first link. This link listed commands in

categories that lined up with items in my task list, so I used this as my base documentation. [1]
3|Page

Emacs Notes: I searched "emacs commands," just like I did for vim, but a useful documentation page
was harder to find. [2]

Task 2: Edit the file

Scores

The command was in the

original documentation

The command was found

quickly and completed
without hassle
4
5

Vim
5
Emacs
5
Vim Notes: There are a couple of different commands that can be used to begin editing a file, but they
are both simple and easy to remember. After editing the file, users have to press the escape key to
return to command mode. This was not explicitly stated, but it is common to use the escape key to
return to previous modes, so it was an intuitive command.

Emacs Notes: No command is required to begin typing in the file. Anything typed without a control
character first is automatically added to the file.

Task 3: Save and quit

Scores

The command was in the

original documentation

The command was found

quickly and completed
without hassle
5
3

Vim
5
Emacs
5
Vim Notes: Since vim is not a control key based editor, it took a slight adjustment to use a colon to
begin a command, but the command itself is intuitive and simple.

Emacs Notes: Saving required entering a sequence of two commands. At first, I did not realize that it
required both, but thought either command would save the document. Once the command was
entered, a prompt came up at the bottom of the screen asking if I wished to save the file and showed
the name it was to be saved as and a list of options to respond. The responses 'y' and 'n' were assumed
to be yes or no, but the rest were unclear. After typing 'y' the document saved and closed as expected.

Task 4: Quit and discard changes

Scores

Vim
Emacs
Vim Notes: None.

The command was in the

original documentation
5
5

The command was found

quickly and completed
without hassle
5
5

4|Page

Emacs Notes: The command was very similar to save and quit, but there was a warning that there was
unsaved changes. This seemed to be a good safety measure to ensure the user was notified of unsaved
changes.

Task 5: Undo/Redo a change

Scores

The command was in the

original documentation

The command was found

quickly and completed
without hassle
3
4

Vim
5
Emacs
5
Vim Notes: The undo command removed everything that had been typed since the insert or append
mode was entered. Although this is a predictable behavior, I would have expected a word at a time to
be deleted or brought back.

Emacs Notes: The command is simple and undoes typing phrase by phrase, similar to Microsoft Word.
Once all typing has been undone, it begins to redo the typing. There is not a separate redo command as
far as I could tell.

Task 6: Copy and paste

Scores

The command was in the

original documentation

The command was found

quickly and completed
without hassle
1
3

Vim
4
Emacs
4
Vim Notes: The terminology and instructions for a simple copy/paste were not intuitive or clear. The
user has to enter "Visual Mode" and then select text using the arrow keys on the keyboard. Selecting
text with the mouse did not seem to be an option. The terms "copy" and "paste" were also not used.
The commands were instead called "yank" and "put. If the documentation was not well organized, I am
not sure I would have found the "yank" and "put" commands without a separate google search inquiring
about copy/paste.

Emacs Notes: The terminology for copy/paste for emacs also included yank, but the documentation
was more straightforward. To select the text, a mark has to be set at the beginning of the text to be
copied. Then, the cursor needs to be moved to the end of the text before the copy command is entered.
The "yank" is the command that pastes the text in a new location.

Task 7: Cut and paste

Scores

The command was in the

original documentation

Vim

The command was found

quickly and completed
without hassle
2
5|Page

Emacs
4
4
Vim Notes: The command for "cut" was called "delete." I interpreted delete as gone forever, however
the "delete" command actually saved the text to be pasted elsewhere. I did not realize this until I
conducted a google search about cutting text in vim and the "delete" command was discussed in more
detail. As with copy/paste, the "visual mode" had to be used to select text to be cut.

Emacs Notes: To cut and paste was very similar to copy and paste.
Task 8: Change tab width and tab to spaces
Scores

The command was in the

original documentation

The command was found

quickly and completed
without hassle
4
1

Vim
1
Emacs
1
Vim Notes: Although this process was not described in the original documentation I found, a simple
Google search quickly showed me how to change tabs to spaces and change the tab width.

Emacs Notes: Emacs does some auto-indentation when typing a C program. It does a pretty good job of
keeping text aligned properly, however, the tab key does not actually work and the auto-inserted tabs
are not converted to spaces. This creates some frustration and I am not sure how well it would work
with a language like Python, where indentation is key.

Task 9: Match brackets/parentheses

Scores

The command was in the

original documentation

The command was found

quickly and completed
without hassle
5
3

Vim
N/A
Emacs
N/A
Vim Notes: Matched brackets and parentheses were automatically highlighted when the cursor was
next to them.

Emacs Notes: The brackets and parentheses were matched as I typed them, but they were not
highlighted when the cursor was next to them.

Task 10: Change the color of keywords

Scores

The command was in the

original documentation

Vim
Emacs

1
1

The command was found

quickly and completed
without hassle
3
1

6|Page

Vim Notes: This was not a command listed in my original documentation, but a simple search allowed
me to change the color scheme of the editor. There did not appear to be a way to customize the
scheme, but there were enough schemes available to satisfy most users.

Emacs Notes: After reading through six different links from a Google search, I still could not figure out
how to change the colors in emacs. This seems like more work than it's worth to change the colors.

Task 11: Save file under a different name

Scores

The command was in the

original documentation

Vim
Emacs
Vim Notes: None.

5
5

The command was found

quickly and completed
without hassle
5
5

Emacs Notes: None.

Task 12: Evaluate documentation
Scores

I used only one

documentation page to
complete the entire task list

Documentation was quick to

find and
commands/explanations
were understood well
enough to complete the tasks
4
2

Vim
1
Emacs
1
Vim Notes: The copy/paste and cut/paste commands were not intuitive and I had to consult other

search results to complete these tasks. I also had to use other documentation to change tab settings and
colors. Despite having to find more documentation, it was easy to find the information necessary to
complete the tasks.

Emacs Notes: I often had to read through more than one link from a Google search to learn how to
complete a task.

Task 13: Write a Hello World program

Scores

I learned the vital commands throughout

testing and remembered them well enough
to complete the task
4
2

Vim
Emacs
Vim Notes: Vim is an insertion mode editor, so the user has to change modes to edit the document or
enter commands to manipulate the document. This idea is different from the visual, mouse oriented
7|Page

editor many people are used to. Due to this, there were a few times that I forgot to switch modes, but
being in the wrong mode never caused catastrophic results. With more use, I think that I would get used
to this concept and no longer have issues with it. I recalled the commands for most of the tasks
completed throughout the usability test, but had to address the documentation a couple of times
because of the initial confusion with copying and cutting text. Despite this, I was quickly able to write,
compile, and run a simple C program using vim as the editor.

Emacs Notes: In the time it took to get through the testing I forgot how to complete some of the most
basic tasks. Since many of the commands have two parts, I had a very hard time remembering them all. I
was also disappointed with the inability to change tab or color settings, both of which added stress to
the completion of a simple program.

Recommendation
The total scores, shown below, declare Vim as the recommended editor. These totals reflect
the ease of finding useful documentation and remembering commands to complete simple
tasks. I agree with this numerical result, as I had a much more difficult time writing a simple
program in Emacs following the testing and endured more stress trying to learn the commands.
The scores are not drastically different, however, and I think that with more time and focus,
Emacs could also make a good command line editor. There is just a slightly steeper learning
curve associated with it if users can tolerate the difficulty of changing color scheme or tab
settings.

Vim
Emacs
Total possible

Scores
86
77
115

References
[1] http://www.fprintf.net/vimCheatSheet.html
[2] http://www.cs.colostate.edu/helpdocs/emacs.html

8|Page