Drag-and-Type: A New Method for

Typing with Virtual Keyboards on

Small Touchscreens

A seminar report
submitted in partial fulfilment of
the requirements for the award of the degree of
BACHELOR OF TECHNOLOGY
in
Computer Science & Engineering
from

University Of Calicut

Submitted By
NASEEB RAHMAN U.P (CEAMECS120)

MEA Engineering College

Department of Computer Science and Engineering
Vengoor P.O, Perinthalmanna, Malappuram, Kerala-679325
APRIL 2016

Department of Computer Science and Engineering

MEA ENGINEERING COLLEGE

PERINTHALMANNA-679325

Certificate
This is to certify that the seminar report entitled Drag-and-Type: A New Method
for Typing with Virtual Keyboards on Small Touchscreens is a bonafide
record of the work done by NASEEB RAHMAN U.P (CEAMECS120) under
our supervision and guidance. The report has been submitted in partial fulfilment of
the requirement for award of the Degree of Bachelor of Technology in Computer
Science & Engineering from the Univeristy of Calicut for the year 2016.

Mr. Sreeram. S
Head Of Department
Dept.of Computer Science and Engineering
MEA Engineering College

Mr.Ismail P.K
Seminar Guide
Assistant Professor
Dept.of Computer Science & Engineering
MEA Engineering College

Acknowledgements
First of all, I praise THE GOD. He showed me the right path and gave us immense
in all my efforts in completing this seminar. The lord is the one and only one who guided
me for seminar work to be a successful and preparing this report.
I grab this opportunity to express my sincere thanks to Dr. Rajin M Linus,
my respected principal who gave the best facilities and atmosphere for the seminar work
completion.I would like to thank Mr. Sreeram S. , Head of the department,
Computer Science and Engineering for providing permission and facilities to conduct the seminar in a systematic way. We are extremely greateful to our seminar guide
Mr.Ismail P.K, Asst. Professor in Computer Science and Engineering for the
inspiring and sincere guidance throughout the seminar.
My sincere thanks to seminar co-ordinators Mr. Harish Binu K.P. , Mr.
Bineesh V. , Asst. Professors in Computer Science and Engineering for their
wholehearted moral support in completion of this seminar.
Last but not least, I would like to thank all the teaching and non-teaching staff
and my friends who have helped me in every possible way in the completion of my
seminar.

NASEEB RAHMAN U.P (CEAMECS120)

ii

Abstract
Small touchscreens are widely used in consumer electronics, such as smartphones and mobile electronic devices. However, typing on the small touchscreen is still
worth studying. In fact, smartphone users are experiencing difficulties and also many
errors in typing alphanumeric keys with their thumbs because a small virtual keyboard
even with the reduced set of touchable keys can only provide tiny size keys to the users.
This paper studies a new style of typing method called Drag-and-Type, which leverages
the dragging action instead of direct tapping on the touchscreen to ease more accurate
typing on the small virtual keyboard. Although the typing speed is controversial, the
consumers can choose this method when an accurate typing is more required, for example, for a password entry that is quite more sensitive to erroneous key inputs. In that
sense, the proposed method is further explored to the extension called Secure Dragand-Type for securing the password entry against shoulder-surfing and spyware attacks
under the Drag-and- Type paradigm. In the user study, it was found that the proposed
method could be used for secure and accurate password entry on the small touchscreen
regarding the security-sensitive consumer electronics applications.

Contents
Acknowledgements

ii

Abstract

iii

Contents

iv

List of Figures

vi

List of Tables

vii

List of Abbreviations

viii

1 INTRODUCTION
1.1 Virtual Keyboards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2 Drag-and-Type
2.1 Virtual Keyboards for Usability
2.2 Existing System Problems . . .
2.2.1 Visual echo problem . .
2.2.2 Shoulder Surfing Attack
2.2.3 Spyware Attack . . . . .
2.3 Virtual Keyboards for Security
2.4 System Overview . . . . . . . .
2.4.1 Drag-and-Type Methods
2.4.2 Drag-and-Tap . . . . . .
2.4.3 Drag-and-Drop . . . . .
2.4.4 Usability Evaluation . .
2.5 Secure Drag-and-Type Methods
2.5.1 Threat Model . . . . .
2.5.2 Basic Concept . . . . .
2.5.3 Input Interface . . . . .
2.5.4 Usability Evaluation . .
2.5.4.1 Design . . . .
2.5.4.2 Participants .
2.5.4.3 Procedure . . .
2.5.4.4 Results . . . .
2.5.5 Security Evaluation . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
iv

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

1
2
4
4
5
5
5
6
7
8
8
9
10
10
11
12
12
13
13
13
14
14
14
16

2.5.5.1
2.5.5.2
2.5.5.3

Shoulder-surfing Resilience . . . . . . . . . . . . . . . . . 16
Spyware Resilience . . . . . . . . . . . . . . . . . . . . . . 16
Comparison . . . . . . . . . . . . . . . . . . . . . . . . . 17

3 CONCLUSION

19

REFERENCES

20

List of Figures
1.1

QWERTY Keyboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.1

Visual echo problems. (a) An entered key and its visual echo are occluded
under the thumb. (b) Bigger echo of entered key b can be moreeasily
observed not only by the user but also by the adversaries . . . . . . . .
Shoulder surfing attack . . . . . . . . . . . . . . . . . . . . . . . . . . . .
(a) Drag-and-Tap (b) Drag-and-Drop . . . . . . . . . . . . . . . . . . . .
(a) Drag-and-Tap (b) Drag-and-Drop . . . . . . . . . . . . . . . . . . . .
. Prototype design of Secure DnT method. (a) Keyboard layout before
a user drags a pointer. (b) Blank keyboard layout when a user drags a
pointer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Average entry time of each method (regular keyboard Drag-and- Tap,
Drag-and-Drop) in the user experiment. . . . . . . . . . . . . . . . . . .

2.2
2.3
2.4
2.5

2.6

vi

. 5
. 6
. 9
. 10

. 12
. 15

List of Tables
2.1

Comparison Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

vii

List of Abbreviations
DnT

Drag And Type

DnD

Drag And Drop

PDA

Personal Digital assistants

URL

uniform resource locator

viii

CHAPTER 1
INTRODUCTION

A smart phones is now becoming a part of electronics consumers lives and

turns out to be one of the most popularly used consumer electronic devices. Its small
flat touchscreen enables those consumers to navigate various kinds of services and applications very easily, promptly, and intuitively with their fingers. The small touchscreen is
also changing the way of typing alpha- numeric characters on those devices. Without a
physical keyboard, todays smart phones popularly present virtual keyboards, aka software keyboards based on the high-resolution of small touch screens, e.g., 4.8 1280 * 720
pixels (306 ppi) and 3.5 640 * 960 pixels (326 ppi) in commodities. To input alphanumeric keys, for example, consumers may tap their fingers on the small virtual keyboard
through the small touchscreen but there exist at least two concerns that strongly motivate this study. Smart phones uses virtual keyboards for typing alphanumeric characters.
There are different types of virtual keyboards exist.
Password has been the most pervasive means for user authentication since the
advent of computers. Compared to its alternatives, such as biometrics and smart card
which are cumbersome to use and require the existence of an underlying infrastructure,
password is much easier and cheaper to create, update, and revoke. However, the use of
password has intrinsic problems. Among them, secret leakage is one of the most common
security threats, in which an adversary steals the password by capturing (e.g.by shouldersurfing or key logging) and analyzing users input during an authentication session.
Traditional password system ask a user to directly input his entire plain text password
recalled from the users memory so that an observation of a single authentication session
is sufficient to capture the password.in order to prevent secret leakage during password
1

Chapter 1. INTRODUCTION

entry, a user needs to input the password directly, which imposes an extra burden on
the user.

1.1

Virtual Keyboards
Pervasive devices have come to the forefront in computer technology. Small

hand-held devices such as personal digital assistants (PDAs), pagers, and mobile phones,
as well as larger scale devices such as tablet computers and electronic whiteboards, now
play an increasingly more central role in human information interaction. This general
trend is rapidly freeing us from the confines of our laptop or desktop computers and
leading us to a future of pervasive computing, obstacles stand in the way of developing
efficient applications on these devices. An obvious made many common applications,
such as chat,e-mail,or even entering a URL very difficult. Text entry is also problematic
for PDAs and other hand-held devices. Currently text input on these device can be
achieved through reduced physical keyboards, handwriting recognition, and virtual keyboards, but each has critical usability shortcomings. There are two ways of reduce the
size of physical keyboards. One is the shrink the size of each key. This is commonly seen
in electronic dictionaries. Typing on these keyboard is slow and difficult due to their
reduced size. The other method is to use the number of pads in telephones, where by
each number corresponds to multiple letters. The ambiguity of multiple possible letter
is commonly resolved by the number of consecutive taps, or by lexical models. Reducing
error rate has the major goal in handwriting speed limit. It is very difficult to write
legibly has high speed.
First, the smartphone users are frequently experiencing difficulties and also
many errors in typing alphanumeric keys with their thick thumbs because a small virtual
keyboard even with the reduced set of touchable keys can only provide tiny size keys to
the users . Although the higher resolution of touchscreens can facilitate much smaller
keys for constructing a full size keyboard layout, users may prefer a larger key so as to
type characters with thumbs more easily. Unfortunately, such a larger key may only
allow a partial keyboard layout having the reduced set of keys on the small touchscreen,
e.g., separate layouts for alphabets and numeric (and/or special) characters, and pop-up
keys for rendering more characters on the keys at best. Note that the partial keyboard
layout requires a number of switches between distinct layouts. In order to avoid that a
Department of Computer Science & Engineering

MEA Engineering College

Chapter 2. DRAG-AND-TYPE

visual echo can be used as shown in describes about an entered key and its visual echo
occluded under the thumb. A visual echo, the most widely used response method on
the virtual keyboard, can be occluded and hidden under the thick thumb with blunt
touch. This tendency could reduce the benefits from the recent and future advance in
the high-resolution touchscreens and hinder the consumers from being aware of the real
key entry and eventually correct key entry on the touchscreen.
The design of virtual keyboard plays letters and numbers on a touch sensitive
screen or surface. To input text, the user presses keys with a finger or stylus. Such a
keyboard can be scaled to fit computing devices with varying size, particularly small
hand-held devices. One central issue, however,is the layout of the keys in the keyboards.
Due to developers and users existing knowledge, the QWERTY layout used in most
physical keyboards today has the momentum to become the most likely choice. Infact,
some PDA products, have already used QWERTY as their virtual keyboard layout as
shown in figure 1.1.Most of the virtual keyboards provide visual echo technique for
recognizing correct key entry.

Figure 1.1: QWERTY Keyboard

Department of Computer Science & Engineering

MEA Engineering College

CHAPTER 2
Drag-and-Type

2.1

Virtual Keyboards for Usability

A virtual keyboard is commonly used to type characters into a touchscreen-

based electronic device. To enter a character, a user must tap a finger on the corresponding software key instead of pressing the hardware key. There have been various keyboard
designs regarding usability and security issues.A number of virtual keyboards with distinct layouts, such as OPTI, ATOMIK, Metropolis and FITALY have been proposed
by rethinking the standard QWERTY keyboard with regard to usability issues in mobile electronic devices that incorporate a small touchscreen. CATKey was developed to
provide customizable and adaptable functions using QWERTY arrangement. However,
due to the small size of the keys on a small touchscreen, it was hard for users to type
characters correctly on those virtual keyboards.cope with this problem, there have been
various attempts.
One is to overlay larger split-keys in a pie menu represented on a virtual keyboard but, on the other hand, it causes two layered typing, which may be undesirable
for the fast and/or consecutive typing of characters. M. Klima et al.proposed a vector
keyboard that is composed of three major clusters containing 9 characters, respectively.
A user can type characters with their thumbs by drawing a vector from one of the clusters. There still remains a problem that character keys can be visually occluded. S. Zhai
et al. proposed SHARK (Shorthand Aided Rapid Keyboarding) in which an ATOMIK
keyboard is used to type characters by shorthand symbols, the symbols drawn with
a stylus on the touchscreen. Although there have been a number of virtual keyboard
4

Chapter 2. DRAG-AND-TYPE

designs not limited to the above, it is interesting that the most widely used virtual keyboard is the QWERTY virtual keyboard but with the reduced set of touchable keys on
the commodity devices.

2.2
2.2.1

Existing System Problems

Visual echo problem

Figure 2.1: Visual echo problems. (a) An entered key and its visual echo are occluded
under the thumb. (b) Bigger echo of entered key b can be moreeasily observed not only
by the user but also by the adversaries

2.2.2

Shoulder Surfing Attack

Current software interfaces for entering text on touch screen devices make

existing mechanisms such as keyboard typing or hand writing. These techniques are
poor for entering private text such as passwords since they allow observes to decipher
what has been typed simply by looking over the typists shoulder, an activity known as
shoulder surfing. Shoulder surfing using direct observation techniques such as looking
over someones shoulder, to get information. Shoulder surfing is an effective way to get
information in crowded places because its relatively easy to stand next to someone and
Department of Computer Science & Engineering

MEA Engineering College

Chapter 2. DRAG-AND-TYPE

watch as they fill out a form, enter a PIN number at an ATM machine, or calling card
at a public pay phone. Shoulder surfing can also be done long distance with the aid of
binocular or other vision enhancing devices. To prevent shoulder surfing, we need to
shield keypad from view by using our body by copying our hand.

Figure 2.2: Shoulder surfing attack

2.2.3

Spyware Attack
In general, spyware is any technology, such as tracking software that aids

in gathering information about a person without their knowledge. Spyware is any technology that aids in gathering information about a person or organization without their
knowledge. On the internet(where it is sometimes called a spybot or tracking software),
spyware is programming that is putting someones computer to secretly gather information and relay it to advertisers or other interested parties. Spyware can get in a
computer software virus or as the result of installing a new programme. Data collecting
Department of Computer Science & Engineering

MEA Engineering College

Chapter 2. DRAG-AND-TYPE

programmes that are installed with the users knowledge, properly speaking, spyware, if
the user fully understand what data is being collected and with whom it is being shared.
However, spyware is often installed without the userscomment, as a drive by download,
or as the result of clicking some option in a descriptive pop-up window. Software designed to serve advertising known as adware, can usually be thought of as spyware as
well because in variably includes components for tracking and reporting information.
However marketing firms object having their products called spyware. As a result,
McAfee the internet security company and others will refer to such applications as Potentially Unwanted Programs(PUP). The cookie is well known mechanism for storing
information about an internet user on their own computer. If a website stores information about you in a cookie that you dont know about, the cookie can be considered
a form of spyware. Spyware is part of an overall public concern about privacy on the
internet.

2.3

Virtual Keyboards for Security

The virtual keyboards have been studied regarding the security as well. To

defeat shoulder-surfing and spyware attacks on a secret key entry, such as a password
for authentication, researchers designed various kinds virtual keyboards. Tan et al.
proposed Spy-Resistant Keyboard that consists of 42 character tiles and 2 indicator tiles.
Each character tile is assigned three characters in random order. To type a password, a
user must set a shift state and move one of the indicators over the target character tile.
Bai et al. proposed PAS (Predicate-based Authentication Services) in which the user
can indirectly enter the password through the predicates generated by two secret values
and the CAPTCHA table. Zhao and Li developed S3PAS.(A Scalable Shoulder-Surfing
Resistant Textual-Graphical Password Authentication Scheme).
In this method, the user can enter a graphical password by constructing
pass-triangles based on the password and clicking the inside of the triangle. A variety of
authentication methods have also been developed to resist spyware that exploits touchbased screenshot captures. J. Lim proposed an anti-screen capture method based on
a partial image on each input key.In this method, the keypad is arranged at random
after every mouse click. Interestingly, each key alternates three images very quickly
on it: one blank and two partial images of the real key value. Agarwal et al. studied
Department of Computer Science & Engineering

MEA Engineering College

Chapter 2. DRAG-AND-TYPE

Dynamic Virtual Keyboard. The keyboard layout of this method is similar to the random
virtual keyboard that incorporates a dynamic random arrangement. In this method,
the user clicks a specific key called hide keysand then a real input key when all keys
are already hidden. The common limitation of these improved security measures is
undoubtedly their input performance. It takes long to enter passwords, about 49s for
8-text passwords in the Spy-Resistant Keyboard and about 33s for 6-text passwords in
the Dynamic Virtual Keyboard . Readers are referred to Table I in the end of this paper
for more comparisons.

2.4
2.4.1

System Overview
Drag-and-Type Methods
On the flat touchscreen, finger touch actions can be classified into two ac-

tions, i.e., tapping and dragging. The former is activated usually for a click event,
whereas the latter is done for scrolls and/or more functions, such as pointing and navigating. Multiple touch actions may involve simultaneous and/or consecutive actions of
tapping and dragging. These actions are considered for devising a new typing method.
First of all, it is pointed out that the dragging action enables more accurate targeting to
a tiny key on the virtual keyboard. Another point is that the users thumb typing on a
small touchscreen is done eventually as like the hunt and peck typing, aka two-fingered
typing, on a real keyboard. Thus, it is expected that if a small touchscreen represents a
full size keyboard on which tiny keys are located close to each other, then the dragging
actions of pointing would be quite familiar as well as more accurate than the direct tapping actions. Although the accuracy is obtained at the cost of dragging time, it would
be reasonable to think that less erroneous typing is also attractive in a large number of
applications. So two sorts of Drag-and-Type methods are devised in that sense.

The two concerns regarding accuracy and security motivated the authors
to develop a new style of typing method called Drag-and-Type, on the full layout of
the virtual keyboard presented on the small touchscreens. The Drag-and-Type method
leverages the dragging action instead of direct tapping on the touchscreen to ease more

Department of Computer Science & Engineering

MEA Engineering College

Chapter 2. DRAG-AND-TYPE

Figure 2.3: (a) Drag-and-Tap (b) Drag-and-Drop

accurate typing on the small virtual keyboard. In particular, two kinds of Drag-andType methods are proposed: Drag-and-Tap and Drag-and-Drop on the full layout of
the virtual keyboard. The Drag-and-Tap method works with separate tapping actions
on the full size keyboard, whereas the Drag-and-Drop method works with dragging actions only. Although the typing speed is controversial in both methods, the consumers
can choose the Drag-and-Type methods when an accurate typing is more required, for
example, for a password entry that is quite more sensitive to erroneous key inputs.

2.4.2

Drag-and-Tap
This method presumes a full layout of standard QWERTY keyboard in small

size and makes a user navigate the virtual keyboard by dragging one finger, e.g., the left
thumb, and type a highlighted (selected) character by tapping on any blank area with
another finger, e.g., the right thumb. The small red dot located among the keys, y, u,
and h, is used to navigate and select a target key while the larger grey circle below the
keyboard indicates an actual place for dragging. Deep grey keys are used for rendering
more functions onto the keyboard, such as tab, language, shift, backspace, space, and
enter.

Department of Computer Science & Engineering

MEA Engineering College

Chapter 2. DRAG-AND-TYPE

2.4.3

10

Drag-and-Drop
This method also presumes a full layout of standard QWERTY keyboard

in small size, and makes a user navigate the virtual keyboard by dragging one or two
fingers simultaneously, e.g., the left and right thumbs, and type a highlighted (selected)
character by releasing (dropping) the corresponding finger. Figure illustrates a prototype
layout of Drag-and-Drop keyboard using a split QWERTY layout for two fingers. The
small red and blue dots are used to navigate and select target keys, respectively, while
larger grey circles indicates a place for dragging. Deep grey keys are also split for
rendering more functions onto the keyboard. Figure. is a snapshot of the Drag-andDrop method in use. Note that the Drag-and-Drop method can be used in one hand.

Figure 2.4: (a) Drag-and-Tap (b) Drag-and-Drop

2.4.4

Usability Evaluation

Prototype systems of the Drag-and-Type methods were implemented on the smartphone,

as illustrated in Figure and a user study was conducted for evaluating the usability of
each method. In the user experiment, the split keyboard layout is used in Drag-andDrop. Drag-and-Type methods were compared to the regular virtual keyboard with
respect to the speed and accuracy for typing characters.
1) Design.The user experiment was designed as a within group study using 2 Repeated

Department of Computer Science & Engineering

MEA Engineering College

Chapter 2. DRAG-AND-TYPE

11

Measures-ANOVA. In the user study, first independent variable is character type (alphabets, decimals). Second independent variable is virtual keyboard (regular keyboard,
Drag-and-Tap, and Drag-andDrop). The participants conducted one combination of independent variables randomly to reduce learning effects for character and method type.
To evaluate the performance of each typing method, the entry time and error rates were
measured in the evaluation session.
2) Participants.12 participants (9 males, 3 females) with academic education were recruited. Their average age was 26.9 and the average period of using smartphone (cellphone) was 2.3 (10.9). The participants were comprised of 2 lefthanders and 10 righthanders. All of them had normal eyesight and experience of using regular virtual keyboard. The participants received a small gratuity for the user experiment.
3) Procedure.The participants conducted three methods in the within group study. The
order of method and character type was counterbalanced (3! * 2! = 12). They received
an explanation about how to type the characters with each method. They were asked to
type alphabets in sequence, i.e., a to z, for 5 times, and decimals in sequence, i.e., 1 to 0,
for another 5 times, after training themselves up to twenty minutes. After finishing the
experiment of each method, they responded the questionnaire. Likert-type scales were
used for rating 1 (strongly disagree) to 5 (strongly agree).

2.5

Secure Drag-and-Type Methods

Most of applications and web services in the smartphone provide a regular

virtual keyboard, using QWERTY arrangement, when users enter even their secret characters. However, the regular virtual keyboard is not appropriate as a password input
method because it is possible for shoulder-surfing attackers and spyware to intercept
the users sensitive information from the mobile computing devices. To cope with this
problem, some applications offer their own secure virtual keyboards and a number of
authentication methods have been also proposed. However, those authentication methods cant properly defend shoulder-surfing and spyware attacks at the same time. The
extended method of Dragand-Type, called Secure DnT, is designed to be secure against
those attacks in the smartphone.

Department of Computer Science & Engineering

MEA Engineering College

Chapter 2. DRAG-AND-TYPE

2.5.1

12

Threat Model
In this paper, it is assumed that there are two kinds of adversaries observing

the entered secret characters in the mobile computing devices. First adversary is a
human shoulder-surfing attacker, trying to look over someones typing. Second adversary
is a touch-based spyware attack that gathers consumers sensitive information without
their consent by exploiting the touch event information and screenshots.

Figure 2.5: . Prototype design of Secure DnT method. (a) Keyboard layout before a
user drags a pointer. (b) Blank keyboard layout when a user drags a pointer

2.5.2

Basic Concept
The keyboard layout of Secure DnT is composed of alphanumeric characters

in random arrangement. Figure shows the prototype design of Secure DnT method.
The characters of all keys are hidden when a user begins to drag a pointer on the
touchscreen. The keyboard layout remains blank until a character key is entered. After
the character key is entered, the keyboard layout is rearranged in random sequences
and the hidden keys reappear. These mechanisms enableto prevent efficiently shoulder
surfing and spyware attack from stealing users secret characters. A usercan find out
Department of Computer Science & Engineering

MEA Engineering College

Chapter 2. DRAG-AND-TYPE

13

easily the location of own password, whereas it is hard for observers to identify it.
Although adversaries may guess the secret characters, they could not find out exactly
whole of them because of hiding promptly all keys in the keyboard layout.

2.5.3

Input Interface
Secure DnT uses an input interface of Drag-and-Drop that uses dragging

for navigation and releasing for typing. It differs from Drag-and-Drop in using without
visual echo. However, it is possible to enter the character accurately with visual selected
key echo and vibration feedback. A user has to verify the location of the target character
keys before touching on the touchscreen. When the touch event.

2.5.4

Usability Evaluation
Prototype applications (random keyboard and Secure DnT) were imple-

mented on the smartphone and a user study was conducted. In this study, the Secure
DnT method was compared to the regular and the random keyboards.

2.5.4.1

Design
The experiment was designed as a within group user study for evaluating

usability using 2*3 Repeated Measures-ANOVA. In the userstudy, first independent

variable is password type (system-chosen password, userchosen password). Second independent variable is password input method (regular keyboard, random keyboard, and
Secure DnT). The participants performed one combination of independent variables randomly to reduce learning effects for password type and password system. To evaluate
the performance of each typing method, the entry time and error rates were measured
in the evaluation session.

Department of Computer Science & Engineering

MEA Engineering College

Chapter 2. DRAG-AND-TYPE
2.5.4.2

14

Participants
New 18 participants (11 males, 7 females) with academic education were

recruited in the local university. Their average age was 27.7 years and the average
period of using smartphone (cellphone) was 2.6 (11.4) years. The participants were
comprised of 2 left-handers and 16 righthanders. All of them had normal eyesight and
had experience of using regular virtual keyboard. The participants received a small
gratuity in return for the user experiment.

2.5.4.3

Procedure
The participants performed three methods in random sequences. The order

of method and character type was counterbalanced based on a Latin square design.
They received an explanation about the instruction of each method and were allowed
training time for entering abcd1234 three times, respectively. In the evaluation test,
participants were asked to enter two types of passwords twice, one for practice and the
other for test. The participants had system-chosen passwords and user-chosen passwords
that consist of 8 alphanumeric characters. System-chosen passwords are generated by
software program. User-chosen passwords are made by the users avoiding very simple
passwords, e. g., qwer0987. It is familiar to users, but it is hard for attackers to
guess. They memorized two passwords before entering the passwords. After finishing
the experiment of each method, they responded the questionnaire.

2.5.4.4

Results
The password entry time for each method was measured from the begin-

ning of the application execution to the last release of the pressed key. The fastest
password input method was regular keyboard with user-chosen passwords (mean: 5.913,
sd: 0.994). Regular keyboard with system-chosen passwords (mean: 6.138, sd: 1.061),
random keyboard with system-chosen passwords (mean: 20.361, sd: 2.682), random
keyboard withuser-chosen passwords (mean: 20.729, sd: 2.846), Secure DnT with userchosen passwords (mean: 21.940, sd: 2.106), Secure DnT with system-chosen passwords
(mean: 22.435, sd: 2.661) followed it. Figure shows the average entry time for each
method. In 2*3 (password type password input system) Repeated Measures-ANOVA,
Department of Computer Science & Engineering

MEA Engineering College

Chapter 2. DRAG-AND-TYPE

15

there was a significant main effect for password input system (F(2, 34) = 959.466, p
0.001). However, there was no significant main effect for password type (F(1, 17) =
0.129, n.s.(p = 0.724)). The interaction effect of password type and password input
system was not significant (F(2, 34) = 0.706, n.s.(p = 0.501)). Regarding these results,
Secure DnT was slower than the regular keyboard and random keyboard. It was reasonable, considering the security level of each method (more detailed in the security
evaluation). In the questionnaire, the participants evaluated regular keyboard (mean:
4.39, sd: 0.698) more fast to use than random keyboard (mean: 1.67, sd: 0.686) and
Secure DnT (mean: 1.78, sd: 0.808).

Figure 2.6: Average entry time of each method (regular keyboard Drag-and- Tap,
Drag-and-Drop) in the user experiment.

The failed sessions and backspace counts for each method were measured
in the evaluation session. There was no failed session in the regular keyboard and
Secure DnT. However, for the random keyboard, two participants succeeded in second
trial (5.6backspace was one with user-chosen password in the random keyboard and
Secure DnT, respectively. Thus, there was no significant main effect for password input
methods.

Department of Computer Science & Engineering

MEA Engineering College

Chapter 2. DRAG-AND-TYPE

2.5.5
2.5.5.1

16

Security Evaluation
Shoulder-surfing Resilience
Participants were asked to mount a shoulder surfing attack by observing

the users password entry on the smartphone. The attack experiment was conducted
with 10 participants (7 males, 3 females) whose average of age was 27.2 years and had
joined the usability experiment before. All of the participants had normal eyesight.
They were given 5 video records for authentication sessions that an operator enters
the 8-text passwords with the regular keyboard (no echo mode and visual echo mode),
random keyboard and Secure DnT method, respectively. Each video was adjusted the
entry time of usability results and recorded with the digital camcorder. The participants
planned their own strategy for shoulder-surfing attacks before starting the experiment.
The 23-inch computer monitor (1920 * 200 pixels) was used for playing the recorded
videos. The size of played smartphone was set similar to real smartphone size. The
entered passwords were shown as in the shape of asterisk. So, the participants could
observe just keyboard layout for each method.
In the attack experiment, the results were surprising. The highest success
rate of guessing the passwords is regular keyboard with echo (86.8 percentage). Regular
keyboard without echo (73.5 percentage), Random keyboard (12.3), and Secure DnT
(2.8 percentage) followed it. . In case of regular keyboard with echo, the success rate
of identifying 8-text passwords was 32 and missed one text was 42 percentage. Regular
keyboard without echo was harder than regular keyboard with echo to find out the
passwords due to the hidden keys under the thumb. On the other hand, the failed rate
of identifying all 8-text passwords with Secure DnT was 82 percentage. The participants
had difficulty in identifying the passwords in the random keyboard layout both random
keyboard and Secure DnT. Random keyboard is more vulnerable than Secure DnT,
because random keyboard can give some hints about moving direction of a users finger
percentage.

2.5.5.2

Spyware Resilience
An attacker who is familiar with the regular keyboard (with visual echo),

random keyboard, and Secure DnT simulated a spyware attack, recording the data
Department of Computer Science & Engineering

MEA Engineering College

Chapter 2. DRAG-AND-TYPE

17

of touch events and capturing screenshots for each method. The attacker implemented
malicious applications that have two additional functions, gathering coordinates of touch
events and screenshots when a user touches up on the smartphone. Also, the pointer
location option in the developer options of system settings was checked to present visually
current touch actions.
The results of the spyware attack were impressive. The attacker tried to
find out the entered key by exploiting both screenshots and touch coordinates in the
keyboard layout. The attacker could identify the typed character in the regular keyboard
and random keyboard. It is easy to distinguish the pressed key both methods. In case
of the regular keyboard, it uses QWERTY arrangement and appears certain location
in the application each time. Thus, only one screenshot was needed for analyzing the
regular keyboard to get information for keyboard location. On the other hand, multiple
screenshots were needed for analyzing random keyboard. Figure illustrate one of the
screenshots together with touched area of each method, respectively. It is possible for
the regular keyboard to find the typed character using only information of touch events
due to the fixed keyboard layout. Furthermore, the attacker could just identify the
entered key using only screenshots in the regular keyboard and random keyboard.
However, it was unsuccessful to identify the typed character in Secure DnT.
That is why it hides all keys in the keyboard layout when a user touches down on
the smartphone, as illustrated in Figure Although attackers may gain the screenshots
and coordinates of touch events for Secure DnT, they cant find out the character.
In the questionnaire regarding security for shoulder-surfing and spyware attacks, the
participants rated regular keyboard (mean: 1.39, sd: 0.608) less secure than random
keyboard (mean: 2.89, sd: 0.832) and Secure DnT (mean: 4.67, sd: 0.594).

2.5.5.3

Comparison
Secure DnT was compared with other authentication methods regarding its

usability and security. TABLE I summarizes the comparison results of each authentication method. The fastest is the regular keyboard (5.91s 6.14s), but it is vulnerable to
shoulder-surfing and spyware attacks. The random keyboard is a little faster than the
Secure DnT, but it has low security for spyware attack. The security of Dynamic Virtual

Department of Computer Science & Engineering

MEA Engineering College

Chapter 2. DRAG-AND-TYPE

18

Keyboard is similar to Secure DnT method, but its entry time (about 5.48s for one-text
password) is about twice as long as Secure DnT (2.74s - 2.81s for one-text password).
Method

Password Length

Entry Time (s)

Shoulder-surfing

Spyware

Regular Virtual Keyboard

Random Virtual Keyboard
PAS
S3PAS
Dynamic Virtual Keyboard
Spy-resistant Keyboard
Secure Drag-and-Type

8
8
Two secret strings
3-5
6
8
8

5.91-6.14
20.36-20.73
55.53
71.66
32.87
49
21.94-22.44

Weak
Moderate
Strong
Strong
Strong
Strong
Strong

Weak
Weak
Moderate
Moderate
Strong
Strong
Strong

Table 2.1: Comparison Table

Similarly to Dynamic Virtual Keyboard, Spy-resistant Keyboard is resilient

to both attacks and it has also long entry time (about 49s). The user experiment result of
CHC [18] was referred for comparing the entry time of S3PAS method. Its execution time
is slowest (71.66s) among other authentication methods and PAS followed it (55.53s).
Moreover, S3PAS and PAS are vulnerable to intercept attacks analyzing the multiple
authentication sessions.

Department of Computer Science & Engineering

MEA Engineering College

CHAPTER 3
CONCLUSION

In this paper, the new Drag-and-Type method (Drag-andTap and Drag-andDrop) and itsextension called Secure DnT were proposed. The Drag-and-Type was a
novel typing method based on the dragging actions on a small flat touchscreen. The
prominent feature of the Drag-and-Type method was accuracy. The consumers are able
to type more accurately but more slowlyon the full-size virtual keyboards than on the
existing virtual keyboard. The Dragand-Type method was extended to its secure virtual
keyboard version called Secure DnT to deal with shouldersurfing and spyware attacks.
The Secure DnT method was more efficient and/or more secure compared to the related
authentication methods. The user studies and the attack experiments conducted in this
paper confirm that it would be promising to adapt the Drag-and-Type method when a
more accurate typing is preferred, and the Secure DnT method when a more accurateand
securing typing is required on the consumer electronic devices. Specifically, a secure (and
accurate) password entry can be achieved by the Secure DnT method. The limitation
is that the Secure DnT can only resist a touch-based spyware attack. In the future
study, a new method will be explored to resist an advanced spyware attack based on
recording the whole interactions between consumer and electronic device through the
small high-resolution touchscreens.

19

REFERENCES

[1] Y. Yoon and G. Lee, Square: 3*3 keypad mapped to geometric elements of a square,
IEEE Trans. on Consumer Electronics, vol. 54, pp. 1274-1280, Aug. 2008.
[2] V. Balakrishnan and P. Yeow, A studyof the effect of thumb sizes on mobile phone
texting satisfaction, Journal of Usability Studies, vol. 3, pp. 118-128, May 2008.
[3] L. Cai and H. Chen, TouchLogger: Inferring keystrokes on touch screen from smartphone motion, in Proc. USENIX Conference on Hot Topics in Security, San Francisco,
USA, Aug. 2011.
[4] T. Kwon, S. Na, and S. Park, Drag-and-Type: A new method for typing with
virtual keyboards on small touchscreens, in Proc. IEEE International Conference on
Consumer Electronics, Las Vegas, USA, pp. 460-461, Jan. 2013.
[5] I. S. Mackenzie and S. X. Zhang, The design and evaluation of a highperformance
soft keyboard, in Proc. SIGCHI Conference on Human Factors in Computing Systems,
Pittsburgh, USA, ACM press, pp. 25-31, May 1999.
[6] S. Zhai, M. Hunter, and B. A. Smith, Performance optimization of virtual keyboards, Human-Computer Interaction, vol. 17, 2002. [7] J. D. Ichbian, Method
for designing an ergonomic one-finger keyboard and apparatus therefor, In US patent
5487616, 1996.
[8] K. Go and Y. Endo, CATKey: Customizable and adaptable touchscreen keyboard
with bubble cursor-like visual feedback, in Proc. IFIP TC 13 International Conference
on Human-Computer Interaction, Rio de Janeiro, Brazil, LNCS 4662, pp. 493-496, Sept.
2007.
[9] K. Go and L. Tsurumi, Arranging touch screen software keyboard split- keys based
on contact surface, in Proc. CHI10 Extended Abstracts on Human Factors in Computing Systems, Atlanta, USA, ACM press, Apr. 2010.
[10] M. Klima and V. Slovacek, Vector keyboard for touch screen devices, in Proc.
International Conference on Ergonomics and Health Aspects of Work with Computers,
20

References

21

San Diego, USA, LNCS 5624, pp. 250-256, July 2009.

[11] S. Zhai and P. O. Kristensson, Shorthand writing on stylus keyboard, in Proc.
SIGCHI Conference on Human Factors 97-104, Apr. 2003.
[12] D. S. Tan, P. Keyani, and M. Czerwinski, Spy-Resistant Keyboard: More secure
password entry on public touch screen displays, in Proc. Australia Conference on
Computer-Human Interaction, Canberra, Australia, Nov. 2005.
[13] X. Bai, W. Gu, S. Chellappan, X. Wang, D. Xuan, and B. Ma, PAS: Predicatebased authentication services against powerful passive adversaries, in Proc. IEEE Annual Computer Security Applications Conference, Anaheim, USA, pp. 433-442, Dec.
2008.
[14] H. Zhao and X. Li, S3PAS: A scalable shoulder-surfing resistant textual-graphical
password authentication scheme, in Proc. IEEE International Conference on Advanced
Information Networking and Applications Workshops, Niagara Falls, USA, vol. 2, pp.
467-472, May 2007.

Department of Computer Science & Engineering

MEA Engineering College