Central Taiwan University of Science and Technology

Graduate Institute of Healthcare Administration

Master Thesis

Adviser: Yung-Fu Chen, Ph. D

Microsoft Kinect
A Microsoft Kinect based virtual rehabilitation system

:
Graduate Student: Erdenetsogt Davaasambuu

2013

ii

iii

iv

Acknowledgements
Many thanks to my advisor, Prof. Yung-Fu Chen, for providing me the
opportunity and financial support to study in CTUST. I also would like to express
my appreciation to Dr. Chung-Liang Lai, Chief of Department of Physical
Medicine and Rehabilitation, Taichung Hospital, for helping and giving me
professional advices on this study. During my stay in Taiwan, I have made a lot
of Taiwanese friends, who are always with me when I was in difficult situations.
Without the assistance and participation of the staffs and patients of National
Taichung Hospital, my thesis cannot be finished in time. I deeply appreciate Dr.
Chin-Tun Hung, Chairman of the Department, and Dr. Chin-Chih Ho, and other
faculty members and staffs of the Department of Healthcare Administration.

ABSTRACT
Virtual reality technology is currently widely applied in physical
rehabilitation therapy. The ability to track joint positions for Microsoft Kinect
might be useful for rehabilitation, both in clinical setting and at home. Currently,
most systems developed for virtual rehabilitation and motor training require quite
complex and expensive hardware and can be used only in clinical settings. Now,
a low-cost rehabilitation game training system has been developed for patients
with movement disorders; it is suitable for home use under the distant supervision
of a therapist. This research explores the potential and the limitations of the
Kinect in the application of e-rehabilitation. Commercial Kinect games present
disadvantages that are mainly bounded to the fact that they are not developed
for rehabilitation purposes. We evaluated the tools that could be used to help
promote physical rehabilitation at home by reducing the frequency of hospital
visits, resulting in the reduction of healthcare cost. A prototypic system has been
developed for the evaluation of 5 different games, which are useful for erehabilitation. As we see from the result, it indicated a slightly positive outcome
for the patients after got involved the treatment. Also the questionnaire outcome
reveals that the designed exercises were perceived as effective and easy in
operation (p<0.05).

Keywords: Virtual reality; E-rehabilitation; Kinect, Balance training;

vi

TABLE OF CONTENTS
ACKNOWLEDGEMENTS...................

ABSTRACT............................................................

vi

TABLE OF CONTENTS...........................................................

vii

LIST OF TABLES.....................................................................

ix

LIST OF FIGURES...................................................................

CHAPTER ONE: INTRODUCTION......................................

1.1

Virtual Reality...........................................................

1.2

Virtual rehabilitation.................................................

1.3

Motivations and Objectives..............................................

1.4

Related works.....................................................

CHAPTER TWO: MICROSOFT KINECT

DEVELOPING ENVIRONMENT.

10

2.1

Microsoft Kinect........................................................

10

2.2

Design guidelines...........................................

11

2.3

Choosing Open Source Drivers and SDKs........................

12

2.4

Kinect API..........................................

14

2.5

Skeleton stream..........................

15

2.6

Math formulas implementation.....................

17

Smoothing skeleton data

19

Software development......................

20

2.7

Login and user management

vii

24

CHAPTER THREE: METHODOLOGY................................

26

3.1 Interface Design............................................................

26

3.2 Development of Kinect Virtual Rehabilitation System......

28

Ping Pong.....

28

Balance shift....

29

Window cleaner....

31

Step on It..................

32

Coin Collection....................

33

3.3

Experiment............................................

33

3.4

Subject and Statistic analysis................................

36

3.5

Rehabilitation Assessment.....................................

37

Biodex Balance Machine.....................................

39

Berg Balance Scale..................................................

40

Get up and Go test.......................

40

Kinect Virtual Rehabilitation system...............

41

CHAPTER FOUR: EXPERIMENTAL RESULTS...........

43

CHAPTER FIVE: DISCUSSION.AND CONCLUSIONS.....

46

REFERENCES...........................................................................

50

APPENDICES............................................................................

56

Appendix A: Form of Initial assessment .......

56

Appendix B: Form of Final assessment .......

57

viii

LIST OF TABLES
Table 2-1

Comparisons of Microsoft Kinect SDK and OpenNI S..13

Table 2-2

Kinect programming smoothing parameters..

Table 3-1

Demographic information of the recruited patients...

Table 4-1

Comparisons of balance ability for patients before

20
37

ddsfsdand after treatment with Kinect Virtual Rehabilitation

dfsdfc system.

43

Assessment of user attitude.....................................

45

Table 4-2

ix

LIST OF FIGURES
Figure 1-1

Benefit/challenges in VR rehabilitation application ...

Figure 2-1

Kinect components..........................

11

Figure 2-2

Kinect SDK and OpenNI skeleton joints ....

14

Figure 2-3

Kinect application interface ........

15

Figure 2-4

SDK skeleton detection processing ......

16

Figure 2-5

Pytagorean Theorem .........

17

Figure 2-6

Law of Cosines .........

18

Figure 2-7

Scale function.................

18

Figure 2-8

System Architecture Design..............

22

Figure 2-9

Activity Diagram Balance shift..............

23

Figure 2-10

System login page.................

24

Figure 2-11

User creation window..............

25

Figure 3-1

Carousel styled system main menu ........

26

Figure 3-2

Metro styled menu.......

27

Figure 3-3

Standard cursor and feedback graphics........

28

Figure 3-4

Ping Pong.....................................

29

Figure 3-5

Game for training balance........

30

Figure 3-6

Window Cleaner..................................

31

Figure 3-7

Step on it game....

32

Figure 3-8

Coin collection.................................

33

Figure 3-9

Human center of gravity and line of gravity............

35

Figure 3-10

Balance shift.............

36

Figure 3-11

Experimental procedure...............

38

Figure 3-12

Biodex balance machine..........

39

Figure 3-13

Photograph of a patient testing the system.......

41

xi

CHAPTER ONE
INTRODUCTION
VR offers a possible solution, which allows the user to directly interact with
a computer-simulated environment. It can lead to new and exciting discoveries in
these areas which impact upon our day to day lives.
Modern input devices have been massively influenced by VR and may
become the corner stone of further virtual reality developments (Virtual reality
blog, 2009). Nintendo Wii and Microsoft Xbox Kinect are good examples. The
former uses a controller, which can be latched to the hand, to make movement
becomes a form of input, while the later uses a camera to track a player's
movements, which are then reflected in-game.
Our society is aging rapidly. Currently, a little under 8% of the worlds
population is 65 or older and this percentage is expected to reach 16% by 2050.
The growth in the elderly population is more accentuated in developed countries
where life expectancy continues to rise (World Population Ageing, 2009). As a
result, the number of patients with motor function disorders can drastically
increase while the ability to care for them will be limited by public expenditure
and human resources. Thus, there is high demand for computer-aided tools which
support in-home rehabilitation (Gonzalez, Hayashibe & Fraisse, 2012).
In this chapter, the background information about virtual reality, virtual
rehabilitation, and related works are described.

1.1 Virtual reality

Virtual reality (VR) is a simulated environment that is created with special
software and presented to the user in such a way that the user is able to recognize
himself and operate in that environment (Virtual reality, 2012). Generally, in VR,
the user wears a head-mounted display or glasses displaying three-dimensional
images as part of their experiences. Some systems enable the user to experience
additional inputs by providing various sensory stimulation, such as audios, videos.
and images, to form part of the VR environments. Many newer environments
1

embed touch or force feedback by integrating wearable devices such as data glove
and helmet, further enhancing the realistic experiences.
A virtual environment should provide the appropriate real-time responses
when the users explore their surroundings. If a delay incurred between the users
actions and system responses, their feeling immersed in the VR environment may
be disrupted. The users become aware that they are not in a realistic but artificial
environment, which results in an adjustment of their behaviors accordingly by
responding with a stilted, mechanical form of interaction (Virtual reality blog,
2009).
The use of interfaces that require physical activity for an optimal user
experience becomes an increasing trend in gaming. Rehabilitation forms an
essential component of the therapeutic continuum in patients with multiple
injuries or motor disability. Effective rehabilitation programs help patients
optimize their level of physical, psychological, and social functions, as well as
reduce the length of patient stay, re-admission rates and use of primary care
resources.

1.2 Virtual rehabilitation

Virtual rehabilitation is one kind of therapy for treating patients based
entirely on VR simulated exercises. If there is no conventional therapy provided,
the rehabilitation is said to be virtual reality-based; otherwise, if virtual
rehabilitation is in addition to conventional therapy, the intervention is virtual
reality-augmented." (Burdea, 2002). The use of virtual reality systems may:
Allow for repeated and consistent practice of the same task (Rizzo and Kim,
2005; Rose, Brooks, & Rizzo, 2005)
Enable clinicians to progress difficulty and challenge levels (Rizzo and
Kim, 2005)
Enable clinicians and researchers to easily record and analyze performance
outcomes (Rizzo and Kim, 2005)

 Provide a safe environment to undertake tasks which may be difficult or

unsafe in real life (e.g. crossing a street, operating a motorized wheelchair)
(Rizzo and Kim, 2005; Schultheis and Rizzo, 2001)
Offer appealing games that may make therapy tasks more fun and engaging,
which may increase compliance with therapy (Harris and Reid, 2005; Rizzo
and Kim, 2005)
Enhance motivation to practice which may lead to longer practice duration
or more practice repetitions (Rizzo and Kim, 2005)
Provide enhanced real-time feedback about task performance or task results
which may be beneficial for learning (Holden, 2005; Rizzo and Kim, 2005)
Be useful as home interventions involving independent practice (Rizzo,
Strickland, and Bouchard, 2004)
Provide potential for telerehabilitation, if the virtual reality system is
internet-deliverable (Levac and Missiuna, 2009) (as clinicians can monitor
clients' practice from afar)
VR is a technology that undoubtedly has great potential to be used in
various branches of medicine. Although the systems associated with the use of
virtual reality have been present on the market for many years, their use is not
very popular. This is a modern approach to rehabilitation, which is fun for the
patient (Boian, Lee, Deutsch, Burdea & Lewis, 2002).
According to WHO, over a billion people, about fifteen percent of the
worlds population have some form of physical disability, around 150 million
people significant difficulties in functioning of limbs (WHO, 2013). Virtual
rehabilitation systems have advanced significantly over the past decade. It offers
the possibility to be precisely adapted to the patients therapy and to be specific.
It can provide realistic training for patients in different scenarios and phases of
the rehabilitation. It is now conceivable that computer-based rehabilitation
programs could be developed using current, widely available, affordable virtual
reality platforms, such as the Microsoft Kinect. Many VR systems have been
developed specifically for use in rehabilitation to improve movement skills. These
3

are often expensive systems which are limited to use in research laboratories
(Deutsch, Borbely, Filler, Huhn, and Guarrera-Bowlby, 2008). Figure 1-1 shows
list of benefits and challenges of virtual rehabilitation (Morganti, 2006)

Figure 1-1. Benefit/challenges in VR rehabilitation application

(Source: Morganti, 2006)
There are many reasons why virtual reality applications are so effective for
rehabilitation. First, virtual reality is an interactive, experiential medium. In the
same way that children and teenagers intuitively grasp computers, virtual reality
users become directly engaged with the effects of the virtual experience. In
addition, virtual reality creates a safe setting where patients can explore and act
without feeling threatened. Patients can make mistakes without be afraid of
dangerous, real, or humiliating consequences (Wiederhold, 2006). Moreover,
unlike human trainers, computers are infinitely patient and consistent. In cognitive
rehabilitation, virtual reality can be manipulated in ways that the real world cannot.
For example, virtual reality can convey rules and abstract concepts without the
use of language or symbols for patients with little or no grasp of language. Indeed,
virtual reality has been used to support individuals with intellectual disabilities by
training them in many skills of independent living, such as grocery shopping, food
preparation, orientation, road crossing, and vocational training (Standen and
Brown, 2005).
4

Virtual rehabilitation have some benefits to motivate patients. Cycle going

like this:
When patients will exercise harder and longer when immersed in a virtual
rehabilitation environment providing positive reinforcement.
This phenomenon is called a Virtuous Cycle, where patients receiving
positive feedback feel an increased sense of motivation, which encourages
them to perform more repetitions and/or exercise longer to surpass their
previous performance.
This motivation and encouragement is important for many aging adults,
particularly those who are reluctant to participate in the therapy process or
are fearful due to physical limitations.
Mentioned before when virtual reality exercises are combined with
traditional therapy techniques, rehabilitation outcomes can exceed
outcomes generated by traditional therapies alone.
There are two ways to classify Virtual Rehabilitation (Burdea. 2002).
The first one is based on specific patient condition that subdivided by:
Musculo skeletal Virtual Rehabilitation (Suffered a bone or muscle injury.)
Post-stroke Virtual Rehabilitation. (Stroke survivors.)
Cognitive Virtual Rehabilitation (Significant disability from individual
reasons.)
In second series classification is related to the rehabilitation protocol is
distinguished VR-augmented and VR based therapy. Before we mentioned about
these.

1.3 Motivations and Objectives

There are a variety of VR games developed under currently available
commercial game platforms such as Nintendo Wii and PlayStation Move to
provide people to entertain from the virtual environment. Different levels of
challenge options can be selected by people with different motor skills. However
these commercial games are mainly bounded to the fact that they are not
developed for rehabilitation purposes. Most of these videogames requires the
patient to hold a remote controller, which is difficult for some patients with
serious motor injuries.
After the prevalence of activity-promoting game systems, such as Nintendo
Wii and Sony PlayStation, injuries associated with handheld controller overuse
and repetitive motion were frequently found. Compared to the aforementioned
video game systems, Microsoft Xbox Kinect allows the users to interact with the
machine through body gesture without needing any handheld controllers, which
is believed to be able to significantly prevent the occurrence of musculoskeletal
injuries (Tanaka, Parker, Baradoy, Sheehan, Holash, and Katz, 2012).
Kinect is small and affordable, making it an excellent tool for use in
home-based rehabilitation (Chang, Chen and Huang, 2011). In particular, when
used at home, a Kinect-based system can encourage increased use of the upper
extremity for older adults who generally lack of movements. The system
presented here uses this device to track the patients hand when playing a game
designed specifically for upper limb rehabilitation in older adults. Furthermore,
no extra accessories, such as markers or hand supports, are required in the
developed system. The research objectives of this study are as follows:
Develop specialized exercise games for balance shift, upper limb, and
lower limb trainings using Microsoft Kinect.
Investigate the possibility of using the Microsoft Kinect to aid rehabilitation.
Increase physical activity of patients, as well as improve satisfaction and
enjoyment of therapy.

 Demonstrate Kinect to be useful for rehabilitation, both in clinical and

home settings.

1.4 Related works

Even before the release of the Kinect, some of project using Nintendos Wii
video gaming system. Such gesture interaction technologies are not new; however,
their recent availability as interface means within affordable mass-market gaming
products can be seen as evidence of a broadening usage beyond solely
entertainment. Often, Nintendos Balance Board is used as an input device, for
example, in various games for balance training (LaBelle, 2011). The Kinect, on
the other hand, is small and affordable enough to be used in virtually any home
environment, and it does not require patients to wear anything that could limit
their movement (Chang, Chen and Huang, 2011). According to an investigation
of self-reported cases, they reported 9 types of injuries identified; among them,
hand lacerations related to overuse or incorrect use of handheld controller were
the most commonly observed injuries (Sparks, Coughlin, and Chase, 2009). Other
studies have also identified the Kinects potential for use in physical therapy.
Chang, Chen and Huang (2011) developed a Kinect-based rehabilitation system
to assist therapists in their work with students who had motor disabilities. The
program used the motion tracking data provided by the Kinect to determine
whether the patients movements reached the rehabilitation standards and to allow
the therapist to view rehabilitation progress. The Kinect has been used for medical
purposes outside of physical therapy as well (Chang, et al., 2011). Rizzo and
others at the University of Southern California studied how video games that
require player movement could motivate persons at risk for obesity to engage in
physical activity (Lange, Suma, Newman, Phan, Chang, Rizzo and Bolas, 2011).
To demonstrate the concept, they developed a system using the Kinect in which
the popular computer game World of Warcraft could be controlled with user
gestures instead of mouse and keyboard commands.
A motor rehabilitation system can be developed applying different tools for
interaction such as haptic sensors, markers and cameras. The use of haptic sensors
is common on upper limb rehabilitation, where it is used to interact with the
system and also to enable feedback for the user (Luo, Kline, Fischer, Stubblefield,
8

Kenyon, and Kamper, 2005). Techniques that apply markers are commonly used
due its low cost and easiness of use. These markers are used as references to
extract information about the scene orientation and the positioning of the objects
in order to provide guidance to the treatment (Sparks et al, 2009). Another option
is the use of accelerometers and gyroscopes provided by the Nintendo Wii remote.
Despite being simple to implement, these approaches have the disadvantage that
the patient needs to hold or attach objects to his/her body, which is not always
applicable (Da Gama, Chaves, Figueiredo and Techrireb, 2012). Nevertheless,
none body reference for interaction is used on them, turning difficult to analyze
movement carefully, which is a powerful tool not only for the current patient
evaluation but also for the storage and future analysis of his progress on the
rehabilitation treatment (Timmermans, Saini, Willmann, Lanfermann, teVrugt.J,
and Winter, 2007). Nintendo Wii introduced a new style of VR in 2005 using a
wireless controller that interacts with the player through a motion detection
system and its avatar representation in the video. The controllers use embedded
acceleration sensors that can respond to changes in direction, speed and
acceleration to enable participants to interact with the games. A sensor, mounted
on top of a TV, captures and reproduces on the screen the movement from the
controller as performed by the participants. The feedback provided by the TV
screen generates a positive reinforcement, thus facilitating training and task
improvement

CHAPTER TWO
MICROSOFT KINECT DEVELOPING
ENVIRONMENT
2.1 Microsoft Kinect
Kinect is a motion sensing input device by Microsoft for the Xbox 360
video game console and Windows PCs. Based on a webcam-style add-on
peripheral for the Xbox 360 console, it enables users to control and interact with
the Xbox 360 without the need to touch a game controller, through a natural user
interface using gestures and spoken commands. It does not require users to hold
or wear any specialized equipment for tracking. The reasonably high accuracy (xy
resolution = 3mm, z resolution = 1 cm ) and low price makes it a good tracking
alternative for a home based rehabilitation system. (Schnauer and Mossel, 2011)
compared the performance of this device with a Motion Capture system and found
Kinect cannot measure many parameters and has lower accuracy. However, their
study demonstrated a custom game that was nevertheless controlled well with
Kinect.
Kinect competes with the Wii Remote Plus and PlayStation Move with
PlayStation Eye motion controllers for the Wii and PlayStation 3 home consoles,
respectively. Normal cameras collect the light that bounces off of the objects in
front of them. They turn this light into an image that resembles what we see with
our own eyes (Webb and Ashley, 2012). On the other hand, Kinect is a depth
camera, which records the distance of the objects that are placed in front of it. It
uses infrared light to create an image (a depth image) that captures not what the
objects look like, but where they are in space.
The reference device created by PrimeSense includes an RGB camera, an
infrared sensor, and an infrared light source. Microsoft licensed PrimeSenses
reference design and PS1080 chip design, which processes depth data at 30 frames
per second. The chip also automatically alignes the information for the RGB
camera and the infrared camera, providing RGBD data to higher systems.
10

Microsoft added a four-piece microphone array to this basic structure, effectively

providing a direction microphone for speech recognition that would be effective
in a large room. Figure 2-1 demonstrates the Kinect components.

Figure 2-1 Kinect components (Source: http://

http://praveenitech.wordpress.com/tag/rgb-camera/)
The Kinects infrared projector shines a grid of infrared dots over
everything in front of it. These dots are normally invisible to us, but it is possible
to capture a picture of them using an IR camera. The overarching goal of this
development was to determine whether the Kinect and the data it supplies are
useful in balance rehabilitation.

2.2 Design guidelines

The design should not only focus on the patients needs but also on the
rehabilitation needs. Our system fulfills the five features of a suitable game
described by Burke, Morrow, McNeill, McDonough, and Charles (2008), which
meets the requirement of a home-based system. These features include:
Level of difficulty
Direct feedback
Easy to configure
Low cost
Minimal therapist involvement

11

In this study, a physical therapist, computer specialist and a biomedical

engineering specialist involved in the study participated in the discussion. The
features considered during the design process were ranked to determine their
importance. We determined to adopt that the Microsoft metro for graphic
Interface design because it is simple and easy to operate for patients. Users
interacting with the application should be able to:
Perform therapy of lower extremities using their legs and feet, such as
Step on It game which will be mentioned in Chapter 3.
Perform therapy of upper extremities using their arms and
hands(Window Cleaner and Ping pong mentioned in Chapter 3)
Perform stretching exercises using their full body (Coin Collect
mentioned in Chapter 3)
Easy to use and highly customizable.

2.3 Choosing Open Source Drivers and SDKs

In December 2010, OpenNI and PrimeSense released their own Kinect
open source drivers and motion tracking middleware (NITE) for PCs running
Windows, Ubuntu and MacOSX.
In February 2012, Microsoft released a commercial Kinect Software
Development Kit (SDK) for Windows that includes Windows 7 compatible PC
drivers for the Kinect device that does not support older Windows versions or
other operating systems (Webb et al, 2012). Microsofts SDK allows developers
to build Kinect enabled applications in Microsoft Visual Studio using C++, C# or
Visual Basic. A comparison of the capabilities and advantages of the OpenNI tool
and the Microsoft Kinect SDK is shown in Table 2-1.

12

Table 2-1. Comparisons of Microsoft Kinect SDK and OpenNI.

Microsoft Kinect
SDK
Raw depth and
image data
Joint tracking
without calibration
Easy installation

OpenNI SDK

Yes

Yes

Yes*

No

Yes*

No

Yes*

No

20*

15

Only Works in

MacOSX,

Audio processing,
speech recognition
capability
Number of joints
available
Platform
Resource, sample
and support

Windows
Excellent*

Ubuntu, Windows*
Adequate

The advantages of the Microsoft SDK are its abilities to track joints without
needing calibration, to track 20 joints at the same time, and for easy installation.
On the other hand, OpenNI skeleton tracking requires the user to hold a psi pose
for calibration, which might be difficult for many patients. Figure 2-2 shows the
joints which can be detected by Microsoft SDK (Fig. 2-2a) and OpenNI SDK (Fig
2-2b) . As compared in Table 3-2, Windows SDK seems to be a better choice by
considering its ability to track seated skeletons using only the top 10 joints.

13

(a)

(b)

Figure 2-2. Kinect SDK and OpenNI skeleton joints (Modified

from: OpenNI and Kinect Official Website)

2.4 Kinect API

The Kinect for Windows SDK works as an interface between the Kinect
device and an application. When the application needs to access the sensors, it
sends an API call to the driver. The Kinect driver controls the access to sensor
data. Figure 2-3 illustrates the layer structure of the Kinect API and the sensors.
As shown in this figure, the installed drivers for the sensors sit with the
components of system device drivers and can talk to each other. The drivers help
stream the video and audio data from the sensors and return it to the application.
These drivers help detect the Kinect microphone array as a default audio device
and also help the array interact with the Windows default speech recognition
engine. Another part of the Kinect device driver controls the USB hubs on the
connected sensor.
14

The Kinect SDK libraries can be classified into the two following
categories:

The libraries used for controlling and accessing Kinect sensors

The libraries used for accessing microphones and controlling audio

The first category deals with the sensors by capturing the color stream,
infrared data stream, and depth stream, tracking human skeletons, and taking
control of sensor initialization. Audio APIs control the Kinect microphone array
and help capture the audio stream from the sensors, control the sound source, and
enable speech recognition. Figure 2-3 showed architecture of Kinect API.

Figure 2-3 Kinect application interface (source: Jana. 2011)

2.5 Skeleton stream

The Kinect sensor returns raw depth data from which we can easily identify
the pixels that represent the players. Skeleton tracking is not just about tracking
the joints by reading the player information; rather, it tracks the complete body
movement. The Kinect sensor can identify the pixel range of a player from the
depth data. In the initial steps of the process, the sensor identifies the human body
object, which is similar to another object captured by the sensor. In the absence
of any other logic, the sensor will not know if this is a human body or something
else. The tracking procedure is as follows (Jana, 2012):
15

1. Depth data is processed in the rendering pipeline process and matches with
decision forests labeled data and generates the inferred body segments.
2. Once all parts are identified based on the labeled data, the sensor identifies
the body joints.
3. The sensor then calculates the 3D view from the top, front, and the left of
the proposed joints.
4. Then the sensor starts tracking the human skeleton and body movement
based on the proposed joint points and the 3D view..
The Kinect for Windows SDK provides a set of APIs allowing us easy
access to the skeleton joints. Figure 2-4 shows the process flow in creating joint
points from raw depth data:

Figure 2-4 SDK skeleton detection processing.

The fundamental approach of gesture recognition is to deal with the
skeleton's joint points and apply basic logic to perform some action. If the
performed action is matched with predefined set of condition, application can
understand that the user has performed a certain gesture, otherwise not. Each joint
is measured in three dimensional (X, Y, Z) plane. X and Y coordinates actually
indicate the joint location in the plane, and Z indicates how far the joint is from
the sensor.
16

2.6 Math formulas implementation

Frequently, system code uses geometry and trigonometry. In general
mathematics, to calculate the distance between two points, we need to make use
of the Pythagorean Theorem(fig 2-5).

Figure 2-5. Pytagorean Theorem.

According to the rule of right triangle, squared hypotenuse is equal to the
sum of the squares of the other two sides. The same formula is applied for the
given two coordiantes (X1, Y1) and (X2, Y2) in the XY plane. The calculation for
the distance between two points on a horizontal line, then, may be generalized as
(x1- x2) and for two points on a vertical line the distance is (y1- y2). Then the
formula for finding the distance between any two points based on the Pythagorean
theorem as follows in equation 1.
2 = (1 x2 )2 + (1 y2 )2

(1)

Aforementioned The Pythagorean theorem works well for three

dimensional planes as well, and the distance between points (X1, Y1, Z1) and (X2,
Y2, Z2) can be calculated by the equation 2.
3 = (1 x2 )2 + (1 y2 )2 + (1 z2 )2

(2)

The Law of Cosines, as shown in Figure 2-6, calculates the angle of type
of triangle. This is useful when determining the angle between two joints. Appling
the formula for this purpose requires a third point, which can be another joint
position, but generally should be a point along the X-axis from the base point. The
largest angle calculable by the Law of Cosines is 180. When calculating the angles
17

between joints, this means additional calculating to determine angles from 180 to
360, but this is trivial.

Figure 2-6 Law of Cosines (modified from:

http://www.beginnerstaichi.com)
The Coding4Fun Kinect Toolkit is a set of extension methods and controls
to make developing applications for the Kinect using the Kinect for Windows
SDK easier.
The Coding4Fun Kinect Toolkit has a ScaleTo extension as part of the
library. This adds the ability to take a joint and scale it to any display resolution.
The scaling function is demonstrated in Fig. 2-7.

Figure 2-7 Scale function

In Fig. 2-8, maxPixel = width or height, depending on which coordinate
your scaling is; maxSkeleton = 1; and Position = the X or Y coordinate of the joint
you want to scale. When you want to move eclipse control by joint movement
then use function call like
Canvas.SetLeft(eclipse, Scale(640, 1, joint.Position.X));
18

Canvas.SetTop(eclipse, Scale(480, 1, -joint.Position.Y));

To depend on your needs change to 640 & 480 with a different scale.
If you include the Coding4Fun Kinect Toolkit, instead of re-writing code, you
could just call it like so:
scaledJoint = rawJoint.ScaleTo(640, 480);.
A lot of coding and customized function implemented in this system so let
explain about healthcare side.
Smoothing skeleton data
Kinect sensor does not have sufficient resolution to ensure consistent
accuracy of the skeleton tracking data over time. This problem manifests itself as
the data seeming to vibrate around their positions. The Kinect for Windows SDK
uses the Holt double exponential smoothing procedure to reduce the jitters from
skeletal joint data. The smoothing parameters solve the jittering problem by
filtering the skeleton data and applying a smoothing algorithm to it .In my case
developing in menu section, hand tracking have to be very smooth and jitterless.
Code shown in below.
SkeletonStream.Enable(new TransformSmoothParameters()
{
Correction = 0.5f,
JitterRadius = 0.05f,
MaxDeviationRadius = 0.04f,
Smoothing = 0.8f
};

19

Table 2-2. Kinect programming smoothing parameters

Parameter

Description

Default
Value

Smoothing

Specifies the
amount of
smoothing.

0.5

Correction

Specifies the
amount of
correction.

0.5

Specifies the jitterreduction radius, in 0.05

meters.
Specifies the
Max
maximum radius
Deviation
that filter positions 0.04
Radius
can deviate from
raw data, in meters.
(Source: Microsoft MSDN. 2013)
Jitter
Radius

Comments
Determines the amount of smoothing
applied while processing. If increase
this value you will get smoother
skeleton data, however, it increases
the latency. With the smoothing
value as zero, you will get the raw
skeleton data
Specifies the amount of correction
needed for the raw data. With lower
values more correction is applied,
the raw data is corrected, and the
data looks smoother.
The default value of 0.05 represents
5cm. Any jitter beyond the radius is
clamped to the radius.
This is the max limit of the deviation
that is allowed to be considered for
determining a jitter.

2.7 Software development

Our software is written in visual studio 2012 and C# language and design
language XAML used Extensible application markup language XAML language
is used for initialization of User Interface (UI) elements in a structured and
manageable way. XAML elements map directly to Common Language Runtime
(CLR) 2 object instances. Anything created in Xaml can be expressed using
any .NET language, such as C# or Visual Basic. Visual Studio is an integrated
development environment by Microsoft and used be programmers to develop
console applications or applications with a graphical user interface. Patients
performing progress and data stored in MySQL database management system.
Our project consists of a native Windows Presentation Foundation (WPF)
application that will authenticated users to access our physical therapy system.
System have following feature attached.

20

 The authentication feature. The system should allow authenticated users

to login and allow access to their information. No user in the same
category is allowed to access the other users information in the system.
Customizable. The system should change its functionality according to
the type of users in the system. The therapists are represented with
different functionalities within the same application in comparison to the
patients.
Database The system will include a database to store the information
about patients and therapists. We will separate the database access from
its implementation for security and portability reasons.
Figure 2-8 presents the architectural design of the whole system. As can be
seen in the diagram, the patient interacts with the Kinect which sends information
to the Software. Software controlled by patients hand movement to select module,
change module or exit the system. After choose module, the new window will
popup which contains exercise function. The software also contains the Main
Window in two forms, the XAML file and the Class file. XAML, which stands
for Extensible Application Markup Language, contains the Camera viewers and
the main page design. This is the page one will first see when starting up the kinect.
It displays whether the Kinect is connected or not. If it is, then it continues to show
the main background image and whatever the programmer has allowed to show
from the C# class on this page. Ive used this page as the menu for my software.

21

Figure 2-8 System Architecture Design

The last thing included in the Game are the components such as the text,
images and sounds used in the application. After the software detects the motion
of the user it sends information to the Display unit and speaker, whether on
monitor or TV in which the user can interact with.
Here is also Balance shift module activation diagram provided because
evaluation focused on balance therapy. Module designed for following
requirements:

Easy to operate and understandable.

Can be customizable for each individual patient.

Detect balance shift accurately

22

Provide realtime audio visual feedback.

Figure 2-9 Activity Diagram Balance shift

Our system can track only one closest user. First you get all the skeletons
in the returned collection. Then we find all the skeletons that belong to the closest
player currently being tracked and find the joints. Skeleton stream provides up to
six skeleton hipcenter joint data and skeleton IDs. When choosing closest skeleton,

23

compare Z coordinate of all skeletons then return skeleton ID with minimum Z

distance.
Login and user management
Login page is the first page the user interacts with. If an unauthorized user
attempts to enter the system, an error message is shown under the login button. In
the view-model of the login page we use the Login service contract of the database
connection to query if the user exists in the database. Login service will return a
status Enum value that we can branch to identify where to navigate to. Figure 210 shows system login page.

Figure 2-10 System login page

If the login status is returned as patient, user is redirected to the software
main menu. Otherwise login status returned as admin, user redirected to user
creation and user information management window (Figure 2-11).

24

Figure 2-11 User creation window

Database have designed some tables for portability reasons. First table
stored username, password and some extra information about patient. Activity
table created for user login date, duration and perform count.

25

CHAPTER THREE
METHODOLOGY
3.1 Interface Design
The main purpose of this software was to demonstrate that games
developed with the Microsoft Kinect is useful for rehabilitation. This work
presents a virtual rehabilitation system consisting of 5 games proposed and
developed by an interdisciplinary group including experts majoring in computer
science, computer engineering, biomedical engineering, physical therapy, and
rehabilitation.
As shown in Figure 3-1, the personalized gesture-based carousel spinning
menu provides the patients with easier interface to adjust the main program
according to individual needs. A patient can swipe the next item from right to left
or swipe back the item from the left to right side using ones right hand. The Users
can raise up the right hands above their heads in order to enter and load a new
screen to initiate an exercise.

Figure 3-1 Carousel styled system main menu

26

One-handed gestures, such as swipe gestures, are also easier for users to
learn, remember, and operate than two-handed gestures. For the SwipeToRight
gesture, we will use the following requirements:

Each new position should be placed at the right of the previous one

Each position must not exceed the height the first one by more than a
given distance (20 cm)

The time between the first and last positions must be between 250 and
1500 ms

The gesture must be at least 40 cm in length

Swipe and pushing gestures are most frequently used in Kinect

programming. The metro styled menu, much like mouse cursor, is simpler to use
by providing visual cues to the patients to guide to scroll the icons. And then, they
can just press to push the selected exercise icon to load the game.

Figure 3-2 Metro styled menu

Figure 3-3 (a) and (b) show the icons with default targeting state and
actionable state, respectively. As a user extends his or her arm in a pressing motion,
a hand color fill goes up and down within the cursor (Fig. 3-3c) to indicate how
far the user has achieved the requirement of the posture. This feature was shown
to be able to improve pressing accuracy and learnability. It also makes it easier
27

for users to cancel before a press has been made, or retarget if a press is about to
happen over the wrong control. The cursor also has a visual state, Fig. 3-3 (d) that
indicates when the user has completed a press.
When Kinect detects that the users hand is in a closed fist state, as shown
in Fig. 3-3 (e), the cursor changes to a gripped visual and shows a color consistent
with the fully pressed state. This confirms to the user that the grip is detected, as
well as being a strong visual cue for how to make a recognizable gripped hand.
Figure 3-3 (f) compares the right hand icon with the left hand cursor.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 3-3 Standard cursor and feedback

3.2 Development of Kinect Virtual Rehabilitation System

The developed system contains five games, including Ping Pong, Balance
Shift, Window Cleaner, Step on It, and Coin Collection games, which can be used
for upper limb exercise and balance therapy. Plenty of parameters and different
levels of difficulty can be set to customize the users need. The operations of
individual games are described as follows.
Ping Pong
The aim of this game is to defeat the competitor with higher score. When
the game starts, the ball randomly bounces down to the ground slowly. The user
can move his/her hand and elbow to change the direction of the paddle. The score
will be recorded at the bottom left corner of the screen. Players who earn more
points will be awarded. This game was designed for upper body, especially for
training flexion and abduction of the elbows and hands. The moving speed of the

28

ball will increase if the exercise has been carried out well. Figure 3-4 shows the
graphic user interface of the Ping Pong game.

Figure 3-4 Ping Pong

Balance Shift
For Wii balance board compatible games, the users interacts with the game
by shifting their weight on the board. However, our software, instead of using Wii
balance board, detects balance shift by processing Kinect skeleton frames. Figure
3-5 shows the game developed for training patients how to make correct balance
shifts. The exercise can be divided into easy (Level 1) and difficult (Level 2)
modes providing different degrees of challenge to patients. The default mode is
Level 1 which asks the patient to stand in straight and do balance shift. Level 2 is
a little more difficult than Level 1. At level 2, patients have to bend their knees
before starting doing balance shift. The software can automatically detect the
patients knee angles to determine whether they are standing or knee-bending.
Real-time audio and visual feedbacks are also provided. As shown in the figure,
when the patient makes a shift, the gauge styled indicator shows which side the
patient has shifted. When a shift has been completed and the system has identified
that one has done a correct action, the system play a sound showing it is a
29

successful shift and the counter increases by 1. Because some severe patients
cannot make balance shifts with great angles, the software provides the function
for the therapist to adjust the left and right threshold angles to the neutral position
(90o) for determining whether a balance shift is counted as successful according
to the patients status. The adjustable angle ranges from -10o (right shift) to +10o
(left shift), resulting the shifting angle spanning from 80o (right shift) to 100o (left
shift). For example, some serious disabled patients can start with a small shifting
angle and increase to a larger angle when ones health status is getting better.

Figure 3-5 Game for training balance

30

Window Cleaner
This game targeted to upper limb exercise using both shoulder extension,
shoulder abduction and elbow flexion. The objective of this game is supposed to
clean a series of mirrors as quickly as possible, by wiping off the virtual dirt . The
background image of the game can be changed and both hands can be switched at
any time. In addition, the game provide the function to select different levels of
difficulty to help motivate the players in playing the game. Figure 3-6
demonstrates the graphic user interface of the game.

Figure 3-6 Window Cleaner

31

Step on It
Provides training on lower limb and trunk balance of the player. The Game
scenario is coming from dance pad, also known as dance mat, dance platform, or
flitter deck, which is is a flat electronic game controller used for input in dance
games. As you can see from the Figure 3-7, the player should stand on the center
of 33 virtual matrix square panels on one foot and move another according to
the guide of the game. When one panel randomly changes the color, the player
has to step one foot on the colored panel while keeping the other foot on the central
panel of the matrix.

Figure 3-7 Step on It game

32

Coin Collection
Knowing that game will be based on an augmented reality scenario, this
exercise displays the tracked body and coins falling from the sky. Using the
players head, left hand, right hand, left foot, and right foot joints, one can catch
coins and thus get scores. As illustrated in Figure 3-8, the score, game time, and
moving angle of the player will be displayed to help the user understand and feel
comfortable with the game.

Figure 3-8 Coin Collection game

3.3 Experiment
In this pilot study, in order to test the usefulness and effectiveness of the
Kinect Virtual Rehabilitation System, the Balance Shift game was used to train
patients with stoke to recover their balance ability. Stroke is one of the most
common causes of death in the developed countries. Stroke patients often suffer
from hemiparesis, affecting their balance ability and consequently their selfdependency and quality of life. Balance rehabilitation can be a long and tedious
process. Virtual rehabilitation systems have been reported to provide therapeutic
benefits to the balance recovery of stroke patients while increasing their
33

motivation (Gonzalez, Hayashibe, and Fraisse, 2013). Although therapeutic

advances have reduced the mortality during the acute stages of stroke, patients
still experience severe neurological disabilities. Deficits in motor control,
abnormal synergistic organization of movements, muscle weakness, sensory
deficits, and loss of range of motion can all reduce the quality of life in stroke
patients. The reluctance to bear weight on the paretic leg may persist and cause
gait disturbances, despite conventional physical therapy to correct asymmetric
standing posture (Ishii, Matsukawa, Horiba, Yamanaka, Hattori, Wada and Ojika,
2010) Moreover, an asymmetric posture could foster further disuse and hinder the
recovery of motor function in the lower limbs during motor convalescence. The
balance training game was developed with the following goals:

Improving balance function and safety

Improving visual motor control and tolerance of motion

Increasing activity levels

Reducing falls or risks of falls

Balance Retraining Therapy is a specialized form of physical therapy to

decrease primary symptoms of movement-related dizziness and imbalance
through a customized "hands-on" approach. Additional symptoms addressed in
therapy may include decreased strength, loss of range-of-motion, muscle tension,
anxiety, and fatigue.
To skillfully control the center of gravity (CoG) is important for keeping
ones body in balance. CoG is defined as:
The balancing, equilibrium or pivoting point in the body.
The point where the sum of all the forces acting on the body is zero.
The point at which all the weight of the body may be considered to
be concentrated and about which all the parts exactly balance.
In the human body, when standing in the anatomical position the CoG is
located anterior to the second sacral vertebra.. The location of the CoG can be
outside the human body during activities depending on the relationship of body
34

segments. A humans CoG trajectory is useful to evaluate the dynamic stability

during daily life activities such as walking and standing up (Center Of Gravity,
2013).
The CoG is an imaginary vertical line passing through the CoG down to a
point in the base of support. In the human body it passes from the vertex through
the body of the second sacral vertebra down to a point between the feet when
standing in the anatomical position. The gravitation pull acting at the CoG of any
segment is along the LoG. As shown in Fig. 3-9, the CoG is located at the position
which is slightly above the hip center along the line of gravity (McCreary. 1993).
In this study, the CoG is detected from the Kinect infrared image.

Figure 3-9 Human center of gravity and line of gravity (modified

from http://killergearz.wordpress.com/2010/05/11/facts-about-humanbody/)
The system was also designed for wheelchair users to support their balance
improvement. Moderate to severe strokes often cause a loss of sitting balance in
patients so balance is one of the first things that needs to be addressed in stroke
35

rehabilitation. If one cannot sit up, it is impossible to do activities of daily living

(ADLs) such as dressing, bathing, and toileting. One cannot learn to stand safely
until sitting balance is maintained first.
The experient was conducted at the Department of Rehabilitation of
National Taichung Hospital. In the first series, the participants were asked to stand
with a level, shoulder-width distance, by keeping their eyes open (Fig 3-10a), and
their weight-shifting was recorded and calculated.

(a)

(b)

Figure 3-10 Balance shift a) Correct movement b) Wrong shift

(modified from http://www.beginnerstaichi.com)

3.4 Subjects and Statistic analysis

A total of 10 patients (5 males and 5 females) with age ranging from 47 to
87 years old were recruited to participate the study. After the stroke, they have
difficulties in balancing their limbs because of neurologic injures.
All tests were conducted in the Department of Rehabilitation, National
Taichung Hospital. Each participant underwent 10 sessions of treatments within
10 days, each lasting for 3 minutes. Among them, data of 2 patients were excluded
36

because they hadnt finished all of the treatment sessions. Table 3-1 shows the
demographic information of the participants.
Table 3-1 Demographic information of the recruited patients.
Male

Female

Total

52.5 4.7

55.7 4.3

54.1 (4.5)

Sex (number)
Age (years)

In order to assess the effectiveness of the developed virtual rehabilitation

system, patients activity during individual treatments were logged into the
database for later analysis. A statistic software package (SPSS 18) was used for
descriptive and inferential statistical analyses. The outcomes were tested with
paired-sample t-test and one-sample t-test with significance level defined as
p<0.05.

3.5 Rehabilitation Assessment

All the participants were assessed with the Berg balance scale, Biodex
balance machine, Get up and Go test, and our developed Kinect rehabilitation
system, as well as the user satisfaction survey at the beginning of, ten days after,
and at the end of the treatment. The experiments were divided to two sections:
Initial assessment
Final assessment
The experimental procedure is shown in Figure 3-11. The tests used for
evaluating the patient status include:
Kinect 2 min Test conducted by our developed software
Biodex Balance Test conducted by a machine)
Berg Balance Scale (BBS) Test scaled by a questionnaire
User Satisfaction Survey scaled by a questionnaire
Get up and Go test

37

Initial assessment:
BBS, Biodex, Kinect
rehab and Get up and Go
tests

Recording duration of

making 20 left and

20 right balance
shifts

Ten days treatment

Recording special
events and
collecting patient
opinions

Final assessment:
1. BBS, Biodex, Kinect
rehab and Get up and Go
tests
2. Questionnaire survey

Data
Analysis
Figure 3-11 Experimental procedure

38

Biodex Balance Machine

The

participants

underwent

balance

ability

examination

before

participating in the study. First, participants balance ability was measured using
a Biodex balance machine. This system uses a multiaxial testing platform which
can be set at variable degrees of instability. None of the subjects had experienced
vestibular system dysfunction, cerebral concussions, or central or peripheral
neurological dysfunctions. Biodex Balance System is a widely used device that
quantifies dynamic balance performance. It evaluates the ability to maintain
equilibrium while standing on a movable support surface with varying degrees of
instability (Biodex Balance System. 2012). Like other clinical instruments, the
first prerequisite to use of BBS is to determine the degree to which its scores are
reproducible with repeated measurements in conditions where the response
variable (e.g., balance performance) is stable.

Figure 3-12 Biodex Balance machine

Establishing reliability is necessary for either situation, when using BBS to
discriminate balance among individuals or using it to evaluate balance changes
over time following an intervention program.

39

Berg Balance Scale

The Berg Balance Scale (BBS), named after Katherine Berg, is a widely
used clinical test of a person's static and dynamic balance abilities. For functional
balance tests, the BBS is generally considered to be the gold standard. It was
developed to measure balance among older people with impairment in balance
function by assessing the performance of functional tasks. It is a valid instrument
used for evaluating the effectiveness of interventions and for quantitatively
describing functions in clinical practice and research. A 14-item scale was
designed to measure the balance status of an older adult in a clinical setting.
The test takes around 1520 minutes and comprises a set of 14 simple
balance-related tasks, ranging from standing up from a sitting position to standing
on one foot. The degree of success in achieving each task is given a score ranging
from zero (unable) to four (independent) with the final measure calculated by
summing all the scores. The BBS has been recently identified as the most
commonly used assessment tool across the continuum of stroke rehabilitation. It
is considered a sound measure of balance impairment.
Get up and Go test.
The scoring is based on the observation of a patient's movements with
deviations from a confident, normal performance, using the following scale:
1 = Normal
2 = Very slightly abnormal
3 = Mildly abnormal
4 = Moderately abnormal
5 = Severely abnormal
where "Normal" indicates that the patient gave no evidence of being at risk
of falling during the test or at any other time; and "Severely abnormal" represents
that the patient appeared at risk of falling during the test. Intermediate grades
reflect the presence of any of the following as indicators of the possibility of
falling: undue slowness, hesitancy, abnormal movements of the trunk or upper
limbs, staggering, stumbling. Patients with a score of 3 or more on the Get Up and
40

Go test indicates that they are at the risk of falling (Podsiadlo and Richardson,
1991).
Kinect Virtual Rehabilitation System
In the second series, the participants were asked to stand in shoulder-width
stance on a fall-protection platform by keeping their eyes open to conduct balance
shifts for two minutes to determine their balance ability. The patient was asked to
do exercise with the developed game for 2 minutes with the number of movements
to the right and left sides being recorded. Balance shifting gesture was adopted
from ancient Chinese Taichi movements, which emphasize on weight-shifting,
postural alignment, and coordinated movement skills. Figure 3-13 shows an
example of a patient doing exercise in the rehabilitation center of a hospital. The
experimental setup is equipped with a 32-inch monitor, hand-hold supporting rail,
computer, and the Kinect device.

Figure 3-13 Illustration of a patient testing the system

Usage attitudes have been assessed using the questionnaire after the
subjects have used the e-rehabilitation system. In this study, the nine recruited

41

volunteers were asked to answer the question items included in the questionnaire
after having used the system.

42

CHAPTER FOUR
EXPERIMENTAL RESULTS
Table 4-1 compares the treatment outcomes of the initial and final tests
using pair-sample t-test. As shown in the table, the Medial Lateral Index of the
Biodex assessment, Get Up and Go Scale, BBS Sum, Left Shift Count, and Right
Shift Count show significant improvement (pair-sample t-test, p<0.05) after
treatments with Kinect Virtual Rehabilitation System (KVRS). However, no
significant improvement (p>0.05) was observed for Overall Stability Index and
Anterior/Posterior Index of the Biodex assessment. In this study, the patients were
only asked to practice medial/lateral balance shifts, thats why the
Anterior/Posterior Index was not significantly improved, The Overall Stability
Index was calculated based on both Anterior/Posterior Index and Medial/Lateral
Index.
Table 4-1 Comparisons of balance ability for patients before and after treatment
with Kinect Virtual Rehabilitation System.
Initial Test
Mean(SD)

Final Test
Mean (SD)

0.46(0.14)

0.58(0.11)

-0.075 (0.14)

-1.42

0.197

0.38(0.12)

0.39(0.11)

0.005 (0.15)

0.09

0.929

0.22(0.07)

0.30(0.92)

-0.07(0.08)

-2.39 0.048*

6.46(6.74)

2.71 0.030*

-2.00(2.00)

-2.82 0.025*

Mean Differ.
(SD)

tvalue

pvalue

Overall
Stability Index
Anterior/Poster
ior Index
Medial Lateral
Index
Get Up and Go
Score

23.78(11.86) 17.31(5.92)

Beg Balance
Scale

50.50(3.85)

52.50(2.82)

43

Left Shift
Count

14.00(6.39)

36.13(11.29)

-22.12(11.23)

-5.57 0.001*

14.00(6.80)

35.88(11.58)

-21.87(11.55)

-5.35 0.001*

Right Shift
Count

Usage attitudes were assessed using the questionnaire after the patients
have finished all the 10 sessions of treatments using the KVRS. The results were
compared with the neutral value (3) and tested with one-sample t-test with
significance defined as p<0.05 (Table 4-2).
The eight volunteers were asked to answer the question items in the
questionnaires after used the system. As indicated in Table 4-2, the patients
expressed that the KVRS is interesting (p<0.05) and easy to control by hand
(p<0.05); the GUI of KVRS is easy to use (p<0.001); the feedback message is
clear and easy to understand (p<0.05); the treatment is effective (p<0.01); and
they feel comfortable with the game (p<0.01) and can immerse in the virtual
environment (p<0.01). However, they dont agree that KVRS can simulate the
realistic environment (p>0.05).

44

Table 4-2 Assessment of user attitude.

Question

I think playing the KVRS

Mean
(SD)

Mean
p-value Differenc Lowest Highest
e

3.88(1.03)

2.42 0.046

0.88

4.13(0.83)

3.81 0.007

1.12

4.25(0.70)

5.00 0.002

1.25

4.25(1.03)

3.41 0.011

1.25

3.25(0.88)

0.79 0.451

0.25

4.13(0.99)

3.21 0.015

1.12

4.38(0.74)

5.22 0.001

1.37

4.50(0.54)

7.93 0.000

1.50

games is interesting*.
I think I can immerse in the
designed environment when
playing the KVRS games**
I think treatments with
KVRS games is effective**
I think the KVRS games can
be easily controlled by
hands*
I think the KVRS games can
simulate the realistic
environment.
I can understand the
messages feedback by the
KVRS games*
I am comfortable with the
KVRS games***
I think the KVRS games are
easy to operate***

45

CHAPTER FIVE
DISCUSSION AND CONCLUSIONS
Spark et al. (2011) categorized Wii-related injuries into 4 different types:
tendinopathy, bursitis, enthesitis, and epicondylitis (Sparks, Coughlin, and Chase,
2009). According to an investigation of self-reported cases, 9 types of injuries
have been identified; among them, hand lacerations related to overuse or incorrect
use of handheld controller were the most commonly observed injuries (Sparks et
al, 2009). Other injuries, such as Tendinitis occurred in the thumb (Macgregor,
2000; Koh, 2000; and Karim, 2005) and Wii-itis in the shoulder and upper arm
(Bonis 2007; Sperling, Nett, and Collins, 2008), related to intensive use or
improper use of handheld controller were also reported in the medical literature.
Little resistance offered by the light-weight handheld controller to the
aggressive user leading to awkward deceleration to the upper extremity might be
the reason causing such injuries (Sterling et al, 2008). In addition, for sport-related
video games, such as tennis, bowling, golf, baseball, soccer, boxing, fighting,
physical fitting, and other sporting games, currently 1available on popular video
game platforms, injuries related to forceful and prolonged movements of body
and extremity are frequently observed because, unlike authentic sports, physical
strength and endurance are generally not limiting factors to discontinue the
sporting activities (Bonis 2007; Sterling et al, 2008). Compared to other video
game platforms, Microsoft Xbox Kinect allows the users to interact with the
machine through body gesture without needing any handheld controllers, which
is believed to be able to significantly prevent the occurrence of musculoskeletal
injuries.
Health professionals are always searching for a more effective treatment
that focus not only at the elimination of the pathology symptoms but also hold the
patient involved to it during the entire treatment in order to achieve the cure. The
importance of taking into consideration related human factors, such as patient
satisfaction and motivation is the key to ensure patient involvement and to achieve
46

a successful treatment. These factors are even more important for the success of
physiotherapy treatments due to the fact that the patients recovery is directly
associated with his/her continuous effort, commitment and discipline during the
entire rehabilitation process. This process consists in a series of sessions where
the patient must perform therapeutic exercises.
Sedentary activities, such as watching television, play video game, and
conducting cognitive work, may risk people in overconsumption of food, resulting
in the acquisition of obesities, cardiovascular diseases, and cancers (Chaput,
Klingerberg, Astrup and Sjdin, 2011). A more recent study also reported that
compared with inactive individuals, those with low-volume physical activities (an
average activity of 92 min/week) can reduce the risk of all-cause mortality of 14%
and increase the life expectancy of 3 years. In addition, Short-term of vigorousintensity activities have the same effect as mid-term of moderate-intensity
activities and long-term of mild-intensity activities (Wen, Wai, Tsai, Yang, Cheng,
Lee, Chan, Tsao, and Wu, 2011). Graves and Stratton (2008) reported that playing
bowling, tennis, and boxing of the Wii Sports consumed at least 50% more energy
than sedentary gaming for adolescents. Compared to a traditional sedentary video
game, it was observed that the heart rate, oxygen uptake, and energy expenditure
of schoolchildren were significantly higher for 2 Kinect activity-promoting video
games (Smallwood, Morris, Fallows, and Buckley, 2012).
In motor rehabilitation, there are essentially three major advantages that
virtual reality offers over traditional therapy alone. First, virtual reality creates a
safe, controlled environment for repetitive practice, and repetitive practice is
crucial in learning motor tasks. Second, virtual reality provides immediate, realtime feedback about performance. Finally, because of its interactive nature, virtual
reality can increase motivation by making the experience fun (Holden, Bettwiler,
Dyar, Niemann, and Bizzi, 2001)
This work presented a rehabilitation system based on markerless interaction
in a virtual reality environment. The Kinect has shown much potential for use in
balance therapy. The system developed in this project were primarily intended
47

for use by balance professionals, but as mentioned above the Kinect could also
prove to be a useful tool in at-home therapy software. This system uses the Kinect
device for interaction and it is based on games controlled by therapeutic
movements, which were set by physiotherapists, inducing the user to do exercises
correctly. It is capable of identifying whenever the patient is doing it correctly,
otherwise warning him/her and also taking down the statistics in a report for a
further professional analysis.
Statistical analyses of paired t-test effect showed significant difference
between the initial and final assessment in the Berg Balance Scale, Get Up and
Go test, and Kinect virtual rehabilitation test (p<0.05), but not in the Biodex
assessment. P value is less than 0.05 which it is we can conclude that the groups
had trained effectively. In fact the test is effective patients who took the training
did better after taking training. As we see from the result, it indicated a positive
outcome for the patients after got involved the treatment.
Usage attitudes were assessed using the questionnaire after the subjects
have used the e-rehabilitation system. The outcomes were tested with one-sample
t-test with significance defined as p<0.05. The eight volunteers were asked to
answer the question items in the questionnaires after used the system. The
experimental result showed that the virtual training had a significant time effect
in the balance recovery of patients. With regards to the Berg Balance Scale, Get
up Go Test and Kinect Rehab system, the participants conditions improved
significantly between the initial and the final assessment. Ten days training is very
short time for balance recovery but Berg Balance Scale test and Get up and Go
test showed slight positive difference.
The system reached the five features considered during the design
process. Each feature is listed below:

Level of difficulty: All patients were able to play the game and

improve their scores over the study. Furthermore, none of the patients felt
extremely tired after a session.

48

Direct

audio and visual feedback:The game score was

displayed at all times and users were able to see it and comment about it.
Patients remembered the score from the previous day and tried to improve
it. Patients quickly understand that moving their arm caused the cursor on
the screen to move and were also able to control it.

Easy to configuration: No configuration required to play the

game. Before playing the game, only tracking of the hand had to be started.

Low cost: The total cost of this prototype system was 700

USD(kinect sensor 150$ computer 550$).

Minimal therapist involvement:There was no need for a

therapist to be present during the treatment. Only one person was with the
patient during the sessions and the only tasks performed were helping the
patient stand up in the front of kinect and start the game (Burke et al, 2008).
Compared to other video game platforms, Microsoft Xbox Kinect allows
the users to interact with the machine through body gesture without needing any
handheld controllers, which is believed to be able to significantly prevent
musculoskeletal injuries. In conclusion, the developed Kinect virtual
rehabilitation system is useful for rehabilitation either in clinical setting or at
home. Adopting the VR rehabilitation system at home is more flexible, cheap, and
convenient for the patients and allows for more frequently repetitive exercises.

49

REFERENCES
Biodex balance system. (2012). Machine instruction material. Retrieved
Apr 30, 2013, from http://www.Biodex.com/.
Boian, R. F., Lee, C. S., Deutch, J. E., Burdea, G. C., & Lewis, J. A.
(2002). Virtual reality-based system for ankle rehabilitation post stroke.
Proceedings of the First International Workshop on Virtual Reality
Rehabilitation , 7786, Citeseer.
Bonis, J. (2007). Acute Wiiitis. New England Journal of Medicine,
2431-2432.
Deutsch J.E., Borbely M., Filler J., Huhn K., Guarrera-Bowlby P.
(2008). Use of a low-cost, commercially available gaming console (wii) for
rehabilitation of an adolescent with cerebral palsy. Physical Therapy, 88(10),
1-12.
Burdea, G. C. (2002). Virtual rehabilitation--benefits and challenges.
Methods of Information in Medicine, 42(5), 519-523.
Buker, J. W. (2010). Augmented Reality Games for Upper-Limb Stroke
Rehabilitation. 2nd International Conference on Games and Virtual Worlds for
Serious Applications 2010, pp. 75 78.
Burke, J. W., Morrow, P. J., McNeill, M. D. J., McDonough, S. M., &
Charles, D. K. (2008). Vision Based Games for Upper-Limb Stroke
Rehabilitation. Machine Vision and Image Processing Conference, 159-164.
Center

Of

Gravity,

(2013).

Retrieved

Mar

25,

2013,

from

www.kau.edu.sa/Files/0052891/Subjects/CENTER-4.DOC.
Morganti, F. (2006). Virtual interaction in cognitive neuropsychology.
Cyberteraphy.
Chang, Y. J., Chen, S. F., & Huang, J. D. (2011). A Kinect-based system
for physical rehabilitation: A pilot study for young adults with motor
disabilities. Research in Developmental Disabilities, 2566-2570.

50

Chang, C, Y., Rizzo, A., Lange, B., Suma, E, A., & Bolas., & Newman,
B. (2011). Development and evaluation of low cost game-based balance
rehabilitation tool using the microsoft kinect sensor. International Conference of
the Engineering in Medicine and Biology Society, 18311834.
Chaput, J. P., Klingerberg, L., Astrup, & A., Sjdin, A. M. (2011).
Modern sedentary activities promote overconsumption of food in our current
obesogenic environment. Obesity Reviews, 12-20.
Da Gama, A., Chaves, T., Fugueiredo, L., & Teichrieb, V. (2012).
Guidance and Movement Correction Based on Therapeutics Movements for
Motor Rehabilitation Support Systems. IEEE Proceedings of XIV Symposium of
Virtual and Augmented Reality, 191-200.
De Mauro, A. (2011). Virtual Reality Based Reahabilitaion and Game
Technology. eHealth & Biomedical Applications Vicomtech, 48-52.
Freitas, D. Q., Da Gama, A., Figueiredo, L., Chaves, T. M., MarquesOliveira, D., Teichrieb, V., & Arajo, C. (2012). Development and Evaluation
of a Kinect Based Motor Rehabilitation Game. Proceedings of SBGames
international conference, 2012, 144-152.
Gerling, K. M., Schild, J., & Masuch, M. (2010). Exergame Design for
Elderly Users: The Case Study of SilverBalance. ACE '10 Proceedings of the
7th Int. Conf. on Advances in Computer Entertainment Technology, 66-69.
Gonzalez, A., Hayashibe, B., & Fraisse, P. (2012). Subject-specic Center
of Mass Estimation for In-home Rehabilitation- Kinect-Wii board vs. ViconForce plate. International Conference on NeuroRehabilitation, 705-709, Spain.
Herbelin, B., Ciger, J., & Brooks, A. L. (2008). Customization of
gaming technology and prototyping of rehabilitation applications. Seventh
International Conference on Disability, Virtual Reality and Associated
Technologies with ArtAbilitation, 211-218.
Holden, M, K. (2005). Virtual environments for motor rehabilitation:
Review. Cyberpsychology and Behavior, 187-211.

51

Holden, M. K., Bettwiler, A., Dyar, T., Niemann, G., & Bizzi,E. (2001).
Retraining movement in patients with acquired brain injury using a virtual
environment. Stud Health Technol Inform, 81, 192-8.
Ishii, F., Matsukawa, N., Horiba, M., Yamanaka, T., Hattori, M., Wada,
I., & Ojika, K. (2010). Impaired ability to shift weight onto the non-paretic leg
in right-cortical brain-damaged patients. Clinical Neurology and Neurosurgery,
406-412.
Jana, A. (2012). Kinect for Windows SDK Programming Guide. Packtt
Pub, Birmingham.
Graves, L., & Stratton, G. (2008). Energy expenditure in adolescents
playing new generation computer games. Br J Sports Med, 592-594.
Harris K., & Reid. D. (2005). The influence of virtual reality play on
children's motivation. Can.J.Occup.Ther., 72, 21-29.
McCreary, M. (1993). Muscles: Testing and Function p.75. [Book]
Mirelman, A., Patritti, B. L., Bonato, P., & Deutsch, J. E. (2010). Effects
of virtual reality training on gait biomechanics of individuals post-stroke. Gait
& Posture, 31(4):433437.
LaBelle, K. (2011). Evaluation of kinect joint tracking for clinical and
home stroke rehabilitation tools. [thesis]
Lange, B., Suma, E, A., Newman, B., Phan, T., Chang, C. Y., Rizzo, &
A., Bolas, M. (2011). Leveraging Unencumbered Full Body Control of
Animated Virtual Characters for Game-Based Rehabilitation. HCI (14) 2011,
243-252.
Levac, D., & Missiuna, C., (2009). An update on the use of virtual
reality technology to improve movement in children with physical
impairments. CanChild Online published.
Luo, X., Kline, T., Fischer, H., Stubblefield, K., Kenyon, R., & Kamper,
D. (2005). Integration of Augmented Reality and Assistive Devices for PostStroke Hand Opening Rehabilitation. Proceedings 2005 of IEEE Engineering
in Medicine and Biology 27th Annual Conference, 6855-58.

52

Microsoft press. (2012) Human interface guidelines Kinect SDK 1.6.

Retrieved 24, May, 2012, from http://msdn.microsoft.com/enus/library/jj663791.aspx.
Sperling, J. W., Nett, M. P., Collins, M. S.(2008). Magnetic resonance
imaging of acute wiiitis of the upper extremity. Skeletal Radiol, 481-83.
Nintendo. (2012). Official website. Retrieved Nov 20, 2012, from
http://www.nintendo.com/wii/what-is-wii.
Podsiadlo, D., & Richardson, S. (1991). The timed "up & go": A Test of
Basic Functional Mobility for Frail Elderly Persons. Journal of the American
Geriatrics Society, 121-132.
Koh, T, H. (2000). Ulcerative nintendinitis: a new kind of repetitive
strain injury. Medical Journal Aust,. 671-673.
Macgregor, D, M. (2000). Nintendonitis? A case report of repetitive
strain injury in a child as a result playing computer games. Scottish Medical
Journal,. 150.
Rizzo, A., Lange, B., Suma, E, A., & Bolas, M. (2011). "Virtual Reality
and Interactive Digital Game Technology: New Tools to Address Obesity and
Diabetes," Journal of Diabetes Science and Technology, 258-264.
Rizzo, A., & Kim, G. J. (2005). A SWOT analysis of the field of virtual
reality rehabilitation and therapy. Presence, 14(2), 119-146.
Rose, F. D., Brooks, B. M., & Rizzo, A. (2005). Virtual reality in brain
damage rehabilitation: Review. Cyberpsychology & Behavior, 241-262.
Schonauer, C., Pintaric, T., Kaufmann, H., Jansen - Kosterink, S. &
Vollenbroek-Hutten, M. (2011). Chronic pain rehabilitation with a serious
game using multimodal input. 2011 International Conference on Virtual
Rehabilitation (ICVR), 1-8.
Schnauer, C., & Mossel, A. (2012). Distributed Interaction with Kinect
and Mobile Phone in Augmented Reality. IGDA Unity3D, Vienna.
Schultheis, M. T., & Rizzo, A. (2001). The application of virtual reality
technology in rehabilitation. Rehabilitation Psychology, Vol 46, 296-311.

53

Smallwood, S. R., Morris, M. M., Fallows, S. J., & Buckley, J. P.

(2012). Physiologic responses and energy expenditure of Kinect active video
game play in schoolchildren, Arch Pediatr Adolesc Med, 1005-1009.
Sparks, D., Coughlin, L., & Chase, D. (2009). 'Wii have a problem: a
review of self-reported Wii related injuries. Informatics in Primary Care, 55-57.
Standen, P. J., Brown, D. J., (2005). Virtual Reality in the Rehabilitation
of People with Intellectual Disabilities. Review, CyberPsychology & Behavior,
8(3), 272-282.
Stevens, J, A., Corso, P. S,. Finkelstein, E, A., & Miller, T, R. (2006). The
costs of fatal and non-fatal falls among older adults. Injury Prevention, 12(5),
290295.
Timmermans, A., Saini, P., Willmann, R. D., Lanfermann, G., teVrugt, J.,
& Winter, S. (2007). Home Stroke Rehabilitation for the Upper Limbs. 29th
Annual International Conference of the IEEE Engineering in Medicine and
Biology Society, 4015-4018.
Webb.J & Ashley.J. (2012) Beginning Kinect Programming with the
Microsoft Kinect SDK. Apress, New York.
Wen, C. P., Wai, J. P., Tsai, M. K., Yang. Y. C., Cheng, Y. D., Lee, M.
C., Chan, H. T., Tsao, C. K., Tsai, S. P., & Wu, X. (2011) Minimum amount
of physical activity for reduced mortality and extended life expectancy: a
prospective cohort study, Lancet, 1244-1253.
Wiederhold, B. K. (2006). Evaluation of virtual reality therapy in
augmenting the physical and cognitive rehabilitation of war veterans.
Proceedings. 6th Disability, Virtual Reality & Assoc. Tech. 91-96 , Esbjerg,
Denmark.
Wikipedia BBS (2013). Berg Balance Scale. Retrieved April, 23, 2013,
from http://en.wikipedia.org/wiki/Berg_Balance_Scale.
World Population Ageing, (2009). Population Division of the
Department of Economic and Social Affairs of the United Nations.Population
Division of the Department of Economic and Social Affairs of the United
54

Nations. Retrieved at Feb 20, 2013, from

http://www.un.org/esa/population/publications/WPA2009/WPA2009_Working
Paper.pdf.
Tanaka, K., Parker, J, R., Baradoy, G., Sheehan, D., Holash, J, R., & Katz,
L. (2012). A Comparison of Exergaming Interfaces for Use in Rehabilitation
Programs and Research. Loading, 69-81.
WHO. (2013). Children and Young People with Disabilities Fact Sheet.
Retrieved at May 20, 2013, from
www.unicef.org/disabilities/files/Factsheet_A5__Web_NEW.pdf.
Newquist, & Harvey, P. (1992). Virtual Reality's Commercial Reality.
Computerworld, 93-95.

55

APENDICES
APPENDIX A
FORM OF INITIAL ASSESMENT

56

APPENDIX B
FORM OF FINAL ASSESMENT

57