2015 International Conference on Soft Computing Techniques and Implementations- (ICSCTI)
Department of ECE, FET, MRIU, Faridabad, India, Oct 8-10, 2015
Vision based Computer Mouse Control using Hand
Gestures
Sandeep Thakur M.E. Scholar Rajesh Mehra Associate Professor
sandeepthakur27@yahoo.com rajeshmehra@yahoo.com
Department of Electronics & Communication Engineering
National Institute of Technical Teachers Training & Research Chandigarh, UT, India
Abstract This paper delineates a vision based interface for
regulating a computer mouse via 2D hand gestures. The
evolution of Human Computer Interaction (HCI) has diverted
the interest of researchers towards natural interaction techniques
in recent years. Numerous applications of real time hand gesture
based recognition in the real world have been deployed where we
interact with computers. Hand gestures rely upon camera based
color detection technique. This method mainly focuses on the use
of a Web Camera to develop a virtual HCI device in a cost
effective manner. This paper proposes a vision based system to
control various mouse activities such as left and right clicking
using hand gestures to make the interaction more efficient and
reliable.
Keywords Human Computer Interaction (HCI), Vision based
system, Centroid, Hand gesture, Graphical User Interface (GUI)
I. INTRODUCTION
In this high tech world, its almost impossible to imagine
life without computers. The invention of computers is one of
the greatest humankind's inventions. Computers have become
an essential part of almost every day use for individuals of
every age. In daily life, we interact many times with computers
to make our work easier. Thus, Human Computer Interaction
(HCI) has become a hot topic for research.
In our daily life, vision and gestures are important
approaches for communication among human beings and the
same role is played by mouse in Graphical User Interface
(GUI) based computers. So, a combined methodology can be
used to make a better interactive system for HCI developing a
technology in the field of digital signal processing,
thus making it more attractive and efficient [1]. An image
signal has a lot of information which can be utilized to control
the different functions of computer. Hence, hand gesture can
play a crucial role in this domain.
In this paper, a vision based approach is presented
for detecting hand gestures and executing varied functions such
as left and right clicking which are usually performed by a
computer mouse. Here, a web camera is employed to collect
the images of hand gestures and then further used for tracking
purpose. These images are processed in MATLAB software
978-1-4673-6792-9/15/$31.002015 IEEE
Buddhi Prakash M.E. Scholar
coolbp.buddhiprakash@gmail.com
(version R2013a from Mathworks, USA) to find out centroid
of each image. Hand movement also moves the centroid thus
making it the principle of sensing for the alteration of cursor on
computer screen.
The hand image is treated here as the mother image. The
left and right click functions of a mouse are implemented by
folding the first and middle fingers of hand respectively and
develop a baby image. So, by comparing the length of fingers
in baby images with those in mother image gives an idea about
the functionality performed by the hand gesture. When the
length of finger crossovers the threshold length in baby image,
it executes a clicking operation. Here, the efficiency of tracking
the hand is improved by using red and blue colored caps on the
fingers to make centroid looking more prominent.
This paper takes off with a brief introduction and few
insights into the HCI system. In next section, video
acquisition and region extraction are explained in detail with
real time results. Finally, conclusion is highlighted.
II. HCI SYSTEM
In early stage of HCI development, mechanical devices
have been used to recover the information of the hand gesture
[4] for example: data glove devices [5,6]. However, the
development of digital signal processors has enhanced a lot in
this recent decade offering a better performance in the field of
image processing. This creates a space for more natural vision-
based technology to work with hand gestures [7]. So, many
researchers have started their work in the field of visual mouse
but with different approaches. Mr. Erdem et al. developed an
approach in which he controlled the motion of mouse by
tracking the finger tips and a click occurred when users hand
passed over the well defined region on the screen [8]. Apart
from this, Chu-Feng Lien controlled the mouse cursor and its
click by using finger-tips only [9]. His approach was based on
image density. Paul et al. used another approach to control
mouse cursor by special hand signs and thumb motion for
clicking purpose [10]. With a new approach, Marcel and his
colleagues tracked the skin-color blobs of the user hand to
Track 2 : Communications - 85
recognize the hand gestures which was baseed on Input-Output
Hidden Markov Models and controlled thee mouse activities
[13].
In this paper, work has been started with an idea of
implementing mouse activities by feeding saame hand gestures
through web cam as are made with hand on real hardware
mouse. So, the movement of cursor on desktoop is controlled by
hand motion and clicking is performed by folding
f the finger.
To increase the efficiency of system, color caps are used on
fingers to recognize the hand gestures.
ON
III. VIDEO ACQUISITIO
In this section, the video acquisition proccedure is explained
which is used for developing the vision baased virtual mouse
system. The different steps of the system deesign are described
through the following steps:
A. Real-time Video Acquisition
An interactive system always comprises of sensors which
provides real time inputs to the system. Heere, a web-cam is
used to capture the real time video input at fixxed frame rate and
resolution. Then image frames are extracted from video and
each frame is processed. Each frame inn the database is
represented as (mn) matrix of defined resoolution where each
element consists of (13) matrix of red,, green and blue
channels each. These elements are termed as pixels [14,15,16].

Fig. 1. RGB Image

B. Flipping of Individual Video Frames

After real-time video acquisition when video is previewed,
it is observed that video is horizontally inverted. It means that
if we move our hand toward left, the image of o the hand moves
towards right and vice-versa. So, it is requirred to horizontally
flip the image which is done in MATLA AB using flipping
function. Fig. 2. Flowchart of video acqquisition and region extraction system

C. Extraction of Red and Blue component

A flipped image frame conntains all colors in it. To control
cursor, it requires detecting reed and blue color from flipped
image. So subtraction method is used in which firstly a gray
scale image is generated from m flipped image and then it is
subtracted from red band image i and blue band image
individually. In result, the redd and blue components of the
image in grayscale color modell have been obtained.
Fig. 3. Flipped Image

Track 2 : Communications - 86

Fig. 4. Grayscale Image
Fig. 5. Red component Image
Fig. 6. Blue component Image
IV. REGION EXTRACTION
A. Filtering
In the detected red and blue components of the images,
some pixels are scattered which creates salt and pepper like
noise. Therefore, a median filter is applied to remove the noise
[14].
Fig. 7. Red and Blue filtered Images
B. Binary Image Generation
In last section, we obtained a gray scale image but in
MATLAB, binary images for finding the region of red and
blue color to use different property functions on the objects
are required. Therefore, gray to binary image conversion is
required here.
A pixel of grayscale image have the value between the
range of 0 to 255 where 0 and 255 signifies the pure
black and white color respectively. So, to convert a grayscale
image into binary image, we have to fix a threshold level
which will divide 0 to 255 levels into two levels only. In our
work, we use 18% as a threshold level which means that all
the pixels having the value greater than 18% of the maximum
pixel value are converted into white color that is 1 and rest of
pixels are converted into black that is 0. Thus, we obtained a
monochromatic image consisting of only black and white
colors.
Fig. 8. Red and Blue binary Images
C. Removal of small objects
During real time video acquisition, sometimes unwanted
red and blue color objects also come in video from the
background. In binary image, these unwanted objects form
small connected pixel components along with the main object.
These small objects create problem in the detection of red and
blue colors of fingers cap. Thence, we remove all connected
components (unwanted objects) from the binary image that
have fewer than specified pixels, producing another binary
image. In Fig. 9. small white object regions have been
removed.
Fig. 9. Small Red and Blue objects removed Image
D. Centroid Detection
Now, the image is ready with main object from the last
section but still the mapping of object with cursor is pending.
So, the properties of the object which can establish a
relationship between object and cursor are required.
Therefore, firstly the labeled matrix is found in which
labeling of all objects take place then several properties for
these objects are calculated. In this work, calculations are
done for the centroid of red object and major-axis length for
the red and blue object region. Mathematically, they can be
calculated by segmenting the colored object from the image
Track 2 : Communications - 87
and it is approximated as an ellipse. Now centroid of the
object is given by following the equation [3]:
 (1)
where xi and yi are x and y coordinates of ith pixel in the
object region and k denotes the number of pixels in the object
region.
Fig. 10. Centroid and Boundary box
In MATLAB, an inbuilt function is applied on red and blue
object region individually available in the binary image which
in turn surrounds the main object by a rectangular bounding
box as the output for the current frame. The center of red color
bounding box provides the centroid of the red detected colored
object. Thus, the coordinates of centroid x (horizontal) and y
(vertical) for red object region are mapped with coordinates of
cursor. MATLAB do not have any inbuilt function to access
the mouse driver but it can be implemented by integrating the
JAVA high level language along with MATLAB. In this way,
the red capped finger is tracked by the system for each frame
and movement of cursor is controlled.
Fig. 11. Resultant RGB Image with centroid and boundary box
E. Mouse Left and Right Click
In this work, blue color is employed for left click and red
color for right click. At initial stage, hand is opened fully so
that fingers can be expanded to its maximum length and this
can calculate its length during the acquisition of first frame
which treated as mother image. Now a threshold length is
fixed for clicking purpose which is selected to 60% here and
whenever a left click is required, blue color finger is folded
which will be detected in the image and treated as baby image.
If the length of finger in baby image reduces as compared to
the threshold length, it will execute a left click. Similarly,
right click is implemented for red color finger. Here, major-
axis length property is used for red and blue color object in
MATLAB to calculate the length of finger.
V. CONCLUSION
In this paper, a vision based virtual mouse is designed to
track the hand gestures using colors caps on the fingers. This
system has been developed in MATLAB environment. This
vision based technology overrules the touch technology as it is
less expensive. In this technology we use a digital camera
which is already embedded in electronic devices. After this
study, we believe that this technology has great future in HCI
based systems. It can be widely used in the fields of robotics
[11,12], biomedical instrumentation, computer gaming and
many more.
REFERENCES
[1] Abhik Banerjee, Abhirup Ghosh, Koustuvmoni Bharadwaj, Hemanta
Saikia, Mouse control using a web camera based on color detection,
International Journal of Computer Trends and Technology , Volume 9
number 1, pp. 15-20, March -2014.
[2] Sugreev Kaur, Rajesh Mehra, High speed and area efficient 2D DWT
processor based image compression, Signal & Image Processing : An
International Journal, Vol.1, No.2, pp. 23-31, December- 2010.
[3] Sandeep Konam, Sai Maheswara Reddy, A novel approach towards
optimization of vision based virtual mouse interface, International
Journal of Engineering Research and Technology, Vol. 3 Issue 3,
pp. 1126-1131, March -2014.
[4] P. Kortum, HCI beyond the GUI: Design for Haptic, Speech, Olfactory,
and other nontraditional interfaces, Morgan Kauf-mann Publishers, pp.
75-106, 2008.
[5] J.Weissmann and R. Salomon, Gesture recognition for virtual reality
applications using data gloves and neural networks, International
Conference on Neural Networks, Vol 3. pp. 2043-2046, 1999.
[6] T. G. Zimmerman, J. Lanier, C. Blanchard, S. Bryson, and Y. Harvill,
A hand gesture interface device, CHl+GI, pp. 189-192, 1987.
[7] Yee Yong Pang, Nor Azman Ismail, Phuah Leong Siang Gilbert, A real
time vision-based hand gesture interaction, Fourth Asia International
Conference on Mathematical/Analytical Modelling and Computer
Simulation, pp. 237-242, 2010.
[8] A. Erdem, E. Yardimci, Y. Atalay, V. Cetin, A. E. Computer vision
based mouse, Acoustics, Speech, and Signal Processing, Proceedings.
(ICASS). IEEE International Conference, 2002
[9] Chu-Feng Lien, Portable vision-based HCI A real-time hand
mouse system on handheld devices, National Taiwan University,
Computer Science and Information Engineering Department.
[10] Hojoon Park, A method for controlling the mouse movement using
a real time camera, Brown University, Providence, RI, USA,
Department of computer science, 2008.
[11] Abhishek Acharya, Rajesh Mehra, Vikram Singh Takher, FPGA based
non uniform illumination correction in image processing application,
IJCTA, Vol-2(2), pp. 349-358, January-2011.
[12] Bhawna Tiwari, Rajesh MehraDesign & implementation of Reed Solo-
mon Decoder for 802.16 network using FPGA, IEEE International
Conf.on Signal Processing, Computing & Control, pp. 1-5, March-2012.
Track 2 : Communications - 88
[13] S. Marcel, O. Bernier, J. E. Viallet, and D. Collobert, Hand gesture
recognition using Input-Output Hidden Markov Models, Proc. of the
FG2000 Conference on Automatic Face and Gesture Recognition,
2000.
[14] Abhinaba Bhattacharjee, Indrani Jana, Ankana Das, Debasish Kundu, Mr. Buddhi Prakash Sharma: Mr. Sharma is
Sudipta Ghosh, Sauvik Das Gupta, A novel probabilistic approach of currently working in Electronics & Communication
colored object detection and design of a Gesture based real-time mouse
tracking along with virtual teaching intended for color-blind people,
Engineering Department of B. K. Birla Institute of
2nd International Conference on Signal Proces- sing and Integrated Engineering & Technology, Pilani, Rajasthan, India since
Networks, pp 512-519, 2015. 2014. He is currently pursuing M.E from National Institute of
Technical Teachers Training and Research, Chandigarh India.
[15] Buddhi Prakash Sharma, Rajesh Rana, Rajesh Mehra Face recognition
He has completed his B. Tech from ICFAI University,
using Gabor Wavelet for image processing applications, Proc. of Int.
Conf. on Emerging Trends in Engineering and Technology, pp 201-206 , Dehradoon, India. Mr. Sharma is having six years of teaching
2013 experience. He has more than eight papers in his credit which
are published in refereed International Journals and
[16] Payal Aggarwal and Rajesh Mehra High speed CT image
Conferences. His research areas are Advanced Digital Signal
reconstruction using FPGA (IJCA), Vol.22 No.4, pp 7-10 , May 2011
Processing, VLSI Design and Image Processing. Mr. Sharma
is member of IEEE.

Er. Sandeep Thakur: Er. Sandeep Thakur is

presently working as Assistant Professor in Electronics &
Communication Engineering Department at Atal Bihari
Vajpayee Government Institute of Engineering and
Technology, Pragatinagar, Shimla.(H.P.) since 2013. He is
pursuing his ME from ECE Department NITTTR,
Chandigarh. He has received his Bachelor of Technology from
Sant Longowal Institute of Engineering and Technology,
Sangrur, Punjab in 2011. Er. Sandeep Thakur has 3 years of
academic experience. His research areas are Advanced Digital
Signal Processing, Image Processing & VLSI Design.

Dr. Rajesh Mehra: Dr. Mehra is currently

associated with Electronics and Communication Engineering
Department of National Institute of Technical Teachers
Training & Research, Chandigarh, India since 1996. He has
received his Doctor of Philosophy in Engineering and
Technology from Punjab University, Chandigarh, India in
2015. Dr. Mehra received his Master of Engineering from
Panjab Univeristy, Chandigarh, India in 2008 and Bachelor of
Technology from NIT, Jalandhar, India in 1994. Dr. Mehra
has 20 years of academic and industry experience. He has
more than 250 papers in his credit which are published in
refereed International Journals and Conferences. Dr. Mehra
has 55 ME thesis in his credit. He has also authored one book
on PLC & SCADA. His research areas are Advanced Digital
Signal Processing, VLSI Design, FPGA System Design,
Embedded System Design, and Wireless & mobile
Communication. Dr. Mehra is member of IEEE and ISTE.

Track 2 : Communications - 89