INTERNATIONAL JOURNAL OF RESEARCH IN TECHNOLOGY (IJRT) ISSN No.

2394-9007
Vol. V, No. I, February 2018 www.ijrtonline.org

Hand Gesture Recognition using Soli & Sixth Sense

Shristi Shreyasi, Elakya R.

Abstract— Today, gesture recognition, a touch free control
which is gaining over the tech world. Having the ability to control
elements of device or environment with meager hand movement
and certain position that is gestures. This project combines the
potential of sixth sense - an augmented reality gesture instance
based wearable device and Soli - a Google microchip which has
purpose built interaction that uses radar for motion tracking of
human hand. One of the major disadvantages of Sixth sense tech
is the camera acting as sensor for hand gestures which limits it
actual potential but with Soli embedded as a sensor on the
circuit, the practical power that they will have will have large
dimension. The device that this project proposes will be lot more
smooth and reliable rather than prototypes which depend on
poor sensor thus resulting in poor performance. Soli, the newest
sensor in market is much simpler in gesture approaches and has
leveled layer with ability to solve much complex identification
error. This hand gesture environment can be helpful in hospitals,
homes, vehicle and in areas where multitasking and simplicity is
in demand. This project makes recognizing hand gestures more
simple and the gives the augment reality of sixth sense a new level
of performance and much better user experience.
Keywords: Project Soli, Sixth Sense Technology, Augmented
Reality, Soli Sensor, Gesture Recognition.
I. INTRODUCTION
The Sixth Sense device made by Pranav Mistry has been listed
under the 2009 Young Innovators under 35[1]. The concept of
virtual world colliding with the real world was marked as the
future of technology [2]. The underlying tech principles in
Sixth Sense are effective for the augment reality wearable
device's acceptance and prototype model in real life. But, one
of the biggest disadvantages was its camera which was acting
as a sensor for recognition critical gesture. The camera
captures images as well as track the movement and gesture of
the user's hand. It acts as digital eye which sees the world and
processes it. The camera sole task is not only recognizing
hand gestures but also to perform an action of image capture
when it is directed to do so. The information taken by the
camera is processed by a smart phone which analyses it to
know what the user wants. Other than camera, the markers and
the smart device interpreting the information makes three
disadvantages in total.
Manuscript received on February, 2018.
Shristi Shreyasi, Research Scholar, Department of Computer Science &
Engineering, SRM Institute of Science & Technology, Ramapuram, Chennai,
Tamil Nadu, India.
Prof. Elakya R., Asst. Professor, Department of Computer Science &
Engineering, SRM Institute of Science & Technology, Ramapuram, Chennai,
Tamil Nadu, India.
Soli, a microchip has the entire sensor and the antenna array in
an ultra compact 8mm X 10mm dimension. Soli is an
amalgamation of sensor and processor. The combined effort of
Soli with Sixth Sense Technology will create a much better
wearable interactive device.
II. RELATED WORK
The paper 'Sensing the Sixth Sense' [3] includes three major
technology that is showcased by the sixth sense technology
which are augmented reality, hand gestures and computer
vision. This technology when executed by the existing
prototype of the sixth sense tech showcases a much wider
scope of the device in real life applications. Point of
betterment of the prototype is fusing the sensor, the markers
which act as identifiers and the processing function into a
single module. A single unit which can consist all the three
elements makes the existing prototype much more mobile and
wearable. The delicate structure of the device is also
transformed into a much sturdy architecture. Another paper
'Sixth Sense Technology' discusses the incorporation of
speech IC [4] or voice mode of the sixth sense technology. It
works on speech conversion to commands rather than having a
gesture recognition framework. Gesture recognition is most
pressed on need of today's technology as from mouse click ,
the tech transitioned into touch click and now the future lies in
imaginative click i.e. hand gesture for a click.
A. Sixth Sense Technology:
Sixth Sense is a wearable gestural interface that augments the
physical world around us with digital information and lets us
use natural hand gestures to interact with that information. The
Sixth Sense technology contains a pocket projector, a mirror
and a camera contained in a pendant-like, wearable device.
Both the projector the camera and sensors are connected to a
coding device in the user’s pocket.
The projector projects visual information enabling
surfaces, walls and physical objects around us to be used as
interfaces; while the camera recognizes and tracks users' hand
gestures and physical objects using computer-vision based
techniques. The software program processes the video stream
data captured by the camera and tracks the locations of the
colored markers at the tips of the user’s fingers. The
movements and arrangements of this pattern are interpreted
into gestures that act as interaction instructions for the
projected application interface.

Impact Factor: 4.012 58

Published under
Asian Research & Training Publication
ISO 9001:2015 Certified
INTERNATIONAL JOURNAL OF RESEARCH IN TECHNOLOGY (IJRT) ISSN No. 2394-9007
Vol. V, No. I, February 2018 www.ijrtonline.org
processor and identifier for the model. This enables us to
modify the sixth sense technology to much compact and
mobile device.
CAMERA A. System Architecture of the Model:
MICROPHONE

PROJECTOR
MIRROR

COLOURED MARKERS

MOBILE COMPUTING DEVICE

Fig. 1: Sixth Sense Technology

B. Components of the Device:

The hardware components are coupled in a pendant like
mobile wearable device. Fig. 2: Camera, computing device and the identifiers are replaced by
 Camera single microchip
 Projector As shown in the figure above the camera, computing device
 Mirror and the identifiers are replaced by single microchip. The
 A Mobile Computing Device model proposes portability and wearable context in much
 Colour Markers mobile and compact way. Soli has in built radar and antenna
 Microphone system which perform detection i.e. identification of gesture
A webcam captures and recognizes an object in view and and processing the gesture at a single stage.
tracks the user’s hand gestures using computer-vision based
B. Hardware Architecture of SOLI:
techniques. It sends the data to the smart phone. The camera, in
a sense, acts as a digital eye, seeing what the user sees. It also
tracks the movements of the thumbs and index fingers of both
of the user's hands. It is at the tip of the user’s fingers. Marking
the user’s fingers with red, yellow, green, and blue tape helps
the webcam recognize gestures. The movements and
arrangements of these makers are interpreted into gestures
that act as interaction instructions for the projected
application interfaces. The camera recognizes objects around
you instantly, with the micro projector overlaying the
information on any surface, including the object itself or your
hand. The projector projects visual information enabling
surfaces, walls and physical objects around us to be used as
interfaces. The microphone in the dive is attached in such a
way that it captures the sound signals of user touching the Fig. 3: Hardware Architecture of SOLI
paper. This data is passed to computing device for processing. The 8mmX10mm microchip has radar, antenna and the
Later, combined with the tracking information about user’s gesture identifying processing ability in its compact size. [3]
finger, the system is able to identify precise touch events on the
paper. So, the sound signal captured by the microphone C. Modules of the Proposed System:
provides time information whereas the camera performs The modules of the project are as follows:
1. Hand Gestures
tracking.
2. Interpreting the gestures
III. OVERVIEW OF THE PROPOSED DESIGN 3. Computation of Actions
4. Execution of Action via camera and projector
The proposed model incorporates soli microchip in place of
camera acting as a hand gesture sensor, microphone tracking 1. Hand Gestures:
the touch function of the user hand and a smart device which Soli has in built radar and antenna system due to which the
acts as a computing device. Here, the soli microchip device [5] three components of sixth sense technology are combined into
acts combines the function of above three and acts as a sensor, one compact module. The camera - acting as a sensor

Impact Factor: 4.012 59

Published under
Asian Research & Training Publication
ISO 9001:2015 Certified
INTERNATIONAL JOURNAL OF RESEARCH IN TECHNOLOGY (IJRT) ISSN No. 2394-9007
Vol. V, No. I, February 2018 www.ijrtonline.org
,detecting the users hand gestures, microphone - which tracks
users touch on a surface , colour marker- which is used to
identify the motion of free hand. All these components are
amalgamated into one microchip of Soli. The concept of
Virtual Tools is key to Soli interactions: Virtual Tools [5] are
gestures that mimic familiar interactions with physical tools.
This metaphor makes it easier to communicate, learn, and
remember Soli interactions. Soli uses virtual hand gestures
like the following:
Fig. 8: Searching a song from Soli enabled device

Fig. 4: Imagine a virtual button and clicking it

Fig. 9: Adjusting volume of a Soli enabled device

Fig. 5: Imagine a virtual slider using index, thumb

Fig. 10: Searching a location on Soli enabled watch

2. Interpreting the Gestures:

Fig. 6: Imagine a surface in your hands then dragging your thumb over
your folded finger performs the gestures of touching a reading material
page on a device.
Using them as shown:
Soli uses radar concept to detect a particular type of motion
and interaction between two surfaces [5]. A radar system has
a transmitter that emits radio waves called radar signals in
predetermined directions. When these come into contact with
an object they are usually reflected or scattered in many
directions. Radar is an object-detection system that uses radio
waves to determine the range, angle, or velocity of objects. A
radar system consists of a transmitter producing
electromagnetic waves in the radio or microwaves domain, a
transmitting antenna, a receiving antenna (often the same
antenna is used for transmitting and receiving) and
a receiver and processor to determine properties of the
object(s). Radio waves (pulsed or continuous) from the
transmitter reflect off the object and return to the receiver,
giving information about the object's location and speed.

Fig. 7: Setting clock in a Soli incorporated watch

Impact Factor: 4.012 60

Published under
Asian Research & Training Publication
ISO 9001:2015 Certified
INTERNATIONAL JOURNAL OF RESEARCH IN TECHNOLOGY (IJRT) ISSN No. 2394-9007
Vol. V, No. I, February 2018 www.ijrtonline.org
This device can be connected to an online network via
computer or smart device and the gestures recognised by the
above shown compact sensor can be linked to performing the
action that a user want. For this project, other than soli's
virtual gestures much more gesture must be introduced to
sense the motion of user's hand when the projector makes a
surface act as a touch screen.

4. Execution of Action via camera and projector:

The actions of the hand gestures are implemented by camera
Fig. 11: Radar of Soli which takes gesture as input such as capturing a photo and the augment reality features of
the project is implemented by projector on any surface or
3. Computation of Actions: screen but gestures will not be like sixth sense which is shown
The Soli chip incorporates the entire sensor and antenna array as follows but the function of projector is same.[4] The
into an ultra-compact 8mm x 10mm package. Soli sensor projector project a map, news or any data on real life object
technology works by emitting electromagnetic waves in a surface and constantly projects the real time dynamic
broad beam. Objects within the beam scatter this energy, information of the user's choice of subject for information.
reflecting some portion back towards the radar antenna. The display will be much like sixth sense technology but with
Properties of the reflected signal, such as energy, time delay, computation of soli.
and frequency shift capture rich information about the object’s
characteristics and dynamics, including size, shape,
orientation, material, distance, and velocity. Soli tracks and
recognizes dynamic gestures expressed by fine motions of the
fingers and hand. In order to accomplish this with a single
chip sensor, we developed a novel radar sensing paradigm
with tailored hardware, software, and algorithms. Unlike
traditional radar sensors, Soli does not require large bandwidth
and high spatial resolution; in fact, Soli’s spatial resolution is
coarser than the scale of most fine finger gestures.[5] Instead,
our fundamental sensing principles rely on motion resolution
by extracting subtle changes in the received signal over time. Fig. 14: Painting on a surface using Soli to detect our hand motion. For
By processing these temporal signal variations, Soli can this soli's gesture library has to be modified as there is not virtual gesture
for task such as painting in Soli's virtual tool
distinguish complex finger movements and deforming hand
shapes within its field.

Fig. 15: Augmented reality of retrieving the information of flight status

Fig. 12: Soli's Block Diagram by Soli’s hand gesture, online network and projector

Fig. 13: Soli microchip device Fig. 16: Checking out virtual watch time and alarms

Impact Factor: 4.012 61

Published under
Asian Research & Training Publication
ISO 9001:2015 Certified
INTERNATIONAL JOURNAL OF RESEARCH IN TECHNOLOGY (IJRT)
Vol. V, No. I, February 2018
Fig. 17: Viewing ones clicked pictures on a surface, swiping and adjusting
or editing them
Fig. 18: Viewing the Google maps anywhere, any time.
IV. CONCLUSION
Gesture recognition and augmented reality are among the
emerging technology. The transition from hardware based
operation to touch based operation was quick because man is
always finding ways to ease our daily life tasks and making
them more effective and easy to understand. When access to
daily tasks simplifies along with incorporation of information
and data regarding a task then life becomes time efficient,
healthy, safe and intelligent. The amount of data present on a
single object on network if used wisely can result into smart
Impact Factor: 4.012
Published under
Asian Research & Training Publication
ISO 9001:2015 Certified
ISSN No. 2394-9007
www.ijrtonline.org
living. There is seamless amount of data or information that
may help us to make general decisions more quick and
effective. Hand gesture leads to smart living. After touch
technology, hand gestures unlock the future. Each and every
device that we use can be incorporated with hand gesture thus
making itself a smart device and our lives smart lives. And
gesture recognition using soli is the new miniaturization that
we need. The thin line between real and digital is further
diminished by this technology. Future of technology lies in
gestures to park automatic cars, unlock doors, control
electrical devices, and perform online activities anywhere
anytime. After a smart device, soli incorporated sixth sense is
what we will need.
REFERENCES
[1] “2009 Young Innovators under 35: Pranav Mistry, 28”.
Technology Review. 2009. Retrieved August 14, 2011.
[2] “Cyborg: Digital Destiny and Human Possibility in the Age of
the Wearable Computer”, Steve Mann with Hal Niedzviecki,
ISBN 0385658257 (Hardcover), Random House Inc, 304 pages,
2001.
[3] Aastha, Rashmi, Sakshi Bhatia and Geeta Rani - Sensing the
sixth sense technology in International Journal of Information
Technology and Knowledge Management January-June 2012,
Volume 5, No. 1.
[4] S. Sadhana Rao - Sixth Sense Technology in Proceedings of the
International Conference on Communication and Computational
Intelligence – 2010, Kongu Engineering College, Perundurai,
Erode, T.N., India.27 – 29 December, 2010.
[5] Google ATAP -https://atap.google.com/soli/.
62