CHAPTER-1

INTRODUCTION

Weve evolved over millions oI years to sense the world around us. When we
encounter something, someone or some place, we use our Iive natural senses which
includes eye, ear, nose, tongue mind and body to perceive inIormation about it;
that inIormation helps us make decisions and chose the right actions to take. But
arguably the most useIul inIormation that can help us make the right decision is not
naturally perceivable with our Iive senses, namely the data, inIormation and
knowledge that mankind has accumulated about everything and which is
increasingly all available online.

Although the miniaturization oI computing devices allows us to carry
computers in our pockets, keeping us continually connected to the digital world,
there is no link between our digital devices and our interactions with the physical
world. InIormation is conIined traditionally on paper or digitally on a screen. Sixth
Sense bridges this gap, bringing intangible, digital inIormation out into the tangible
world, and allowing us to interact with this inIormation via natural hand gestures.
'Sixth Sense Irees inIormation Irom its conIines by seamlessly integrating it with
reality, and thus making the entire world your computer.

'Sixth Sense Technology it is the newest jargon that has proclaimed its
presence in the technical arena. This technology has emerged, which has its
relation to the power oI these six senses. Our ordinary computers will soon be able
to sense the diIIerent Ieelings accumulated in the surroundings and it is all a giIt oI
the 'Sixth Sense Technology newly introduced.

Sixth Sense is a wearable gesture based device that augments the physical
world with digital inIormation and lets people use natural hand gestures to interact
with that inIormation. It was developed by Pranav Mistry, a PhD student in the
Fluid InterIaces Group at the MIT Media Lab. A grad student with the Fluid
InterIaces Group at MIT, he caused a storm with his creation oI Sixth Sense. He
says that the movies Robocop and Minority Report gave him the
inspiration to create his view oI a world not dominated by computers, digital
inIormation and human robots, but one where computers and other digital devices
enhance peoples enjoyment oI the physical world. Right now, we use our
devices (computers, mobile phones, tablets, etc.) to go into the internet and get
inIormation that we want. With Sixth Sense we will use a device no bigger than
current cell phones and probably eventually as small as a button on our shirts to
bring the internet to us in order to interact with our world. Sixth Sense will allow
us to interact with our world like never beIore. We can get inIormation on anything
we want Irom anywhere within a Iew moments! We will not only be able to
interact with things on a whole new level but also with people! One great part oI
the device is its ability to scan objects or even people and project out inIormation
regarding what you are looking at.

Researchers at the Massachusetts Institute of Technology (MIT) have taken it

upon themselves to create a low-cost high-tech device that will oIIer users the
possibility to interact with each object they come across in real-liIe. That is to say,
when picking up a box oI cereals, inIormation could be projected on your hand or
on a piece oI paper that will let you know iI the brand is good, or what type oI
ingredients the product contains. Every Ilat surIace you Iind, including walls, table
tops and other such things, may become an instant touchscreen, Irom which you
could connect to the Internet and get valuable inIormation. The system is not
surprisingly, made up almost entirely oI items that can be bought Irom stores, as
the MIT team has very little proprietary components in the new sixth sense device.
Basically, the system consists oI a good web camera, a battery-powered projector,
as well as a mobile phone, preIerably with access to the Internet. This basic set-up
will allow users to perIorm such operations as to check Ior the time by drawing a
circle on their wrist, where the projector will display it. "Other than letting some oI
you live out your Iantasy , it can really let you connect as a sixth sense device with
whatever is in Iront oI you. You can use any surIace, including your hand iI
nothing else is available, and interact with the data. It is very much a work in
progress. Maybe in ten years we will be here with the ultimate sixth-sense brain
implant," Patty Maes, a researcher at MIT and a member oI the project
development team, explains. The project idea is oI Pranav Mistry, research
proIessor at MIT Media Lab and Patty Maes, reseracher at MIT. The team
estimates that the basic price Ior the sixth sense device could be as little as $350,
even though it's so high-tech that a person could wear it as jewelry.

CHAPTER- 2


SIXTH SENSE TECHNOLOGY

What is Sixth Sense ? Sixth Sense in scientiIic (or non-scientiIic) terms is
deIined as Extra Sensory Perception or in short ESP. It involves the reception oI
inIormation not gained through any oI the Iive senses. Nor is it taken Irom any
experiences Irom the past or known. Sixth Sense aims to more seamlessly integrate
online inIormation and tech into everyday liIe. By making available inIormation
needed Ior decision-making beyond what we have access to with our Iive senses, it
eIIectively gives users a sixth sense. Earlier Sixth Sense Prototype


Maes MIT group, which includes seven graduate students, were thinking about
how a person could be more integrated into the world around them and access
inIormation without having to do something like take out a phone. They initially
produced a wristband that would read an Radio Frequency IdentiIication tag to
know, Ior example, which book a user is holding in a store. They also had a ring
that used inIrared to communicate by beacon to supermarket smart shelves to give
you inIormation about products. As we grab a package oI macaroni, the ring would
glow red or green to tell us iI the product was organic or Iree oI peanut traces
whatever criteria we program into the system. They wanted to make inIormation
more useIul to people in real time with minimal eIIort in a way that doesnt require
any behaviour changes. The wristband was getting close, but we still had to take
out our cell phone to look at the inIormation. Thats when they struck on the idea
oI accessing inIormation Irom the internet and projecting it. So someone wearing
the wristband could pick up a paperback in the bookstore and immediately call up
reviews about the book, projecting them onto a surIace in the store or doing a
keyword search through the book by accessing digitized pages on Amazon or
Google books. They started with a larger projector that was mounted on a helmet.
But that proved cumbersome iI someone was projecting data onto a wall then
turned to speak to Iriend the data would project on the Iriends Iace. Recent
Prototype

Now they have switched to a smaller projector and created the pendant prototype
to be worn around the neck. The Sixth Sense prototype is composed oI a pocket
projector, a mirror and a camera. The hardware components are coupled in a
pendant-like mobile wearable device. Both the projector and the camera are
connected to the mobile computing device in the users pocket. We can very well
consider the Sixth Sense Technology as a blend oI the computer and the cell
phone. It works as the device associated to it is hanged around the neck oI a person
and thus the projection starts by means oI the micro projector attached to the
device. ThereIore, in course, you turn out to be a moving computer in yourselI and
the Iingers act like a mouse and a keyboard. The prototype was built Irom an
ordinary webcam and a battery-powered 3M projector, with an attached mirror
all connected to an internet-enabled mobile phone. The setup, which costs less than
$350, allows the user to project inIormation Irom the phone onto any surIace
walls, the body oI another person or even your hand. Mistry wore the device on a
lanyard around his neck, and colored Magic Marker caps on Iour Iingers (red, blue,
green and yellow) helped the camera distinguish the Iour Iingers and recognize his
hand gestures with soItware that Mistry created.
CHAPTER-3


WORKING OF SIXTH SENSE TECHNOLOGY

Components
The hardware components are coupled in a pendant like mobile wearable device.

Camera
Projector
Mirror
Mobile
Color
Markers
Camera

A webcam captures and recognises an object in view and tracks the users hand
gestures using computer-vision based techniques. It sends the data to the smart
phone. The camera, in a sense, acts as a digital eye, seeing what the user sees. It
also tracks the movements oI the thumbs and index Iingers oI both oI the user's
hands. The camera recognizes objects around you instantly, with the micro-
projector overlaying the inIormation on any surIace, including the object itselI or
your hand.

Projector

Also, a projector opens up interaction and sharing. The project itselI contains a
battery inside, with 3 hours oI battery liIe. The projector projects visual
inIormation enabling surIaces, walls and physical objects around us to be used as
interIaces. We want this thing to merge with the physical world in a real physical
sense. You are touching that object and projecting inIo onto that object. The
inIormation will look like it is part oI the object. A tiny LED projector displays
data sent Irom the smart phone on any surIace in viewobject, wall, or person.

Mirror

The usage oI the mirror is signiIicant as the projector dangles pointing downwards
Irom the neck.

The mobile devices like Smartphone in our pockets transmit and receive voice and
data anywhere and to anyone via the mobile internet. An accompanying
Smartphone runs the Sixth Sense soItware, and handles the connection to the
internet. A Web-enabled smart phone in the users pocket processes the video data.
Other soItware searches the Web and interprets the hand gestures.

Color Markers


It is at the tip oI the users Iingers. Marking the users Iingers with red, yellow,
green, and blue tape helps the webcam recognize gestures. The movements and
arrangements oI these makers are interpreted into gestures that act as interaction
instructions Ior the projected application interIaces.

Working

The hardware that makes Sixth Sense work is a pendant like mobile wearable
interIace. It has a camera, a mirror and a projector and is connected wirelessly to a
Bluetooth or 3G or wiIi smart phone that can slip comIortably into ones pocket.

The camera recognizes individuals, images, pictures, gestures one makes with their
hands InIormation is sent to the Smartphone Ior processing.

The downward-Iacing projector projects the output image on to the mirror.

Mirror reIlects image on to the desired surIace.

Thus, digital inIormation is Ireed Irom its conIines and placed in the physical
world. The entire hardware apparatus is encompassed in a pendant-shaped mobile
wearable device. Basically the camera recognises individuals, images, pictures,
gestures one makes with their hands and the projector assists in projecting any
inIormation on whatever type oI surIace is present in Iront oI the person. The usage
oI the mirror is signiIicant as the projector dangles pointing downwards Irom the
neck. To bring out variations on a much higher plane, in the demo video which was
broadcasted to showcase the prototype to the world, Mistry uses coloured caps on
his Iingers so that it becomes simpler Ior the soItware to diIIerentiate between the
Iingers, demanding various applications. The soItware program analyses the video
data caught by the camera and also tracks down the locations oI the coloured
markers by utilising single computer vision techniques. One can have any number
oI hand gestures and movements as long as they are all reasonably identiIied and
diIIerentiated Ior the system to interpret it, preIerably through unique and varied
Iiducials. This is possible only because the _Sixth Sense device supports multi-
touch and multi-user interaction. MIT basically plans to augment reality with a
pendant picoprojector: hold up an object at the store and the device blasts relevant
inIormation onto it (like environmental stats, Ior instance), which can be browsed
and manipulated with hand gestures. The "sixth sense" in question is the internet,
which naturally supplies the data, and that can be just about anything -- MIT has
shown oII the device projecting inIormation about a person you meet at a party on
that actual person (pictured), projecting Ilight status on a boarding pass, along with
an entire non-contextual interIace Ior reading email or making calls. It's pretty
interesting technology that, like many MIT Media Lab projects, makes the wearer
look like a complete dork -- iI the projector doesn't give it away, the colored Iinger
bands the device uses to detect Iinger motion certainly might. The idea is that Sixth
Sense tries to determine not only what someone is interacting with, but also how he
is potentially relevant to that situation, and then the projector takes over. All the
work is in the soItware," says Dr Maes. "The system is constantly trying to Iigure
out what's around you, and what you're trying to do. It has to recognize the images
you see, track your gestures, and then relate it all to relevant inIormation at the
same time."


The soItware recognizes 3 kinds oI gestures:
Multitouch gestures : like the ones you see in MicrosoIt SurIace or the iPhone --
where you touch the screen and make the map move by pinching and dragging.

Freehand gestures : like when you take a picture |as in the photo above|. Or, you
might have noticed in the demo, because oI my culture, I do a namaste gesture to
start the projection on the wall.

Iconic gestures : drawing an icon in the air. Like, whenever I draw a star, show me
the weather. When I draw a magniIying glass, show me the map. You might want
to use other gestures that you use in everyday liIe. This system is very
customizable. The technology is mainly based on hand gesture recognition, image
capturing, processing, and manipulation, etc. The map application lets the user
navigate a map displayed on a nearby surIace using hand gestures, similar to
gestures supported by multi-touch based systems, letting the user zoom in, zoom
out or pan using intuitive hand movements. The drawing application lets the user
draw on any surIace by tracking the Iingertip movements oI the users index
Iinger.










CHAPTER-4


RELATED TECHNOLOGIES

Sixth Sense technology takes a diIIerent approach to computing and tries to make
the digital aspect oI our lives more intuitive, interactive and, above all, more
natural. We shouldnt have to think about it separately. Its a lot oI complex
technology squeezed into a simple portable device. When we bring in connectivity,
we can get instant, relevant visual inIormation projected on any object we pick up
or interact with The technology is mainly based on hand augmented reality, gesture
recognition, computer vision based algorithm etc.

Augmented reality

Augmented reality (AR) is a term Ior a live direct or indirect view oI a physical
real-world environment whose elements are augmented by virtual computer-
generated imagery. It is related to a more general concept called mediated reality in
which a view oI reality is modiIied (possibly even diminished rather than
augmented) by a computer. The augmentation is conventionally in real-time and in
semantic context with environmental elements.
Sixth sense technology which uses Augmented Reality concept to super imposes
digital inIormation on the physical world. With the help oI advanced AR
technology (e.g. adding computer vision and object recognition) the inIormation
about the surrounding real world oI the user becomes interactive and digitally
usable. ArtiIicial inIormation about the environment and the objects in it can be
stored and retrieved as an inIormation layer on top oI the real world view. The
main hardware components Ior augmented reality are: display, tracking, input
devices, and computer. Combination oI powerIul CPU, camera, accelerometers,
GPS and solid state compass are oIten present in modern Smartphone, which make
them prospective platIorms. There are three major display techniques Ior
Augmented Reality:

Head Mounted Displays

Handheld Displays

Spatial Displays

Head Mounted Displays A Head Mounted Display (HMD) places images oI both
the physical world and registered virtual graphical objects over the user's view oI
the world. The HMD's are either optical see-through or video see-through in
nature.
Handheld Displays Handheld Augment Reality employs a small computing
device with a display that Iits in a user's hand. All handheld AR solutions to date
have employed video see-through techniques to overlay the graphical inIormation
to the physical world. Initially handheld AR employed sensors such as digital
compasses and GPS units Ior its six degree oI Ireedom tracking sensors.
Spatial Displays Instead oI the user wearing or carrying the display such as with
head mounted displays or handheld devices; Spatial Augmented Reality (SAR)
makes use oI digital projectors to display graphical inIormation onto physical
objects. Modern mobile augmented reality systems use one or more oI the
Iollowing tracking technologies: digital cameras and/or other optical sensors,
RFID, wireless sensors etc. Each oI these technologies have diIIerent levels oI
accuracy and precision. Most important is the tracking oI the pose and position oI
the user's head Ior the augmentation oI the user's view. For users with disabilities
oI varying kinds, AR has real potential to help people with a variety oI disabilities.
Only some oI the current and Iuture AR applications make use oI a Smartphone as
a mobile computing platIorm.



Tracking

Modern mobile augmented reality systems use one or more oI the Iollowing
tracking technologies: digital camera and/or other optical sensors, accelerometers
,GPS , gyroscopes, solid state compasses, RFID , wireless sensors. Each oI these
technologies have diIIerent levels oI accuracy and precision. Most important is the
tracking oI the pose and position oI the user's head Ior the augmentation oI the
user's view. The user's hand(s) can tracked or a handheld input device could be
tracked to provide a 6DOF interaction technique. Stationary systems can employ
6DOF track systems such as Polhemus, ViCON, A.R.T, or Ascension.

Input devices
This is a current open research question. Some systems, such as the Tinmith
system, employ pinch glove techniques. Another common technique is a wand
with a button on it. In case oI smartphone, phone itselI could be used as 3D
pointing device, with 3D position oI the phone restored Irom the camera images.

Computer

Camera based systems require powerIul CPU and considerable amount oI RAM
Ior processing camera images. Wearable computing systems employ a laptop in a
backpack conIiguration. For stationary systems a traditional workstation with a
powerIul graphics card. Sound processing hardware could be included in
augmented reality systems.

Software

For consistent merging real-world images Irom camera and virtual 3D images,
virtual images should be attached to real-world locations in visually realistic way.
That means a real world coordinate system, independent Irom the camera, should
be restored Irom camera images. That process is called Image registration and is
part oI Azuma's deIinition oI Augmented Reality.
Augmented reality image registration uses diIIerent methods oI computer vision,
mostly related to video tracking. Many computer vision methods oI augmented
reality are inherited Iorm similar visual odometry methods.
Usually those methods consist oI two parts. First interest points, or Iiduciary
markers, or optical Ilow detected in the camera images. First stage can use Feature
detection methods likeCorner detection, Blob detection, Edge detection or
thresholding and/or other image processing methods.
In the second stage, a real world coordinate system is restored Irom the data
obtained in the Iirst



stage. Some methods assume objects with known 3D geometry (or Iiduciary
markers) present in the scene and make use oI those data. In some oI those cases
all oI the scene 3D structure should be precalculated beIorehand. II not all oI the
scene is known beIorehandSLAM technique could be used Ior mapping Iiduciary
markers/3D models relative positions. II no assumption about 3D geometry oI the
scene made structure Irom motion methods are used. Methods used in the second
stage include projective (epipolar) geometry, bundle adjustment, rotation
representation with exponential map, kalman and particle Iilters. Current
applications Advertising: Marketers started to use AR to promote products via
interactive AR applications. For example, at the 2008 LA Auto Show, Nissan
unveiled the concept vehicle Cube and presented visitors with a brochure which,
when held against a webcam, showed several versions oI the vehicle. In August
2009, Best Buy ran a circular with an augmented reality code that allowed users
with a webcam to interact with the product in 3D. Support with complex tasks:
Complex tasks such as assembly, maintenance, and surgery can be simpliIied by
inserting additional inIormation into the Iield oI view. For example, labels can be
displayed on parts oI a system to clariIy operating instructions Ior a mechanic who
is perIorming maintenance on the system.AR can include images oI hidden objects,
which can be particularly eIIective Ior medical diagnostics or surgery. Examples
include a virtual X-ray view based on prior tomography or on real time images
Irom ultrasound or open NMR devices. A doctor could observe the Ietus inside the
mother's womb. See also Mixed reality. Navigation devices: AR can augment the
eIIectiveness oI navigation devices Ior a variety oI applications. For example,
building navigation can be enhanced Ior the purpose oI maintaining industrial
plants. Outdoor navigation can be augmented Ior military operations or disaster
management. Head-up displays or personal display glasses in automobiles can be
used to provide navigation hints and traIIic inIormation. These types oI displays
can be useIul Ior airplane pilots, too. Head-up displays are currently used in Iighter
jets as one oI the Iirst AR applications. These include Iull interactivity, including
eye pointing.
Industrial Applications: AR can be used to compare the data oI digital mock-ups
with physical mock-ups Ior eIIiciently Iinding discrepancys between the two
sources. It can Iurther be employed to saIeguard digital data in combination with
existing real prototypes, and thus save or minimize the building oI real prototypes
and improve the quality oI the Iinal product.


















CHAPTER-5


APPLICATIONS

The Sixth Sense prototype implements several applications that demonstrate the
useIulness, viability and Ilexibility oI the system.

The Sixth Sense device has a huge number oI applications. The Iollowing are Iew
oI the applications oI Sixth Sense Technology.

Make a call
Call up a map
Check the time
Create multimedia reading experience
Drawing application
Zooming Ieatures
Get product inIormation
Get book inIormation
Get Ilight updates
Feed inIormation on people
Take pictures
Check the email


Make a call



You can use the Sixth Sense to project a keypad onto your hand, then use that
virtual keypad to make a call. Calling a number also will not be a great task with
the introduction oI Sixth Sense Technology. No mobile device will be required,
just type in the number with your palm acting as the virtual keypad. The keys will
come up on the Iingers. The Iingers oI the other hand will then be used to key in
the number and call.

Call up a map

The sixth sense also implements map which lets the user display the map on any
physical surIace and Iind his destination and he can use his thumbs and index
Iingers to navigate the map, Ior example, to zoom in and out and do other controls.






Check the time

Sixth Sense all we have to do is draw a circle on our wrist with our index Iinger to
get a virtual watch that gives us the correct time. The computer tracks the red
marker cap or piece oI tape, recognizes the gesture, and instructs the projector to
Ilash the image oI a watch onto his wrist.

Create multimedia reading experiences

The Sixth Sense system also augments physical objects the user is interacting with
by projecting more inIormation about these objects projected on them. For
example, a newspaper can show live video news or dynamic inIormation can be
provided on a regular piece oI paper. Thus a piece oI paper turns into a video
display.







Drawing application


The drawing application lets the user draw on any surIace by tracking the Iingertip
movements oI the users index Iinger.






Zooming features
The user can zoom in or zoom out using intuitive hand movements.

Get book information

Maes says Sixth Sense uses image recognition or marker technology to recognize
products you pick up, then Ieeds you inIormation on books. The system can project
Amazon ratings on that book, as well as reviews and other relevant inIormation




Take pictures

II we Iashion our index Iingers and thumbs into a square (the typical "Iraming"
gesture), the system will snap a photo. AIter taking the desired number oI photos,
we can project them onto a surIace, and use gestures to sort through the photos,
and organize and resize them.










CHAPTER-6


ADVANTAGES AND ENHANCEMENTS


Advantages

Sixth Sense is an user Iriendly interIace which integrates digital inIormation into
the physical world and its objects, making the entire world your computer.

Sixth Sense does not change human habits but causes computer and other
machines to adapt to human needs.

It uses hand gestures to interact with digital inIormation.

Supports multi-touch and multi-user interaction . Data access directly Irom
machine in real time .It is an open source and cost eIIective and we can mind map
the idea anywhere.

It is gesture-controlled wearable computing device that Ieeds our relevant
inIormation and turns any surIace into an interactive display.

It is portable and easy to carry as we can wear it in our neck. The device could be
used by anyone without even a basic knowledge oI a keyboard or mouse.
There is no need to carry a camera anymore. II we are going Ior a holiday, then
Irom now on wards it will be easy to capture photos by using mere Iingers.


Future Enhancements

To get rid oI color markers. To incorporate camera and projector inside mobile
computing device. Whenever we place pendant- style wearable device on table, it
should allow us to use the table as multi touch user interIace. Applying this
technology in various interest like gaming, education systems etc. To have 3D
gesture tracking. To make sixth sense work as IiIth sense Ior disabled person.




CONCLUSION

The key here is that Sixth Sense recognizes the objects around you, displaying
inIormation automatically and letting you access it in any way you want, in the
simplest way possible. Clearly, this has the potential oI becoming the ultimate
"transparent" user interIace Ior accessing inIormation about everything around us.
II they can get rid oI the colored Iinger caps and it ever goes beyond the initial
development phase, that is. But as it is now, it may change the way we interact
with the real world and truly give everyone complete awareness oI the
environment around us.