AMITY UNIVERSITY RAJASTHAN, JAIPUR

GRAPHIC SKILLS – IV

SUBMITTED TO: -

SUBMITTED BY:

B.Tech (CSE) SEM IV

 CERTIFICATE

Certified that the report entitled HOLOPORTATION submitted by ………………. with

enrollment number …………..on 20/111/2019 is his own work and has been carried out
under my supervision. It is recommended that the candidate may now be evaluated for his
work by the university.

Student:…………………………
Guide: …………………………….

Date: Date:

TABLE OF CONTENTS
 Abstract………………………………………………(1)
 Chapter 1. Concept of Mixed Reality……………... (5)
o Mixed Reality…………………………………… (6)
o Virtual Reality………………………………….. (6)
o Augmented Reality………………………….. (7)
o AR vs. VR vs. MR…………………………….… (9)

 Chapter 2. Holoportation…………...…………...... (10)

o How it works…………..……… (11)
o Uses of holoportation…….…... (12)

 Chapter 3. MICROSOFT HOLOLENS..………...(17)

o What’s powering hololens…….. (18)
o Abilities……………………….. (19)

 Chapter 4. Development of Hololens………...…... (22)

 Chapter 5. Future of Holoportation...…………... (28)

o Communication……………… (29)
o Education……………………. (30)
o Medical Consultation………… (31)
o Mobile Holoportation…………. (33)
o AI Co-processor……………….. (34)
o
 Chapter 6. Problems of Holoportation...………… (35)
Conclusion…………………………………………. (38)
Reference…………………………………………… (39)
Abstract
The thesis is based around a project aimed to enable real time holoportation on human
models. The term holoportation defines 3D holograms, displayed in mixed reality, used for
real time communication between remote users. Implementation of the system was based on
usage of certain hardware components, namely the Microsoft Kinect v2 depth camera and the
Microsoft HoloLens mixed reality glasses. During the development the project was divided
into sections, as each section required various methods to be researched before the optimal
solution could be found. The thesis describes the technological problem set by the task at
hand, the solution that was determined to be the best, as well as the other considered options.
Each phase of the project required research and implementation, and due to many unknown
variables the development process was not concretely defined. The result is a proof of
concept for a holographic communication platform, uniting a number of solutions and
components which are easily expandable to a commercially usable product.

This thesis implements a system for remote, end-to-end, one way, real time holographic
streaming, which includes audio, mesh and texture streams. Using the Unity3D engine as the
base, a system was developed, using the Microsoft Kinect v2 and HoloLens devices for
server to client streaming over the network. This system provides a proof of concept for a
holographic communication software platform.
 Chapter-1

CONCEPT OF MIXED REALITY

Mixed reality (MR) is a user environment in which physical reality and digital content are
combined in a way that enables interaction with and among real-world and virtual objects.
Unlike virtual reality (VR) which immerses the end user in a completely digital environment,
or augmented reality (AR) which layers digital content on top of a physical environment,

mixed reality blends digital and real world settings. Mixed reality is sometimes considered a
type of augmented reality (AR), but its capacity for interactivity between real-world and
digital elements places it further along the virtuality continuum, which has physical reality at
one extreme and immersive virtual reality at the other.
1.1 MIXED REALITY
We are now living in the era of Augmented Reality and Virtual Reality, but there is
something that exists between these two, and that is the mixed reality. It is also referred as
hybrid reality or extended reality (XR). A headset form factor follows the user's gaze and
maps the user's physical surroundings and software then uses deep learning algorithms
to align digital content with specific areas of the map. MR programming allows digital
objects to interact with physical objects and people to interact with digital objects as if they
are physical. The result is that an ordinary desktop can turn into an interactive
computer touch screen or an MR-generated film character can sit on the homeowner's couch.

Although mixed reality is still in the early stages, it is already being used in many industries
for educational purposes. For example, aircraft manufacturers are using MR as a cost-
effective way to train repair technicians. Instead of pulling an engine out of an aircraft to
conduct a training session, technicians wearing special headsets can view a holographic
image of an engine and use gesture, gaze and voice user interface (VUI) commands to
interact with the hologram, changing perspectives and extracting meaningful information,
layer by layer.

1.2VIRTUAL REALITY
VR stands for Virtual Reality which basically means that it is able to draw the user into an
immersive simulation. The real environment can be totally replaced with the sounds and
sights giving you the feel as though you are actually present in the digital world and part of it.

 Features

To enhance the experience and make virtual reality exciting headsets and cameras are often
used to block out the real world. It’s a totally immersive experience allowing you to interact
with a totally different world.

 Platform

VR has already been used to help with training medical staff and military as well as in
gaming to give an experience on a totally different level. More recently virtual reality has
stepped up to the next level offering users a more realistic experience to lose themselves in.
  Headset

By using a headset people are able to walk around their own surroundings yet feel as though
they are in a totally different place to their reality. They can interact with other users in games
and digital characters without the outside world interrupting.

1.3AUGMENTED REALITY

 Features

AR stands for augmented reality which is a way that you can get an indirect view of a real
life scenario or environment in a supplemented way by use of digital technology. A great
example of this would be being an empty room yet being able to use digital technology to see
how it would look in a different colour with different furnishings before actually purchasing
them. Sound graphics and other sensory input can help to make the experience more realistic.

 Platform

Although the digital and other augmented parts of the augmented reality are superimposed in
real life they are not actually part of it. It can be used to interact with the real world to help
make suggestions of improvement such as recipe ideas based on real ingredients laid out
before the camera.

 Headset

A headset is able to enhance the experience further, such as adding characters in a game that
are able to talk and make suggestions, immersing you in part to give you the feel of realism.
Augmented reality can be best described as a real life overlay rather than a thorough digital
and sensory experience that takes you away from it.

 Prospect

We are already seeing augmented reality being used to day in real life situations such as
training. It is also being used in gaming and more recently with mobile technology using GPS
signals to add more realism and accuracy.
1.4AUGMENTED REALITY vs. VIRTUAL RELITY vs. MIXED REALITY

There are some basic difference between augmented reality, virtual reality and mixed reality
which can be understand easily from the table below :-

CHARACTERSTIC VIRTUAL AUGMENED MIXED REALITY

REALITY REALITY (MR)
(VR) (AR)
Is user aware about
the real world? NO YES YES

Can user interact

with the real and NO YES YES
virtual world in the
real time?
Can real and virtual
interact with each NO NO YES
other in the real
time?
 Chapter-2

HOLOPORTATION

Holoportation is a new type of 3D capture technology that allows high quality 3D models of
people to be reconstructed, compressed, and transmitted anywhere in the world in real-time.
When combined with mixed reality displays such as HoloLens, this technology allows users
to see and interact with remote participants in 3D as if they are actually present in their
physical space.

Holoportation -- which leverages camera arrays for scanning -- compresses, transmits and
reconstructs the 3D scans to Microsoft's HoloLens and other head-mounted displays.

The tech allows wearers of augmented and virtual reality headsets to see and interact with
remote users in 3D as if they were in the same space, according to Microsoft.

Holoportation makes interacting with remote parties as natural as face-to-face communication

2.1HOW DOES THIS HOLOPORTATION WORK?

It uses an array of 3D video cameras that capture pictures of a person from all corners. Then
simultaneously join together a 3D model of the person that can be reorganized, compressed,
and shifted anywhere instantly.

A person wearing Microsoft’s HoloLens (an improved reality headset) can watch, listen and
communicate with another participant who is being involved for this Holographic
teleportation from anywhere in the world. The other participant will look like a hologram to
the person wearing HoloLens.

Unlike other Hologram technologies, the appealing part is the capability to communicate
virtually in the real time and merges the experience with current scaffold-like HoloLens.

By using the Holoportation system, communications that take place can be recorded and
played later at the time, the user wants either with HoloLens or any other display technology.
2.2USES OF HOLOGRAPHIC TELEPORTATION
The basic idea behind Holoportation is that a user can teleport his holographic presence into a
different physical space. This breakthrough is achieved through 3D capture technology and
mixed reality headsets, such as the Hololens. In order to “holoport”, each user must be in a
room equipped with the capture technology, which reconstructs, compresses, and transmits
the user as a 3D model in real-time. Equipped with a Hololens, the user is then able to
interact with his partner’s 3D holographic model in real-time within his/her own physical
environment.

1. A Game-Changer in Telecommunications

Beginning with writing and eventually resulting in telephones, we have developed methods
of communication that allow our thoughts and ideas to be somewhere that we are not. In the
last decade, advances such as Skype and FaceTime have allowed our visual likeness and
voice to be broadcasted anywhere in the world. Still, this has failed to produce the sensation
so vital in human interaction – presence.

Presence gives human interaction its fullness, its immediacy, and its urgency. From these are
evoked the vital and primordial emotions of intimacy, trust, respect, and fear. Unfortunately,
current forms of long-distance communication fail to capture these qualities. As such, the
telecommunications industry has become increasingly interested in how to provide an
adequate form of telepresence, which means to convey the feeling of actually physically
being with someone through technology. Video conferencing and chatting technology are the
crudest versions of this. It allows users to see and hear, but there is lack of dynamism and
interaction. It is becoming abundantly clear that holoportation represents the largest step
forward – and likely the portent of a complete solution – in the search for evocative
telepresence.

2. Business, Industry, and Design Made Slicker and Quicker

We live in a world that is becoming ever-more connected. Increasingly, open markets and the
proliferation of multinational corporations are ramping up the demand for close collaboration
between individuals separated by continents and time zones. Holoportation bridges this gap
in a way no previous technology has been able to. Representatives from the far corners of the
Earth will no longer have to look at each other on 2D computer screens but actually be able to
co-inhabit physical spaces, thereby affording them a host of new opportunities. Architecture
firms will now be able to showcase to their clients’ 3D models in real-time. This level of
clarity in communication, free of the expensive transportation costs associated with face-to-
face meetings, will greatly increase efficiency and satisfaction among both architects and
customers.

Fashion designers will now be able to showcase real-life models wearing the latest trends to
potential retailers the world round in a manner that allows everyone involved close
inspection, foregoing the pomp and excess costs of elaborate fashion shows. Manufacturing
firms often must import equipment and specialists to train their staff in its use. Holoportation
could allay the costs of needing to send such people in the flesh. Instead, through
holoportation, dynamic, real-time training could be provided without inter-continental travel.

3. Hanging Out with Loved Ones Continents Away

Almost all of us have been away from home for extended periods of time and found ourselves
missing loved ones. We write letters, make long distance calls, and plan Skype dates, but it
just is not the same as being in the same room as them. It is here that, for many of us, the
advantages offered by holoporting become apparent: advantages offered by holoporting
become apparent:

The key innovation provided by holoporting is that it opens up spaces for users separated by
distance to engage and play with each other. This type of dynamic, cooperative, and informal
interaction defines most of our important relationships. Without it, our relationships suffer.
Skype and FaceTime are effective for “catching up,” but in truth, they often act as little more
than spruced-up phone calls. As it becomes easier – and expected – for people to move and
relocate about the world, the demand to remain close to loved ones in spirit will only grow.
Very soon, we all may be holoporting home to laugh and lounge with friends and family.

4. Making Our Streets Safer and Our Courts More Just

The underlying technology, irrespective of the holoportation aspect, is the high-quality, real-
time 3D modeling and rendering provided by the 3D-capture devices. When the Microsoft
Research team released information regarding this project, they also announced that it records
the sessions in 3D, virtual quality. These recordings can then re-inhabited in life-size or
shrunk and placed upon a table for easy viewing. As this technology becomes cheaper, more
efficient and widespread, it will have significant ramifications in terms of how we engage
with memory and how systems of law are able to design surveillance.
The applications of this technology for law enforcement and disciplinary functions are clear
and extensive. In all likelihood, the successful implementation of expansive 3D-capture
systems to survey and protect areas in the mold of CCTV monitoring remains a distant
reality. Its feasibility, however, is logical and its applicability obvious. Instead of video
recordings that can offer unclear angles, this technology would provide exact replications of
what transpired, removing doubt and bias from judicial proceedings.

5. Capturing Memories in Their 3D Completeness

Memory underpins how we as people understand ourselves. It does not exist as a pure
recollection of true events but a sort of hybrid, creative synthesis of impressions, stories, and
moods. Additionally, memory does not work for us as a tool of bidding but as a sort of
irrational schizophrenic that spits out moments and thoughts, sometimes without apparent
causal relation. At least that is the way it has always operated.

It is possible that rooms of the future could be rigged with his 3D-capture technology and in
the same way we now decide to photograph or videotape a special moment, we may simply
turn on the 3D capture. Instead of photos and videos, we will have fully-immersive 3D
representations of what happened. Memories will no longer be dim yet poignant feelings but
sharply, engaging visual experiences. Christmas mornings will no longer be captured in 30-
second clips but able to be re-lived at will in virtual totality.

6. Gaming

HoloLens is already changing the face of gaming but introducing games that involve
overlaying the physical environment with virtual elements. Suddenly through the walls of
your living room can appear alien invaders and out of the floorboard rasping skeletons. As
holoportation technology improves, pretty soon your buddies can also appear in your living
room in real-time, fully-rendered speed and clarity.
The simplest conceptions of holoported, mixed-reality gaming could be sport-based games
such as ping-pong. Simply holoport your friend into your home, load up the app and begin
competing.

It could also serve to augment traditional forms of multiplayer gaming by allowing you to
holoport your friend right onto your couch. Then simply load up your favorite split-screen
shooter, co-op zombie survival thriller or sports simulation on your preferred console and get
gaming. The benefits of holoporting allow the full-range of trash-talking, in-your-face
gloating, and sense of camaraderie that can so often feel absent in traditional online gaming
experiences.

benefits of rapid, real communication; loved ones will break the stifling limits of screen-
based interaction, and police will be able to more safely and accurately monitor our streets.
 Chapter-3

MICROSOFT’S HOLOLENS

HoloLens is essentially a holographic computer built into a headset that lets you see, hear,
and interact with holograms within an environment such as a living room or an office space.
Microsoft has built the headset without the need to be wirelessly connected to a PC, and has
used high-definition lenses and spatial sound technology to create that immersive, interactive
holographic experience.

3.1What’s powering the HoloLens?

Under the hood, Microsoft has packed the HoloLens with a series of sensors, along with a
high-end CPU and GPU, which the company claims gives it ‘more computing power than the
average laptop’.
Specifically, the headset features an inertial measurement unit, an ambient light sensor, one
camera for measuring depth, and four ‘environment understanding cameras’, all of which
combine to process information about the environment you’re in and how you’re interacting
with it.

On top of that, a 2-megapixel camera allows you to capture videos and photos of your
holographic exploits, while four built-in microphones will register voice commands.

To create an intuitive interaction with holograms, the HoloLens also uses 32 bit Intel
architecture and a Holographic Processing Unit (HPU) to smoothly process real-time data
captured by the on-board sensors. The custom-built HPU understands gestures, processes
where you’re looking, and maps the surrounding environment in real time.

When it comes to power, the HoloLens features a battery that will allow for two to three
hours use and up to two weeks of standby time. You’ll also be able to use the headset while
it’s charging over Micro USB.

Elsewhere, the development edition of the headset ships with 64GB of Flash memory, 2GB
RAM, and Bluetooth and Wi-Fi connectivity. With Bluetooth, the headset can now be used
with a new Clicker accessory, which comes with the developer edition and allows you to
navigate the HoloLens interface without using gestures. Likely designed to give developers
working long hours a break from using their arms, there’s no word on whether the Clicker
will ship with the final version.

The headset is adjustable, and Microsoft says the weight has been evenly distributed for a
more comfortable fit. At 579g it’s heavier than was originally expected. It’s also heavier than
both the Oculus Rift, which weighs 380g, and the HTC Vive, which comes in at 555g,
making the HoloLens the heaviest of all the forthcoming headsets.

Of course, the battery is necessary due to the fact that HoloLens is completely untethered,
meaning you don’t need a PC or any other external hardware in order to use it. That’s a big
deal considering all the Oculus, Vive, and Playstation VR have to be wired to a PC or PS4 in
order to work.

3.2What will you be able to do with HoloLens?

Among the most interesting, HoloLens will be able to let you view and interact with work
projects such as assembling 3D models, play games like Minecraft, video chat with hologram
versions of Skype contacts, and even watch live content.

An on-stage demonstration further emphasised the ability to create 3D objects, from

assembling to choosing the colours, all using simple gestures that looks very Intel Real Sense.

One of the most fascinating uses, at NASA, which collaborated with Microsoft in the making
of HoloLens, is the potential to control the Mars rover Curiosity via the HoloLens. The
headset will give NASA staff the ability to plant virtual flags in the terrain and work as if
they’re actually on the planet themselves.

Microsoft is also partnering with other companies to use HoloLens in practical ways. Volvo
is expected to bring the technology to its car showrooms where customers will be able to
view different colour options for their chosen car and see safety features in action. The car
maker is also expected to integrate HoloLens tech into its autonomous vehicle projects,
although it remains unclear how.
The first Minecraft game designed for Hololens lets you project maps onto a physical table
simply by saying the words, ‘create world’. Then with your hands, you can play with items
and characters. This can be done all while another player is immersed in the Minecraft world
from a Microsoft Surface tablet.

At a Windows 10 event late last year, Microsoft showed off a brand new demo known as
Project X-Ray, which showcases mixed reality technology.

An unknown wearer took to the stage wearing the HoloLens and quickly donned a wearable
hologram, in the form of some kind of space blaster. He shot through one of the walls on the
stage a few times, but then suddenly a crack started to appear and a mechanic alien burst
through it.

The robots themselves know where the walls and furniture are in your space, meaning the
gameplay is tailored to you. There’s options to change your weapons using voice commands
and there’s a tonne of different enemies that you’ll come up against.

Along with the launch of the HoloLens developer edition came news of the first confirmed
games and apps for the headset. Microsoft confirmed that the dev kit would come with seven
pre-installed AR experiences. On the app side, there’s HoloStudio, Skype, and HoloTour,
while games include Fragments, Young Conker, and RoboRaid.
In March 2016, Microsoft showed off a seriously cool new app called Actiongram, which it
plans to launch in the summer. The storytelling app will allow developers to create videos
featuring holographic effects for use with the headset.
 Chapter-4

DEVELOPMENT OF HOLOLENS

The Microsoft HoloLens holographic device features multiple sensors and processing units,
such as a depth camera, IMU (inertial measurement unit), HD video camera, microphones
and ambient light sensors. Unlike most previous devices, it is completely independent of any
other devices and does not have to be in any way tethered or connected to a desktop
computer. It uses on board processing units, more specifically a Holographic Processing Unit
and an Intel 32 bit architecture compatible processor. It has 2 gigabytes of RAM memory and
64GB of flash memory for the OS and various applications. The device is also compatible
with Unity3D which enables simple development of mixed reality applications. However,
due to the fact that its OS is based on the Universal Windows Platform, there are restrictions
in libraries and dependencies which can be used. For example, all Windows compiled
libraries need to be recompiled for use with the UWP, and some libraries used in Unity3D are
not available either. Although development was significantly streamlined and simplified by
HoloLens’ compatibility with Unity3D, these restrictions also had to be taken into account.
The HoloLens device also has multiple sensors which help it display the user in mixed reality
and give its user the feeling of presence and realism. Among its sensors it has a depth camera
which is used to map the user’s surroundings (usually the room that he is in) so that virtual
objects could be placed in real space and displayed relatively to the floor or some other part
of the room, and even allows the virtual objects to be (partially) occluded by objects such as
tables or chairs. The inertial measurement unit works together with other sensors to make
sure that the virtual objects are seamlessly blended with the real mapped surroundings even in
quick user or head movements. A holographic headset such as the HoloLens most commonly
uses a transparent display to combine projected images and color with light from the
background in order to produce an overlay of the virtual item on the background.
Networking

During development several methods of networking were attempted. One of the first attempts
was using Unity’s built-in networking; however it was unfit for the purposes of this system.
One of the problems with the built-in system is that it is difficult to adapt to high bandwidth
communication with a requirement for reliable, sequenced data transfer in real time. The
large data structures had to be broken up to be sent in fragments using the Reliable
Sequenced or Reliable Fragmented channel configurations. This lead to the need for multiple
uses of array copying and preparing functions which further slowed down the processing.
Since much of the functionality contained within Unity3D’s networking is encapsulated, it is
sometimes hard to identify the point of failure when unexpected complications occur. These
reasons lead to the abandonment of this method.

The second idea was using parts of the initial framework which uses C++ dynamic link
libraries for additional functionality. A network data sending component had been created,
using Boost libraries for C++. Due to the complexity of asynchronous communication with
Boost, and the difficulties concerning data transfer between Unity3D and the dynamic
libraries, working with large amounts of data requires careful use of buffers, which are
difficult to synchronize when working with data coming from Unity. One of the considered
options was to use Media Foundation, a framework compatible with the Universal Windows
Platform which is run by the HoloLens device. Another advantage of the Media Foundation
platform is built in support for hardware encoders such as the Intel Quick Sync video
encoder. However, this relies on the FFmpeg application for color and depth image
processing (here we are considering that the depth frame is encoded. The ultimate solution
was much simpler, a TCP server and client for reliable sequenced communication written in
C++ using the native System.Net library. Data is read from a network stream using a TCP
socket for byte transfer. This code was optimized in later stages of the project. Optimizations
of the system will be described in section 4.1.2. This was mostly inevitable due to the
blocking functions used in network transfer. It is important to notice that since this network
transfer solution works with byte arrays, it is invariant of the type of data which is sent,
meaning that the same component works for mesh, audio and color data.
Depth data processing

When raw data is captured on the Kinect V2 sensor, it is obtainable from the Unity plugin
SDK. This data needs to be processed before it can be displayed on the other end of the
system. Processing includes background removal, mesh construction and data compression.
Due to the fact that the system is localized on two devices, each of these three tasks may
either be performed on the server or on the client side. Because the HoloLens is a portable
device with lower performance than a desktop computer, most of the work was intended to be
delegated to the server. For background removal and mesh construction, samples from the
Unity3D Kinect v2 samples pack were used as a starting point. This code is fast enough and
the quality is satisfactory, both when it comes to background removal and mesh creation. In
the later phases, however, it had to be optimized further, in order to ensure that all
components work well with real time processing. One course of development involved using
the FFmpeg executable to stream the data as an MP4 video stream via a direct connection
from the server to the client. This proved to be difficult to implement due to the fact that the
program interface is scarcely documented and the errors are non-descriptive. The reason this
method was abandoned is that MicrosoftHoloLens runs on the Universal Windows Platform,
which is not compatible with x86 architecture built binaries and libraries, which FFmpeg
uses. The FFmpeg executable is a console application which takes parameters telling it which
data should be processed and how it should be handled. Several Unity scripts had been
written, invoking the external program to research feasible communication methods. The
main problem was finding a way to send the large amounts of data to the external executable
and receive the processed data in return. This was attempted in several ways which, most
notably, included passing the data written on the disk (a very slow method not suitable for
real-time applications but good for development) and passing the data through a pipeline .
The executable takes several parameters, usually these are: source and destination of the data,
data formats and format details, image or sample size, used codecs, etc. The input and output
can be specified as either files on the disk, sequences of files, web data streams or pipelines.
Most widespread formats and codecs are also supported. One of the problems with using
FFmpeg with grayscale represented depth frames is that an 8 bit grayscale image reduces the
precision of the depth data, as it is conventionally stored in 11 bits. Using an unconventional
16 bit grayscale image is a bit more difficult as Unity3D does not support such formats.
When it comes to the implementation of the Unity3D to FFmpeg executable communication
there are several things to address. The FFmpeg process is started from the script using
functions from the Systems.Diagnostics namespace, with the following StartInfo settings:
UseShellExecute is false, CreateNoWindow is true, and we have redirected the standard
input, the standard output and the standard error of the process. The executable uses the
standard error stream to show info to the user, and since we want no window to be displayed,
the standard error stream should be redirected and read. Displaying it with Debug. Log gives
us the only insight into what is happening within the FFmpeg program if processing fails.
Input is given to the process via the standard input and output is read from its standard output.
There was a lot of experimentation with arguments used for starting the program.

Eventually several scripts were written to convert the depth data into a grayscale bitmap with
24 bits per pixel, all three channels (red, green and blue) populated by the same value for the
purpose of testing. Although this might sound like a terrible waste of resources, these bitmaps
can actually be compressed very efficiently, as if only one channel was used. They are easily
converted into byte arrays in Unity, which are then sent as a stream over the network. Byte
arrays can also be easily compressed in real time. For this, C#’s Deflate class was used, as it
provides quick byte stream compression which is suitable for real-time applications. The
Deflate algorithm found in the namespace System.IO.Compression creates a DeflateStream
from an underlying MemoryStream, which is suitable for real time compression. It is a
lossless data compression algorithm which uses LZ77 and Huffman coding. The algorithm
takes a byte array and replaces duplicate sequences with pointers, and replaces normal
symbols with weighted symbols based on frequency of use. Specifically, Huffman coding
replaces common longer byte sequences with their short identifiers which take up less space
in the stream. The LZ77 algorithm replaces repeating sequences with a single copy, plus a
length-distance pair which denotes how many of the following characters are equal to
distance characters behind it.

Before sending the user’s mesh over the network, the mesh needs to be serialized; converted
into a byte array. For that purpose the system uses a mesh serializer, which writes the
individual mesh components (mesh header, vertices, triangle indices, and UVs) to a data
stream which is then saved to a byte array. A complementary functionality is also contained
within the mesh serializer, so that deserialization could be done on the receiver. Since this is a
relatively programmatically simple (yet quite resource consuming) operation, the same class
could be used in the Windows Holographic build with little adjustments. In the optimization
phase of the sender development, this class was revised so that the serialization could be
parallelized. This meant replacing the Unity’s non thread-safe Mesh class used in the
serializer with a custom MeshThreadsafe class which is basically just a holder for the data of
interest within the mesh itself.

Color data processing

Color data read from the Kinect V2 sensor can be either in full HD (1920x1080 pixels) or in
the same resolution as the depth image, which simplifies texture mapping. It also reduces the
network traffic of the system greatly, due to the huge difference in data size of each frame.
Most of the methods used with depth image bitmaps work the same with color bitmaps, so
finding fast and efficient compression and encoding algorithms was an important focus.
Color data, however, due to the fact that it uses four channels (32 bits, RGBA) instead of two
used by depth data, generally takes up a bigger percentage of the network traffic. This is the
first reason why using the full HD texture stream was eventually abandoned in favor of the
smaller depth resolution texture. The second reason is that using the full HD texture requires
additional processing to map the texture correctly to the depth image. This is a consequence
of the fact that the 1920x1080 resolution image not only has a different aspect ratio than the
512x424 pixel resolution image, but also has a different field of view.

Additionally, a delta-encoding script was created to work with general byte arrays, in order to
maximize the compression. Delta encoding works by storing data in the form of full and delta
(difference) frames. Frames are recorded in a fixed interval and stored as a difference relative
to the full frame which is refreshed periodically. This greatly reduces redundancy, since
adjacent frames often share much of their data. Instead of using an existing delta compression
library, a custom-tailored component has been developed to best fit the requirements of this
project.

The sender-side color processing script includes handling the loaded texture, delta-encoding
and compression of the data, as well as supplying the data to the network sender component.

Other components

For the purpose of data sender project optimization a script was written for threading
efficiency testing. The script relies on tested functions to report when they have reached a
section which was determined problematic (taking up a lot of time to be completed) and
when they have left it. Each of the threads calls a function of the ThreadDurationTest, as it is
called, to obtain a unique ID which will later be used to identify it. As a thread calls the
WriteCheckpoint function of the duration test script, a unique timestamp is written to a file,
identifying the start of the “critical” section or its end, if a start was previously written. Once
a file is produced, this data can be processed and visualized. For that, we have developed a
processor script which is written in the Python language and uses several libraries from the
SciPy packages. It reads the checkpoint file and draws a graph from the data. The different
colors represent different threads, and the spikes denote thread activity in the “critical”
section of interest.
 Chapter-5

FUTURE OF HOLOPORTATION

5.1Holoportation: travel without moving an inch

Go a long way without even leaving home. This is the promise made by the inventors of
holoportation solutions. For, with this revolutionary “transport” system you only need a
virtual reality headset, a good internet connection and a few 3D cameras around you. Of
course, the movement is a virtual one. But the effect is stunning.

Equipped in this way, you are projected into an environment that could be on the other side of
the planet or on the other side of the street. Represented by a “holographic” avatar, you can
interact with that environment: handle (virtual) items and talk to the holograms of other
people who are also connected. Imagine: using this augmented reality solution, you can use
your digital double to take part in a business meeting organised in Sydney without even
leaving Paris, you can follow an interactive conference in a New York museum with your
friends…

The applications are almost limitless and they concern all areas of life:

 Communications, of course, where holoportation will transform the principle of

videoconferencing and entertainment (video games, etc.);

 Education (with the possibility of remote participation in tutorials and laboratory

experiments);

 Medical Consultations: the Australian company Silver Chain, which specialises in home
care assistance for the elderly, has thus co-developed a virtual reality control panel which
floats above the “patient” when the nurse is providing care. It is now hoping to take a further
step forward with medical examinations by hologram.
5.1.1ROLE IN COMMUNICATIONS
5.1.2FUTURE IN EDUCTION

Technology is becoming key in customizing the classroom experience for students. Children
with special needs or those who may be behind in certain subjects can benefit from VR
lessons tailored to meet their needs. The more immersive experience helps to engage
students, and allows them to take learning at their own pace. Artificial intelligence-powered
VR “teachers” can help students feel safe and guide them through their lessons. VR, paired
with skilled human teachers, can help ensure that no child feels confused or discouraged.

Engaging Students and Bringing Education to Life

Some of the most exciting learning happens when it doesn’t feel like learning. VR is an
excellent tool for bringing education to life and engaging students. Using VR technology,
students can learn subjects like biology, history, and geography by immersing themselves in
worlds that are thousands of miles (or thousands of years) away. Students often feel more
inspired to learn if they have fun with the experience and get a “hands on” look at what
they’re studying. It’s very different to read about the Amazon rainforest than it is to “go
there” using VR.

VR can also provide an immersive experience for students with virtual fieldtrips. Classes can
visit educational museums like the Smithsonian or even take a tour of the solar system!

Helping to Prepare Students for the Future

We don’t perform our best when we don’t know what we’re getting into. The first days on a
new job, for example, we’re prone to errors and anxieties, which can have varying levels of
consequences. The good news is that VR can help alleviate some of these common issues by
immersing students in the environment they’ll be expected to navigate on the job.

Improving Adult Literacy

While VR in education has largely been focused on students in elementary, middle, and high
schools, it could also play a key role in adult education. With more people choosing to go
back to school or improve their literacy (especially adults for whom English is not their first
language), VR can help make that education more accessible.
When it comes to literacy, VR could provide an immersive, non-judgmental learning
experience for adults who struggle with language. Because the experience helps people to
retain what they’ve learned, VR could help adults in literacy learning courses learn more
quickly and effectively. With around 36 million adults in need literacy education in the US,
VR could make a huge impact on lives across the country.

The Future in the Classroom

Getting students of all ages excited about learning is an important step in getting engagement
and retention. Although VR is just beginning to make its way into the classroom, it has the
potential to both engage and customize the learning experience. Since there are so many
different types of learners, VR could hold the key to ensuring every student gets what they
need to perform their best.

It is possible to show students how physical or chemical processes occur that cannot be
shown in ordinary experiments, which will make it possible to understand and comprehend
the educational material. You can also make a reconstruction of a historical event. It will be
much more interesting than just reading about it. Therefore, there are a lot of options for
using this technology in the study.
5.1.3MEDICAL CONSULATIONS

What is EMMR?

EMMR is a Enhanced Medical Mixed Reality interface. It will use ‘mixed reality’ technology
to combine real life, projected holograms and video conferencing, allowing our nurses,
clinicians and clients better access to information and services.

Clients currently receiving hospital in the home and health services, EMMR will enable real-
time interaction with medical professionals in their homes. This includes doctors who can be
projected as holograms via the headset.

The mixed reality headset will allow nurses to share information hands-free, and in the future
will be able to record patient data, in real time via a virtual dashboard. Using EMMR, a Silver
Chain Group nurse visiting a client at home will have access to all client medical data
through the virtual display, with the ability for a doctor to effectively see through the eyes of
the nurse as the examination takes place.

EMMR helps our clients by removing need for travel to and from appointments, and giving
them access to face-to-face consultations with their doctor via 'holoportation'. They can be
confident that they are receiving the world’s best health and aged care when they need it, in
their own home.

Silver Chain Group will become the first company in the world to apply Microsoft’s
HoloLens technology to virtual hospital services delivered in the community. EMMR will be
the first in a range of services we will deliver that signal a significant shift in the way clients
receive comprehensive, hospital-level care at home, so they can live their lives as they
choose.
5.2MOBILE HOLOPORTATION

With technological advances resulting in a significant decrease in bandwidth requirements,

Microsoft has a working prototype of a holoporatation system that can be used in a moving
vehicle.

Holoportation is a 3D video capture technology that, when combined with a mixed-reality

displays like a HoloLens or HTC Vive, allows users to be present as a 3D image from a
remote location.

Previously, a person could use 3D video technology to capture their image and project it to a
remote location, interacting with remote partners in real time. Advancements in motion
capture technology have allowed researchers at Microsoft to reduce the bandwidth required
for holographic projection by 97% while maintaining video quality.

This means that holographic images can be captured and presented on the go – in this case, an
individual in a moving vehicle can be present holographically at a remote location,
interacting with other participants in real time.

Projection requires only two cameras, although the more that are incorporated, the higher
quality image that is projected. Viewing an image in 3D requires the remote participant to
wear a mixed-reality display device.

The company had to overcome several engineering challenges in order to make mobile
holographic projection viable. The first was the bandwidth issue. By using a unique
compression algorithm, researchers were able to reduce the bandwidth required to 30-50
Mbps, meaning that real-time projection was made possible for a vehicle within Wi-Fi range.

The system will be usable on cellular data soon.

Other issues included reducing the size of the equipment to fit in a vehicle, and making the
system robust enough to handle changes in vibration and lighting that would be present in a
moving vehicle.

The research team found it important to note that the technology was made for passengers
only, to be used in the back seat of the vehicle, without interactions with the driver.
Live 3D capture allows users to interact with remote people in a realistic and natural manner,
but the system also has record and playback abilities. These allow a person to rewind and
review holographic interactions as if they were a third party, experiencing them from an
alternate point of view.

5.3AI CO-PROCESSOR

The AI coprocessor is designed to work in the next version of HoloLens, running

continuously, off the HoloLens battery. This is just one example of the new capabilities we
are developing for HoloLens, and is the kind of thing you can do when you have the
willingness and capacity to invest for the long term, as Microsoft has done throughout its
history. And this is the kind of thinking you need if you’re going to develop mixed reality
devices that are themselves intelligent
 Chapter-6

PROBLEMS WITH HOLOPORTATION

Until the massive success of Pokémon Go in 2016 when augmented reality (AR) was
catapulted into the public’s consciousness, AR was overshadowed by its cousin, virtual
reality (VR). Many were more optimistic about the applications of virtual reality compared to
augmented reality.

However, as AR and VR have evolved over the past year, it has become evident that AR
offers more practical daily use cases. From retail to education to manufacturing, AR is
positioned to drive business value across sectors.

With that, there are still several challenges that lie ahead for the mass adoption of AR in the
short term. Here's a look at three:

1. Holoportation reality hardware

Today, no AR headsets are available for consumers. Microsoft HoloLens and Meta 2have
released developer versions, but they have not yet announced when we can expect their
devices to ship to consumers. Even more, HoloLens and Meta still boast hefty price tags at
$3,000 and $949, respectively.

The Osterhout Design Group (ODG), which has produced headsets for enterprise and military
use for years, unveiled at CES in January 2017 two models of AR/VR smart glasses for
consumers. Despite the announcement, a shipment date has yet to be confirmed.

Let’s not forget to mention the design of the current headsets on the market. Although the
Meta 2, HoloLens and ODG units offer a compelling user experience, they are not sleek
enough for mass consumer adoption. ODG’s new R-8 and R-9 models may be the closest to
consumer-ready, but there is still a way to go in streamlining the design. Also, it’s notable
that the Meta 2 must be tethered to a computer, obstructing all convenience.
As we wait for smart headsets to hit the consumer market, AR must be experienced through
our mobile devices. The issue with our mobile devices is that they aren’t equipped with room
mapping or depth sensing technology. The only phone on the market with this capability is
the Tango-enabled Lenovo Phab 2. Tango uses computer vision to give devices the ability to
understand their position relative to the world around them. This allows you to not only place
virtual objects in the real world, but also these virtual objects can interact with the
environment. This interaction is important as we aim to create the most life-like experience
for the end user and to offer them the most utility.

2. Holoportation content

A second challenge for holoportation is the content. Imagine your iPhone without apps. If
you have the hardware but no content for the end user to consume, it’s all for naught.

When HoloLens and other headsets ship to consumers, they must have apps and programs
that enhance the user experience. Without this utility, consumers will not be persuaded to
adopt the technology.

This is why companies such as Microsoft are doing their best to attract developers to build
apps for their platforms. They need the content before shipping to consumers.

Similarly, organizations that want to use AR internally (e.g., product prototyping) or

externally (e.g., ecommerce) need the 3D content. For instance, if a retailer wants to integrate
AR to its ecommerce mobile app, it needs to secure the 3D product models of all its SKUs.

Securing all this 3D content can be both expensive and timely, but companies are quickly
realizing the importance in the investment. Way fair has built an extensive 3D model library
to fuel its on-going innovation in VR and AR. We are moving towards a reality of mixed
reality, and businesses are readying themselves today to avoid being left behind.

3. Holoportation education

One of the most difficult challenges about holoportation is educating the broader
market. Consumers aren’t exposed to holoportation regularly and don’t see its wide-reaching
applications in their daily lives. However, there are plenty of holoportation experiences
available today that just lack exposure in the consumer market.
Also, it is important to build a strong pipeline of rising talent in the AR/VR community to
continue to innovate and develop this technology. It is critical that students have exposure to
this technology in the classroom. It is easy to be intimidated by emerging tech, but with early
exposure, AR/VR can become more accessible.

France is one of the countries leading the way in giving its student’s exposure to AR early. In
2015, the French government announced an updated curriculum for students that weave
augmented reality into their studies. The government recognizes that teaching emerging
technology in the public school system provides all students with the keys to understanding
the contemporary technical environment and the skills to act.
CONCLUSION

This system provides a proof of concept and a technology demonstration for real time
holographic communication. With some additional optimizations it will be possible to
implement two way communication and one way communication with multiple receivers
(broadcasting; which was the basic intended purpose of the system). Many of the solutions
found during various research phases of the project may prove useful in similar systems even
if they were deemed unsuitable for this project. The system can also be upgraded to use
multiple depth cameras for a more accurate representation of the recorded person, to give the
users a better feel of presence. With multiple depth cameras recording from all angles, a full
3D model of a person could be rendered in real time on the receiver side, unlike the current
mesh produced by one camera, as shown in Figure 5.1. More complex mesh reconstruction
methods which use machine learning can also be implemented to further reduce network
traffic and make the application more accessible. To show more detail of the user’s face, his
mesh may be split into two parts with the head having a more accurate depth sampling and
the body, which requires less detail, being subsampled in lower quality. Finally, although the
system was built for current generation depth cameras and holographic headsets, different
hardware can be used to provide more accurate visualizations or application-specific
advantages. Even if the system is revamped to an entirely different configuration, for
example using a mobile phone instead of a holographic headset or a RGB camera for
background removal instead of a depth camera, many developed components can be reused
with some modifications.