1.

Simultaneous control of multiple drones

2. 3D modelling and mapping if internal
structure of the building
3. SLAM technology

Name: PRABHAKAR JAISWAL

Branch: ELECTRICAL ENGINEERING
 FLEET CONTROL
INTRODUCTION
In the past decade, unmanned aerial vehicles (UAVs) have been widely used in
various civilian applications, most of which only require a single UAV. In the
near future, it is expected that more and more applications will be enabled by
the cooperation of multiple UAVs. To facilitate such applications, it is desirable
to utilize a general control platform for cooperative UAVs. However, existing
open-source control platforms cannot fulfill such a demand because
(1) they only support the leader follower mode, which limits the design options
for fleet control,
(2) existing platforms can support only certain UAVs and thus lack of
compatibility, and
(3) these platforms cannot accurately simulate a flight mission, which may
cause a big gap between simulation and real flight.
To address these issues, we propose a general control and monitoring platform
for cooperative UAV fleet,
namely, CoUAV, which
provides a set of core
cooperation services of
UAVs, including
synchronization,
connectivity management,
path planning, energy
simulation, etc. To verify the
applicability of CoUAV, we
design and develop a
prototype and we use the
new system to perform an
emergency search
application that aims to
complete a task with the minimum flying time. To achieve this goal, we design
and implement a path planning service that takes both the UAV network
connectivity and coverage into consideration so as to maximize the efficiency
of a fleet. Experimental results by both simulation and field test demonstrate
that the proposed system is viable.
Clearly, to facilitate multi-UAV applications, it is desirable to utilize a general
platform to control and monitor UAVs, as illustrated in Fig. 1. In the literature,
there exist some opensource control platforms for UAVs, referred as Ground
Control Stations (GCSs), including Mission Planner [4], Q Ground Control [5]
and DJI FLIGHTHUB [6]. Although all of these GCSs support the basic flight
control functionality, such as flight planning by editing waypoints,
communication with UAVs, user-friendly GUIs, flight trajectory displaying on a
map and real-time vehicle status monitoring, the following limitations limit
their applicability as general control and monitoring platforms for cooperative
UAVs.
• Only leader-follower mode is enabled in the existing GCSs. Though leader-
follower model makes path planning much easier, it cannot fully utilize all
UAVs in a fleet to complete a complex task at the earliest time or with shortest
flying distance.
• Each GCS only supports a specific set of UAVs or UAV flight controllers. For
example, Mission Planner is designed primarily for ArduPilot hardwares and
firmwares; QGroundControl only supports UAVs that communicate using the
MAVLink protocol; the DJI FLIGHTHUB interacts with DJI’s own products only.
• There is a lack of energy simulation module in existing GCSs. Without energy
simulation module, existing GCSs cannot predict energy consumption through
simulation. Thus, the feasibility of a flight cannot be tested before UAVs take
off. As a result, some flights may have to be aborted before their tasks are
completed due to early energy depletion.
To this end, we propose CoUAV, which is a control and monitor platform that
enables easy-to-implement UAV cooperation, to address the aforementioned
limitations. Specifically, to address the first limitation and allow UAVs in a fleet
to maximize the fleet efficiency, we propose a more generic path planning
framework where UAVs do not need to follow leaderfollower mode. Instead,
the proposed framework enables cooperative path planning by introducing
swarm functions (e.g., synchronizations, connectivity maintenance). To
demonstrate the functionalities, we provide an embedded path planning
service for the multi-UAV cooperation by considering both the UAV network
connectivity and coverage. To address the second limitation, we provide the
hardware independence to each UAV by introducing a companion linux kernel
device, which serves as a middelware to interact with UAV autopilots. Since
almost every commodity provider and open-source community offers linux-
based SDK for UAV flight control, such UAV companion devices hide the
hardware and software difference of UAVs from different manufacturers.
Hence, our CoUAV platform is generic enough to work with various UAVs,
regardless their hardwares, firmwares, and communication protocols. To
address the third limitation, we add an energy simulation module to the
CoUAV platform. To make the simulation reliable and close to the real-word
flight, we make efforts to energy prediction, which can avoid the task
abortions in the field. In a fleet with heterogeneous drones, different UAVs
may consume different amount of energy even when they fly at the same
speed/cover the same distance. Our platform provides an accurate energy
model tailored to different types of UAVs, ensuring the feasibility of the real
flight under planned paths. The key differences of functionalities among
CoUAV and popular GCSs are summarized in Table I. Besides the
aforementioned functionalities, CoUAV also offers other features such as GUI
APIs, UAV APIs, and simulation for multiple UAVs, based on which we
implement some basic modules, such as agent manager, emergency monitor,
and message center, for ease of application development. A developer can use
our platform to achieve rapid development without having to implement the
underlying modules. Our contributions can be summarized as follows.
• CoUAV has the advantage to provide effective cooperation services and
manage sophisticated networking protocols. The UAV agent developed in
CoUAV includes an independent middleware that can run a general operating
system, on which many open source projects can be executed. This implies
that CoUAV can support not only existing mainstream protocols, but also any
specialized airborne communication protocols proposed and developed in the
future.
• In addition to the primary cooperation services, CoUAV can further support
sophisticated path planning for a fleet, e.g., connectivity maintenance and
synchronization during the flight. By taking points to visit and UAV number as
input, CoUAV can generate the initial path plan so that the task can be
completed at the earliest time while maintaining the connectivity among UAVs.
The planned path information is then converted to a series of control
commands and disseminated to individual UAVs. When the path is impaired
due to environmental factors, like wind disturbance, the planned path can be
revised and updated.
• CoUAV provides interfaces to incorporate trained energy models as well as
modules to train energy models for different types of UAVs. By collecting
energy data through historic flying tasks, we have learned an energy model for
Pixhawk-Hexa UAVs. Comparing the simulation results with the field test
results, the training energy model can achieve 94.26% accuracy.
• CoUAV can accurately simulate a flight mission. Moreover, CoUAV supports
an easy switch between simulations and testbed experiments executing the
same task. These advantages come from the system design, where the UAV
agent serves as a middleware between the original UAV and the ground station
of the CoUAV platform to hide the hardware difference. As a result, we can
replace any UAV models without affecting other parts of the platform, and also
use the UAV simulator to conduct simulations prior to the deployment. A

demo and the source code of the platform are available for public access in
https://github.com/whxru/CoUAV.

COUAV SYSTEM DESIGN AND IMPLEMENTATION

The CoUAV platform consists of two types of components: the UAV agent
installed in each UAV and the flight monitor operating on a ground station, as
illustrated in Fig. 2. In this section, we present the implementation details of
the UAV agent and the flight monitor on the CoUAV platform.
A. The UAV agent A typical UAV or drone system consists of motors, flight
control system, gyroscope, compass, GPS, remote control and battery. The
main task of the flight control system is to stabilize the vehicle and control its
movement through the control of motors, based on the information from
gyroscope, compass and GPS. The flight control system also provides the drone
information and control interfaces to external devices by a pre-defined
protocol. As shown in Fig. 3, the flight controllers on our current platform are
the APM2.8 board and Pixhawk HEXA borad. We further install a Raspberry Pi 3
motherboard (RPi) as the mounted Linux-kernel device to run the UAV agent
program. The UAV agent is responsible to handle three important types of
information or messages, including vehicle status, device control and
exceptions. UAV APIs that communicate between the flight control board and
the monitor are provided. The detailed illustration of the UAV agent and its
interaction with the UAV flight controller are illustrated in Fig. 4. 1) Handling
status information: To be compatible to various underlying flight control
boards with hardware differences, we access and update the flight status
information through SDKs between the flight control module and the UAV
agent program. UAV’s status information needs to be transmitted to the flight
monitor on the ground. Prior to the transmission, the UAV agent periodically
parses the flight status (from SDKs)
into the formats needed by the exception monitor and the information sender.
The information sender further converts it to a character stream for the
transmission. Although the APM2.8 board and Pixhawk HEXA borad are used
as the flight control in the current CoUAV implementation, our UAV agent
design can essentially work with any mainstream UAV flight controllers
because the UAV agent program serves as the middleware that hides the UAV
difference from the rest parts of the platform, including the flight monitor.
Consequently, for UAVs that utilize other flight controllers, our UAV agent can
bridge them to the flight monitor, since almost every commodity provider and
opensource community offers linux-based SDK for UAV flight control. 2)
Handling control messages: The control message from the flight monitor (on
the ground) is transmitted in the format of a character stream, which flows to
the message listener of the UAV agent on RPi. There are two types of control
messages in CoUAV: control command and parameter setting. For the former
type, commands will be appended to a First-InFirst-Out queue. The UAV agent
has a command handler that can convert each command into the format that
is executable by the flight control module. For the latter type, parameters such
as geo-fence boundaries, communication range and battery life can be handled
by the parameter setting message. 3) Monitoring exceptions: The UAV agent
also has an exception monitor module and the flight status information is
periodically sent to this module for inspection. As a result, the exception
monitor can track vehicle’s status changes and monitor the emergencies. In
case any emergency occurs, the exception monitor either delivers high-priority
commands to the flight controller or reports to the flight monitor through the
information sender. Exceptions in CoUAV include low battery, crossing the geo-
fence boundary, and bad health of connection to the monitor application, etc.
B. The Flight Monitor The main task of the flight monitor is to communicate
with each individual UAV and further offers a series of inevitable services for
their cooperation. In addition, the flight monitor also provides the interfaces to
interact with upper-layer applications and end users through APIs and GUI,
respectively. Fig. 5 demonstrates the GUI from the flight monitor in CoUAV
platform.
 3D modelling and mapping
The power of a drone equipped with a camera is undeniable. The problem is,
once you’re back in the office with photographs in hand, photogrammetry
solutions can take hours to generate the final 3D model. That extra time may
seem like a small price to pay given what you get in return, but it’s still a
bottleneck preventing us from squeezing all the value out of our UAV
solutions.

A new commercial technology developed by Lockheed Martin is set to change

that. Hydra Fusion Tools performs real-time 3D mapping from any drone
equipped with a video camera. It uses structure from motion (SfM) algorithms
to generate 3D point clouds and 3D imagery in real time while your drone is
still in the air.
HOW?

Hydra Fusion Tools is able to model so quickly because it exploits the latest
generation of graphics processing units (GPUs). These tiny computers were
originally developed to process graphics for visually complex video games.
Recently, however, their ability to handle graphical information has been put
to use in commercial technologies like augmented reality, and now,
photogrammetric processing.
If your computer is equipped with a powerful enough GPU, in other words, Hydra Fusion
can process the video beamed down from your drone and generate 3D maps on the fly.
Your computer may already have the specs necessary.

As for how fast it goes, John Molberg at Lockheed told New Scientist that the
company can already “fly a drone along at 30 knots” and create a map as it
goes. (That’s about 34 miles per hour for us laymen.)
The model you generate at that speed might not be survey-grade (30
centimeters per pixel) but common sense dictates that if you slow the drone
down, you can get denser data. This means that after one flight to gather
coarse data, you could fly the site again to gather more detail on areas of
interest. With a long enough battery life, someday you might be able to do all
of this before bringing your drone in for a single landing.

Lockheed is reportedly working with one client to map changes as small as 6

millimeters to measure changes in a railroad track.
 TOP 5 DRONE 3D
MAPPING SOFTWARE
PACKAGES
The commercial drone market has come of age in the past five years or so.
Today’s drones carry high-definition cameras and advanced tracking software
making them suitable for all kinds of commercial tasks including powerline
inspection, forestry management, weather reporting and, of course, mapping.

3D mapping is perhaps the most accessible of these features, with today’s

drones able to map large areas quickly and cost-effectively. To get this
information five years ago would have required renting expensive
photographic equipment and an aircraft to overfly the area, making it cost
effective for only the largest multi-million-pound projects.

Today, by using the latest drone mapping software and a little practice, high-
quality aerial maps can be created by any competent drone pilot. These can be
used by a number of industries, including construction, surveying, mining,
building inspection, forest, and agriculture to create high-resolution mapping
and imagery in real time.

This enables farmers to better plan crop rotation, allows insurance companies

to assess damage to buildings without endangering life, enables forest

management companies to monitor tree crown delineation and helps
architects to create an accurate 3D map of the topography of a site for
planning consideration.

So now you know how useful 3D mapping software is and how various
industries can use it, what are the best 3D mapping solutions available for
drone operators? To help you make an informed decision we’ll take a closer
look at the top 5 drone mapping software solutions on the market.

These are the five biggest players and we’ve assessed each platform for its
image quality, ease of use, range of features and drone compatibility. So
without further ado, let’s get started.

1. PIX4D

Pix4D is one of the most feature-packed

3D mapping software packages on the
market. Unlike other software Pix4D has
designed multiple versions for specific
industries, including agriculture,
surveying, civil engineering, and
construction.

The benefit of this is you get all the

features needed to carry out the task of
mapping for a specific industry. For
example, with agriculture, detailed digital
surface maps can be created allowing
farmers to plan crop cycles and create
more accurate yield estimations.

Pros:
Powerful feature-rich software with a range of versions available to suit
specific industries. Can easily connect with industry-specific
management platforms to help create yield predictions, profit and loss
accounts, planning documents and detailed site plans. Technical support
is also one of the best in the industry.
Cons:
Expensive for the casual user but the range of features available more
than compensates for this. Due to its complexity, the software can be
difficult to master for beginners.

2. CONTEXTCAPTURE

Bentley’s reality modelling software,

ContextCapture, provides you with real-world
digital context in the form of a 3D reality mesh.

A reality mesh is a 3D model of real-world

conditions that contains large amounts of
triangles and image data. Each digital component can be automatically
recognised and/or geospatially referenced, providing you with an
intuitive and immersive way to navigate, find, view, and query your asset
information. You can use reality meshes in many engineering,
maintenance, or GIS workflows to provide precise real-world digital
context for design, construction, and operations decisions.

Hybrid processing in ContextCapture enables the creation of

engineering-ready reality meshes that incorporate the best of both
worlds – the versatility and convenience of high-resolution photography
supplemented, where needed, by additional accuracy of point clouds
from laser scanning.

Develop precise reality meshes affordably with less investment of time

and resources in specialized acquisition devices and associated training.
You can easily produce 3D models using up to 300 gigapixels of photos
taken with an ordinary camera and/or 500 million points from a laser
scanner, resulting in fine details, sharp edges, and geometric accuracy.

Pros:
It has truly great 3D modelling capabilities and also allows users to
incorporate data from other survey data collection methods. This is a
highly recommended alternative to Pix4D for enterprise level
organisations.
Cons:
This software is quite expensive and it also requires some level of
training and expertise – not as intuitive as Pix4D.

3. AGISOFT

The professional version of the Agisoft

platform is great for users seeking a
well-rounded all-in-one 3D mapping
solution. With an excellent 3D
modelling engine aligned with
panoramic stitching and support for
fisheye lenses.

The Agisoft platform can take accurate

measurements, create point clouds
and digital elevation models with ease.

The platform also has support for NIR, RGB, thermal and multi-spectral
imagery so it has all the tools needed to create highly accurate 3D maps
for structural engineers, architects, and GIS professionals.

Pros:
Cost-effective all-in-one software suite with a full range of image sensors
including NIR, RGB, thermal and multi-spectral. The professional version
creates beautiful clear imagery which is as good as more expensive
software. All things considered, this is a well-rounded platform suitable
for most applications.

Cons:
The software feels clunky to use and the support leaves a little to be
desired compared to Pix4D. The one licence, one computer policy may
also be a problem for some larger organisations.
4. DRONE DEPLOY

DroneDeploy is one of the most

popular 3D mapping solutions,
with free versions available for
download on the app store and Google Play. DroneDeploy is the perfect
starting point for anyone wanting to explore drone mapping software. It
also comes with a range of useful features.

Despite being aimed at the consumer market the paid versions of

DroneDeploy offer a range of sensor options which make it suitable for
some commercial applications.

There are three paid versions available, Pro, Business, and Premier
which provide the ability to measure volumes or group control points and
export data and images to third-party software including AutoCAD,
Blender, and Rhino.

Pros:
DroneDeploy is a great platform for users who are just getting started
with 3D drone mapping software. While the free version is too limited for
most commercial enterprise uses. The paid versions do provide
increased image quality and have the ability to connect with a wide
range of third-party drone accessories and software.

Cons:
Surface detail for buildings is somewhat disappointing compared to the
more specialised solutions from Agisoft and Pix4D. This makes the
platform more suited to agricultural applications than construction or
engineering.
5. PROPELLER NETWORK

Propeller’s drone mapping

software gives you the ability to
measure and manage your site
yourself. It features visual tools
that you can use from any device,
anywhere.

Propeller Network is ideal for heavy civil and resource operations and
helps you and your team to work better together to answer questions
about site productivity and progress.

Their system allows you to see earthworks progress, report on inventory,

and track changes with 3D surveys. The data is captured quickly, safely,
and cost-effectively.

Propeller also has the incredibly handy AeroPoints. They are the world’s
first smart Ground Control Point system and have been specifically
designed for drone surveying. They are lightweight and durable and are
already used in thousands of surveys in some of the toughest
conditions.

Pros: Propeller is ideal for team data collections, and is a quick and
efficient data collection and management system. It also has data
analytics that is tailored for your industry, as well as their AeroPoints
feature.

Cons: This platform is aimed at a more experienced user market. So

with that in mind, wouldn’t be ideal for first-time users or new commercial
drone businesses.
SUMMARY

As you can see, users are spoilt for choice when it comes to choosing
high-quality capable drone mapping software. All the platforms
mentioned here provide a very high level of quality 3D map reproduction
and are compatible with the most popular drone manufacturers including
DJI and Parrot.

SO WHICH IS THE BEST DRONE MAPPING SOFTWARE?

In our opinion, the Pix4D platform offers the best all-around performance across
multiple industries. While it is not the cheapest solution it does offer a complete
range of sensors and map options. It can be seamlessly integrated with back-office
systems using the integrated REST API.

The platform is also very intuitive and easy to use with support from Pix4D rated as
second to none.

So if you’re in the market for a powerful 3D mapping solution for your business, you
won’t go far wrong with the Pix4D platform.

PIX4D: INTRODUCTION TO 3D MAPPING

Pix4D is the leading UAV photogrammetry software, specifically designed for users
who want to make 3D maps and point clouds from data captured during a drone
flight.

The COPTRZ 1-day Pix4D training workshop aims to give you an overview of Pix4D
and how to use the software to give you the data output that you require. The
workshop is ideal for those with no previous experience using the software.
Introduction to SLAM technology with Intel
realsense
Robots and Drones need to understand their location and surroundings
more accurately than GPS allows in GPS denied environments and
inside yet un-mapped spaces. RealSense SLAM enables such
applications.
RealSense SLAM uses a fisheye camera, accelerometer, gyroscope,
and depth camera to track a system’s movement in 6DoF. Unlike
commonly used alternative systems, this depth-enhanced visual-inertial
SLAM can track 6DoF motion without any prior knowledge of the
environment. It also allows a location that was mapped previously to be
recognized–this is called relocalization. Together, tracking and
relocalization allow robots and drones to build and share knowledge
about an environment so they can work smarter, or interact
collaboratively.
The depth camera in the RealSense SLAM solution allows a robot to
build a 2D occupancy map. This map shows where obstacles are vs.
space that is free for the robot to operate in. Application logic can tag
locations in this map or otherwise assign meaning to them. This allows
the robot to answer the question not only of ‘where am I now’ but also
‘how do I get to my destination’.
RealSense SLAM and Dense Reconstruction uses little enough CPU
that it leaves ample power for application logic and other sensor
processing. By giving a robot or drone situational awareness, the ZR300
camera with RealSense SLAM helps build better solutions, faster.
The binaries for RealSense SLAM are distributed with sample
applications to help you get up and running quickly. You can review the
code and see how to correctly configure SLAM with your ZR300
camera’s calibration, and how to pass sensor data to SLAM. The header
files and derived API documentation contain definitions of data types,
inputs, and output.
LIST OF TERMS

 SLAM: Simultaneous Localization and Mapping

 6DoF: Six Degrees of Freedom: three for position, three for
orientation
 Camera pose: The x,y,z position and yaw/pitch/roll orientation of
the camera
 IMU: Inertial measurement unit - in this case accelerometer and
gyroscope
 Module: An instance of the rs::slam::slam class from the Intel
RealSense™ SLAM library.

FEATURE LIST

6DOF TRACKING
Provides a camera pose containing yaw/pitch/roll orientation and x,y,z
position of the camera. The camera pose is provided in real-time with
low latency. The right-handed coordinate system is gravity-aligned and
is defined as: Origin located at the projection center of the fisheye
camera, X axis is horizontal where positive is right, Y axis is vertical
where positive is down, Z is horizontal (optical axis of the camera) where
positive is forward. The world origin is the point where tracking first
began, except when a relocalization map is being used. Updates to the
camera pose are typically provided at 30Hz.

RELOCALIZATION
Relocalization allows an area to be “learned.” The system can then
recognize that it has been in a location before, and report a position
relative to a previously-defined coordinate system. Normally, when
SLAM is started, it has no information about where the device is in an
absolute sense. Tracking is relative to the starting point. However, if the
area has been previously learned, a relocalization map can be loaded
that will allow the system to reorient itself right from the beginning.
Relocalization can also be done on-the-fly without loading a database.
The system learns as it goes, and if it returns to an area it has visited
before, it can recognize it and therefore improve the pose estimation. If
there is a difference between where it thought it was and where it
recognizes it is now, it will correct its estimate of where it thinks it is. This
is useful for correcting drift in the pose estimation, and for recovering
from tracking failures. Relocalization is dependent on successful 6DoF
tracking.

2D OCCUPANCY MAPPING
Occupancy mapping is the process of building a 2D map of an area that
contains information about which areas are occupied by obstacles and
which are open. This can be useful for a robot that needs to know where
it can safely navigate. The occupancy map is a grid where each grid
square has a scalar value indicating the confidence that it is occupied.
The physical size of the grid squares is configurable. The accuracy of
the occupancy map is dependent on the accuracy of the 6DoF tracking.

OPTIMAL SLAM USAGE

IDEAL CONDITIONS FOR SLAM
RealSense SLAM performance is dependent on the environment in
which it is operating. Since SLAM uses 2D and 3D cameras, the quality
of the cameras’ view is important. Ideal conditions are well-lit areas,
without large changes in lighting. The camera must be able to identify
features in the scene like corners or points that it can track from frame to
frame. Generally, a more complex scene with lots of objects or geometry
is best. Blank walls, floors, or ceilings are the worst cases. A lot of
reflective surfaces such as glass or mirrors can be problematic. Also,
direct sunlight can interfere with the depth camera, which impacts the
accuracy of occupancy mapping.
RealSense SLAM works best when the scene is mostly static, meaning
not moving. A person or object moving through the scene will generally
not cause problems, but if there are many people or objects in motion,
performance will suffer. If the entire scene is moving, such as in an
elevator, SLAM will not work at all.
IDEAL MOTION FOR SLAM
RealSense SLAM also relies on the IMU. It fuses data from the camera
and IMU in order to track the position of the camera. RealSense SLAM
works best when the camera motion is mostly translation, as opposed to
rotation. The absolute best case is when the camera is moving sideways
(on the X or Y axes). When moving this way, the parallax from the visual
features in the scene is at its best. The second-best type of motion for
SLAM is when the camera is moving forward or backward in a straight
line (on the Z axis). When rotation is necessary, it’s best to rotate slowly.
However, zero radius turns are difficult for the tracker to follow, so it’s
best to rotate while translating.

STARTING SLAM
When SLAM first begins, the camera must be motionless, and there
must be sufficient visual features in view. If the camera is pointed at a
blank wall, ceiling, floor, or objects that are close enough to block the
camera’s view, SLAM may not initialize properly.

INITIAL MOTION AND TRACKING

ACCURACY
The tracking accuracy as reported by the SLAM module will start off low,
and should improve to medium once some motion has taken place (high
accuracy is not currently used, but is reserved for future use). This initial
motion is important for getting good results. Ideally, the initial motion will
be mostly translation, as opposed to rotation. It should not be very fast,
or very slow. Once the tracking accuracy has improved to medium, more
robust tracking can be expected. This typically requires a few decimeters
of initial movement.

TRACKING FAILURE AND RECOVERY

If the tracking accuracy is reported as failed it will not recover and SLAM
must be reset. To reset SLAM, the same conditions for starting SLAM
must be met (motionless, and so on). Tracking failure can be triggered
by fast rotation, a lack of features in the scene, poor lighting, dramatic
changes in lighting, or vibration.
FLYAWAYS
Flyaways are a SLAM failure mode where the camera position is
reported to be rapidly accelerating in a random direction, when it really
isn’t. This will typically continue for a few seconds then tracking will fail.
Flyaways are often caused by the camera being pointed at a featureless
scene, like a wall. Sometimes it is possible to recover by pointing the
camera at a feature-rich scene. Depending on the application, it may be
more expeditious to simply stop moving and reset SLAM.

CPU UTILIZATION
If the CPU becomes overloaded, sensor samples streaming from the
camera may be dropped in order to keep up. This will have a negative
impact on the accuracy of the SLAM output.

RELOCALIZATION
Relocalization can be done on-the-fly, or with a previously generated
map of landmarks. The system can learn an area, then generate a map
to be used in a future visit. If a relocalization map is available, it should
be loaded before starting SLAM.

DRIFT IN POSE ESTIMATION

Drift in the pose estimate can accumulate while tracking over time. The
farther the device moves, the more drift is likely to accumulate. If a robot
visits a room, then moves to a different room, then comes back to the
first room, some drift may have accumulated. However, when returning
to the first room, SLAM can relocalize and correct the drift that has
accumulated in the pose estimate.

IDEAL CONDITIONS FOR RELOCALIZATION

In order for relocalization to occur, the camera must be viewing the
scene from a similar position and angle as it did when it first viewed it.
Therefore, in order to maximize the probability that relocalization will
occur, a robot should attempt to retrace its path when returning to a
previously visited area, and then “look” in a direction that it has looked
before.
OCCUPANCY MAPPING
The accuracy of the occupancy map is dependent on the accuracy of the
6DoF tracking. This means that drift can accumulate in the occupancy
map, as it does in the 6DoF pose estimation. Relocalization can help
reduce the drift in the occupancy map.

HEIGHT OF INTEREST (HOI)

Occupancy mapping must be configured to use a maximum and
minimum height of interest. The HOI is the vertical distance above and
below the camera that we are interested in knowing the occupancy of.
For instance, if a robot is 30cm tall and its camera is mounted 20cm off
the ground, we might set the maximum HOI to 10cm (30cm – 20cm).
This will tell us if the volume up to 10cm above the camera (30cm off the
ground) is occupied. The volume above that is immaterial, because the
robot can drive underneath any obstacles above 30cm. Similarly, we
might set the minimum HOI to -19cm, assuming the robot can drive over
any obstacle 1cm or less in height.

CHANGES IN CAMERA HEIGHT

Since the HOI is defined relative to the camera, changing the height of
the camera while occupancy mapping is running will change the
meaning of the data in the occupancy map. Assuming the minimum HOI
is set to -19cm, if the camera starts out at 20cm above the floor, then
moves to 10cm above the floor, the occupancy map will show that an
area once considered unoccupied is now “occupied” by the floor. A
similar situation can happen if the camera is moved too close to the
ceiling. The 6DoF tracking pose can be used to detect changes in
camera height and adjust the HOI on the fly. This is not done
automatically, and is up to the developer to implement.

SLAM DEVELOPMENT

From a high level, using the RealSense SLAM library is a matter of continuously
feeding camera and IMU data into the module, and continuously receiving the output
of the module as a camera pose and occupancy map. Additionally, the module can
produce files for relocalization and the occupancy map.