3D FLASH LIDAR VIDEO

5.
GENERATION

OF

3D POINT CLOUD
VIDEO
Given the
large variation in scene
composition, complexity, and imaging conditions, robust generation of detailed 3D scene
models is as hard a problem in computer vision as the capability to do so is compelling There
are variety of techniques for deriving 3D structure from motion, multi-view stereo, and the
associated sub problems of feature detection and correspondence matching.
Another way to generate 3D point cloud representations of a scene is to use a
laser ranging device. At this time these devices generally fall under one of two categories, a
single beam point-scanning technology or a focal plane array (FPA) device [4]. The former
technology is more mature, far more common and far more institutionalized than the latter.
Well-performing FPA flash LIDAR devices are just now becoming commercially available.
While commercially available single beam point-scanning LIDAR can generate very high
resolution 3D samplings of static scenes, moving sensor platforms necessitate the complexity
of independently registering each measured pixel using accurate record of the platforms
motion. Even when this is done properly, registering samples which fall on a moving target is
impossible if the precise motion description of the target is unknown. In many cases, the
motion description of a moving target is precisely the information we are trying to measure,
so in these cases point-scanning LIDAR systems are unsuitable.
Generally, a 3D flash LIDAR by contrast uses an array of independent LIDAR
receivers and focuses light returning from the scene onto the array with a lens system. In
many ways it has a user aesthetic which is exactly like any ordinary digital video camera. The
flash is generated by an on board laser module and a beam-spreading optical element. This is
analogous to flash photography with conventional 2D digital video cameras. Since all pixels

3D FLASH LIDAR VIDEO

function in parallel to each other the motion of the platform and motion of the scene between
pixels samplings is zero.

The drawback of this technology is that the highest resolution commercial

models available as of the time of this work were equipped with 128 x 128 pixel arrays. In
general this resolution is considered low for some computer vision applications, however
array size and resolution is expected to increase as the technology continues to mature. In
spite of the resolution these devices are a very convenient way to generate 3D point cloud
sequences at video frame rates.

6. 3D FLASH LIDAR DEVELOPERS

3D Flash LIDAR camera systems are available from Advanced Scientific
Concept, Inc. (ASC 3D) and Raytheon Vision Systems (RVS), both based in Santa Barbara,
California. While ASC has shipped various iterations of its products for Space (STS-127 and
STS-133), unmanned air and ground vehicles and surveillance and remains as the key
contributor to the technologys development, RVS has limited its activity to the NASA Sensor
Test for Orion RelNav Risk Mitigation (STORRM) Development Test Objective (DTO) tested
on STS-134. Little public information is available for the RVS solution.
ASC 3D Flash LIDAR cameras currently have the equivalent of 16,384
range-finders on each sensor chip, allowing the sensor to act as a 3D video camera with
functionality well beyond just range finding. Capable of mapping ~ million points per
second, with processing done on-camera to generate the 128 X 128 range maps at 30 Hz, has
demonstrated single-pulse 3D Flash LIDAR imagery with a sensor capable of such a wide
range of physical ranges from centimeters to kilometers.
The specific model camera used for this work was designed and manufactured
by Advanced Scientific Concepts Inc. In Santa Barbara, CA (ASC). A manufacturers photo of
this camera is shown in Figure 1.

7. ASC 3D FLASH LIDAR VIDEO CAMERA FEATURES

2

3D FLASH LIDAR VIDEO

Fig.7.1

Range: up to 1.5km

Frame rate: up to 30Hz

Size: 11x6x6inches

weight:10 lbs

128x128 pixel arrays

Power: 110V wall outlet or 12V motorcycle battery

Noise free point cloud video

Laser wave length is 1.57micro meter & pulsed for 5ns per video frame

8. SURFACE CRAFT DETECTOR ALGORITHM

8.1. DATA REPRESENTATIONS

3D FLASH LIDAR VIDEO

Dealing with full motion flash LIDAR point cloud video sequences poses several
unique problems beginning first with how best to represent the data. While each pixel location
in the range image represents a location in 3D space, since a point behind a point cannot be
measured with the camera technology we are using, the data is not actually volumetric in
nature. For this reason some researchers describe this kind of data as 2.5D. The output of the
ASC flash LIDAR video camera is interpretable as a range measurement and an intensity
measurement for each pixel on the array. This can be conceptualized as a range image and a
co-registered intensity image. The intensity image is useful for fusion with 2D intensity
camera data. The range and intensity images are represented on a regularly spaced 2D grid
defined in pixel coordinates. Data in the form of 2D scalar arrays such as these are
convenient. They lend well to spatial filtering using finite length kernel operators, however
for our application this is not a necessary operation and not quite what we want. Another way
to represent the data is to use parameters from the camera and lens system to orthorectify the
data. Using a basic triangle approximation of the imaging system the data is sufficiently
orthorectified. After orthorectification, the samples in each range video frame are represented
as an irregularly spaced unoriented point cloud. The irregular spacing comes about as a result
of the spatial sampling frequency of the scene decreasing as a function of range. This is
illustrated in Figure2.

3D FLASH LIDAR VIDEO

Fig.8.1
One approach to operating on irregularly spaced data is to interpolate the
data onto a regular grid and treat it as regularly spaced data. This method is not suitable for
our task because it produces undesirable artifacts at edges between foreground and
background objects which result in erroneously connecting the foreground with the
background later on. Two other methods for dealing with irregularly spaced data is to
interpolate it locally with piecewise continuous functions or to devise operators which can
work directly on irregularly sampled data. These two methods are more suitable for our task.
Local interpolation using a continuous model is used in step 1 of the algorithm and operators
which work on irregularly spaced data are used in step 2.

8.2. ALGORITHM STEPS

8.2.1. FINDING THE GROUND PLANE:

3D FLASH LIDAR VIDEO

Fig.8.2

Fig.8.3
Figure 3 shows orthorectified point cloud data displayed on 3D axes. The

sampled scene depicts a man walking between two fixed lamp posts, and is an example of one
range image in a several second long sequence of range images sampled at 5 Hz. First we
would like to locate the ground plane. Without filtering or classifying the data points in any
way, we can do this by solving for the best fit 3D plane in the data. To do this we use a least
squares orthogonal distance minimization inside of a Random Sample Consensus (RANSAC)
construct. Orthogonal minimization is chosen because we assume error in all three dimensions
of our irregularly spaced data. RANSAC is used so the minimization process is robust to
outliers, which for this fitting would include all sample points not part of the ground. These
points include noise, but also the targets at this stage in the algorithm. Briefly, the RANSAC
procedure is to choose a random subset of the data and to fit a plane to this subset using least
squares. Next, a vote is computed by counting the number of original data points which fall
within some tolerance of the plane. The process is repeated for some number of iterations
using a different random subset of the data each time. Whichever model at the end of the
iterations has the highest vote is chosen as the best fit.

8.2.2. FINDING THE TARGET CANDIDATES:

3D FLASH LIDAR VIDEO

Fig.8.4

Fig.8.5
The 3D plane generated by this robust model fitting is shown superimposed on

the data in Figure 4. All sample points which fall within a small distance from this best fit
plane are removed. Figure 5 shows what is left over. These samples are the target candidates
plus noise; the objects not part of the ground plane. A neighbourhood filtering operation is
applied to the data, whereby at each remaining sample point, the number of neighbouring
points within some radius are determined. If the number of neighbouring points is less than
some small tolerance, the sample point is removed from the dataset. Since we already know
that our spatial sampling rate decreases as a function of range, the neighbourhood search
radius increases as a function of range. Due to this feature, the filter will operate evenly on an
irregularly spaced data such as this. The result of the cleaning filter is shown in Figure 6.

8.2.3. SEGMENTING THE TARGET CANDIDATES:

3D FLASH LIDAR VIDEO

Fig.8.6
At this point we have isolated the sample points which are candidates
for being targets, but how many targets are there? A K-means clustering algorithm is used to
segment and label the candidate samples as separate targets. In statistics and data mining, kmeans clustering is a method of cluster analysis which aims to partition n observations into k
clusters in which each observation belongs to the cluster with the nearest mean. In this case
there are 3 targets detected, the two lamp posts and the man.

9. TARGET TRACKING

3D FLASH LIDAR VIDEO

When the surface craft detector algorithm is performed on each range image in
a sequence of range images all of the necessary information for target tracking is provided.
The centroids of each target sample cluster can be used in conjunction with a tracking
algorithm to provide robust estimation of the target locations in a noisy environment and in
between measurements. The implementation of the surface
craft detector described in Section III-B and used to generate the result depicted in Figure 7
was written in MATLAB and was run on a low end consumer desktop computer. It was not
optimized for real time performance. However, the algorithm presented is simple enough that
it will function in real time if the range images are streamed at 5 Hz. The locations of each
target on the 2D plane can be considered noisy observations and can be refined by an
estimation filter. Figure 8 shows the results of filtering the noisy measurements with a basic
2D Kalman filter. A more sophisticated tracking algorithm and a higher sample rate would
refine this already useful result.

Fig.8.7

10. APPLICATIONS
Automotive

Autonomous Navigation

Collision avoidance
9

3D FLASH LIDAR VIDEO

Aviation

Helicopter brownout

Wire detection

Landing zone mapping

Defence

Unmanned ground vehicles

Unmanned air vehicles (UAV/UAS)

Laser trackers

Marine Robotics

Unmanned Ground Vehicles

Industrial Space

3D FLVC have been tested on STS-127, STS-133 and STS-134 and are planned for
deployment by SpaceX for Autonomous Rendezvous & Docking on their Dragon
Vehicle

Autonomous Rendezvous & Docking

On-orbit Satellite Servicing

Entry Descent and Landing

Autonomous Landing and Hazard Avoidance, Collision Avoidance, Situational

Awareness

Surveillance

Threat Detection

Object ID

Topographical Mapping Transportation

Train collision avoidance (through fog/obscuration), track profiling

Airport bird detection

Traffic control and monitoring system

10

3D FLASH LIDAR VIDEO

11. 3D FLASH LIDAR CAMERA FOR FUTURE NASA MISSIONS

One of the greatest challenges faced in robotic planetary exploration is
achieving a safe landing. In the case of Mars, fairly good images of potential landing sites are
available from the Mars Global Surveyor mission. However, the challenging process of
atmospheric entry and descent to the surface does not result in an exact landing arrival
location. The time it would take to transmit an image of the approaching landing site back to
Earth and new guidance information transmitted back to Mars is far too long for any
intervention in the landing process. The problem was addressed in the first Mars landings by
packing the early rovers in an inflated protective shell that deflated and opened after landing.
There were risks that the inflated package would come to rest with the rovers in the wrong
position but, fortunately, this never happened. This approach is not practical for larger, heavier
spacecraft such as the Mars Science Laboratory or a future Mars sample return mission. Thus,
a system that will make it possible for the spacecraft to guide its own landing is required.
Crucial to an autonomous landing capability is accurate measurements of the
distance to the approaching landing site and a 3D image of the surface location. This
information will allow the spacecraft to autonomously adjust its landing site to the safest
possible location. Advanced Scientific Concepts, Inc. (ASC) has developed a 3D imaging
Flash LIDAR camera that can meet the requirement of this mission. The ASC 3D Flash
LIDAR camera can generate up to 30 3D images per second at ranges from 10,000 to 1 meter
by capturing the time-of-flight round trip of a laser pulse, providing range distance on each
pixel in its focal plane array with each laser pulse. This range data is used to create a real-time
3D representation of the scene as well as determine velocity. Comparisons of images with
stored map information can be used to determine general location. Moreover, software is
being developed to use these images to avoid landing hazards.

11

3D FLASH LIDAR VIDEO

3D Flash LIDAR camera with3 and 9 degree lenses and 15mJ laser (upper right)
Fig.11.1
ASC participated in NASA's Autonomous Landing and Hazard Avoidance
Technology (ALHAT) program where similar safe landing challenges are faced on a mission
to the Moon. The program's objective is to establish a capability to land near specific
resources that are located in potentially hazardous terrain on a future lunar mission. An active
sensor for measuring the topography of the landing site (3D Flash LIDAR) and terrain
analysis algorithms make this possible. The ALHAT program used field tests and high fidelity
Monte Carlo simulations of a lunar landing to evaluate the capability of these technologies
working in concert.
ASC participated in the COTS Autonomous rendezvous and docking (AR&D)
program. On July 15, 2009, ASCs 3D Flash LIDAR camera was tested on the Endeavor
Space Shuttle mission to the International Space Station becoming the 1st 3D Flash LIDAR in
space and again February 24, 2011 on STS-133 successfully demonstrating 3D Flash LIDAR
for AR&D applications. Combined with appropriate software to interpret the images obtained
from the Flash LIDAR, directions can be given to the spacecraft propulsion systems to guide
the spacecraft to a safe rendezvous and docking. This demonstration was supported by Space
X which is developing a vehicle for delivering supplies to the International Space Station.

12

3D FLASH LIDAR VIDEO

ASC is enhancing its 3D Flash LIDAR sensors to meet the requirements for
Planetary Entry, Descent and Landing by (EDL) to produce the real-time 3D sensor that will
be deployed on future lunar, asteroid and planetary missions. Working on a Phase II SBIR,
ASC expects its new 3D sensor packaging will survive through 100kRad and be ready for
integration in 2012 to launch aboard upcoming space explorations, according to Advanced
Scientific Concepts, Inc. (ASC). Designed especially for Automated Rendezvous and Docking
(AR&D) and On-orbit Satellite Servicing (OSS), it was most recently tested on the STS-133
Discovery in February 2011 and has been listed as Technical Readiness Level (TRL) of 8.
11.1. DRAGON EYE-THE FIRST 3D FLASH LIDAR IN SPACE

DRAGON EYE
Fig.11.2

Capable of capturing a full array of 128x128 independently triggered 3D range

pixels per frame up to 30 frames per second in real-time, Dragon Eye is the first 3D FLC in
space. The non-mechanical Dragon Eye boasts ASC's leading-edge technology advantages,
including its Class I eye-safe lasers for illumination, real-time imaging without motion
distortion to provide real-time input for guidance, navigation and control (GNC) systems.

13

3D FLASH LIDAR VIDEO

The Dragon Eye 3D Flash LIDAR camera opens many doors for 3D FLC in
space, making both manned and unmanned AR&D possible.ASC 3D FLC systems can be
found in a wide range of applications, including Automotive, Defence, Surveillance, Robotics
and Aviation. ASC's patented, unique technology brings tremendous value to space-based
applications.

13. CONCLUSION

We have demonstrated in proof-of-concept the use of a novel 3D Flash LIDAR

video camera technology in a target detection, segmentation, and tracking scenario. By using
a sensor which generates point cloud video sequences directly we have circumvented the
problems associated with structure from motion and multi-view stereo 3D model generation.
With marginal optimization and refinement this technology and the associated processing
algorithms can be used to fill a capability gap in many perimeter security and site surveillance
applications. Given the hardware as it exists today, we feel this could provide robust detection
and tracking of human-sized and vehicle-sized targets which is simpler, less expensive, and
far superior to existing systems which attempt to accomplish the same objective.

14. REFERENCES

www.ieee.com

www.wikepedia.com

www.advancedscientificconcepts.com

www.vision-systems.com

B.K.P. Horn. Robot Vision. Boston: The MIT Press, 1986.

R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision, 2nd ed.
United Kingdom: Cambridge University Press, 2003.

14