Realtime Video Stabilization For Moving Platforms

Paper : Real-time Video Stabilization For Moving Platforms
21
st
Bristol UAV Systems Conference April 2007 Page 1 of 14

Real-time Video Stabilization For Moving Platforms

Eddy Vermeulen

Barco nv

Eddy.vermeulen@barco.com
Tel: +32 (0)56 233 202

ABSTRACT

This paper describes a method and practical implementation to stabilize video sequences obtained from a
camera mounted on a moving platform. Specific for these platforms is a twofold camera motion. The first one is
from the motion path of the vehicle itself which has to be followed as closely as possible. The second one is due
to mechanical vibrations of the mounted sensor and/or environmental influences. The second motion
disturbance has to be filtered out as much as possible from the camera image. From a market perspective, the
paper also discusses the need for electronic stabilization in defense & security markets, as well as some future
impacts of technology and market trends.

BIOGRAPHY

Eddy Vermeulen has worked as a product manager for the international market in the defense and security
market unit of Barco since 2004. As product manager, he supports and helps define imaging and network-
centric products. He has worked at Barco in various product support positions in the defense, security, and air
traffic control markets since 1999. Prior to Barco, he worked in technical support positions in the commercial
computing and networking markets.

21
st
Contents

Real-time Video Stabilization For Moving Platforms ............................................................ 1
ABSTRACT.................................................................................................................. 1
BIOGRAPHY ............................................................................................................... 1
Contents................................................................................................................................ 2
Introduction .......................................................................................................................... 3
The Need for Real-time Stabilization .................................................................................... 3
UAV / UGV / USV Platforms ........................................................................................... 3
Ground Vehicle Platform................................................................................................... 3
Naval Platform.................................................................................................................. 4
Mechanical Versus Electronic Stabilization....................................................................... 4
The Stabilization Process ...................................................................................................... 6
Overview .......................................................................................................................... 6
Registration....................................................................................................................... 6
Filtering the Global Motion Parameters............................................................................. 6
Warping ............................................................................................................................ 7
Video Output Formatting .................................................................................................. 7
Optional Generic Warp Pre-processing.............................................................................. 8
Controlling the process...................................................................................................... 9
Practical Implementation..................................................................................................... 10
Overview ........................................................................................................................ 10
Component Implementation : Computational Performance of FPGAs ............................ 10
Board Implementation : Acquisition and Visualization.................................................... 11
System Implementation : External Control & Tuning ...................................................... 12
Future Impact of Current Trends ......................................................................................... 14
Sensor Resolution ........................................................................................................... 14
Processing Components................................................................................................... 14

21
st

Introduction

This paper describes a method and practical implementation to stabilize video sequences obtained from a
camera mounted on a moving platform. Specific for these platforms is a twofold camera motion. The first one is
from the motion path of the vehicle itself which has to be followed as closely as possible. The second one is due
to mechanical vibrations of the mounted sensor and/or environmental influences (e.g. wind). The second motion
disturbance has to be filtered out as much as possible from the camera image.

The first part of this paper looks into the need and advantages of electronic video stabilization, specifically
related to the defense and surveillance markets. The second part describes a stabilization process and associated
tuning issues. The third part handles the physical implementation. Finally, some future impacts of technology
and market trends are considered.

The Need for Real-time Stabilization

UAV / UGV / USV Platforms

UAV platforms used to be mainly intelligence gathering platforms, obtaining aerial digital still images or
making video recordings for analysis or mapping purposes once the vehicle was recovered. Today, an increasing
number of UAV platforms are being equipped with streaming video solutions for immediate observation. This
has become possible due to the increasing payload capabilities, increasing data communication bandwidth, and
improvements in video compression technology.

The result is a wider range of applications that require direct view capabilities. A soldier in the field benefits
from fast environmental reconnaissance without going into the risk area himself. A UAV that is deployed for
border patrol can now provide live tracking of suspicious movement to the operator in the ground control
station.

The value of motion video information from mobile platforms may, however, be affected by the uncontrollable
motion of the sensors. All of these platforms have to deal with mechanical vibration caused by engine parts.
UAV mounted sensors are unstable due to unexpected roll, yaw, and pitch caused by atmospheric conditions.
UGV mounted sensors suffer from terrain roughness. USV mounted sensors deal with unpredictable wave
motion. Detecting or tracking small targets of interest in this unstable motion video is very difficult and
exhausting to an operator.

Remote platforms often use radio links to transfer video. The limited bandwidth of these carrier signals requires
video compression prior to the transfer of the signal. A common compression technique such as MPEG relies
heavily on intra-frame compression. Instead of transmitting full picture contents, only differences between
sequential frames is encoded and embedded in a compressed video stream. As stabilization will eliminate
sudden jumps from a video sequence, it will allow a more efficient encoding when applied at the sensor
platform.

In all of these applications, stabilized video improves the image quality and results in improved detection
capabilities of the operator.
Ground Vehicle Platform

Army warfare today is shifting from a symmetrical to an asymmetric threat. Conflicts are no longer between two
nation armies on a well-defined battlefield, but rather small dispersed units of individuals that move in
unpredictable ways. The battle scene has moved to remotely deployed units, often in urban territory with an
environment of low visibility and possible threats around every corner. With this shift in operational
environment, it is of no surprise that many army programs worldwide are focusing on small combat vehicles,
rather then large main battle tanks. Examples of such programs are Future Combat Systems (FCS) in the US,
Future Rapid Effect System (FRES) in the UK, Splitterskyddad EnhetsPlattform (SEP) in Sweden, and Bulle
Oprationnelle Aroterrestre (BOA) in France.
21
st

These vehicles must be able to operate fully enclosed, securing the crew inside the armor protection. During
such operations, driver and commanders have to rely on externally mounted electro-optical sensors to view the
environment. Engine vibration and rough terrain cause a lot of vibration in these images making it hard to
interpret the imagery or to drive the vehicle. Many of the sub-programs are looking into enhancing local
situational awareness and driver viewing systems.

Also, turret modules and weapon systems are often controlled through a long-range gimbal system. Firing
weapons is probably the worst cause of high frequency vibration of those sensors.
Naval Platform

Surveillance camera's on ships are typically used with low FOV to scan the horizon, the coast (in case of littoral
operations), or the sky for threats. They are typically mounted on high masts that are subject to mechanical
vibration, the movement of the sea, and atmospheric conditions.

Also, gun control is often done remotely in the command center of the ship, based on gun-mounted cameras.
These are subject to high vibration when in use.

In fact, images from naval platforms are often the most difficult ones to stabilize because of the low contrast
image content (think about dark weather conditions, looking at a grayish water horizon against a misty sky).

Mechanical Versus Electronic Stabilization

With most cameras mounted inside gimbals offering mechanical stabilization, one may wonder whether there
still is a need for electronic stabilization. There are several reasons why mechanical stabilization does not
exclude the use of electronic stabilization.

First, mechanical stabilization does not remove all image vibration. The capability of a gimbal to compensate
for vibration is typically specified by the minimum angular movement that can be mechanically stabilized. It is
measured in micro-radians. Based on this stability limit, the camera resolution and the field of view, one can
calculate the remaining picture movement in pixels at the observing workstation as illustrated in the tables
below.

Horizontal transition in pixels 768 x 576
Stability (uRad) 5 10 20 50 75 100 200
FOV ()
0,25 1 2 4 9 13 18 35
0,3 1 1 3 7 11 15 29
0,5 0 1 2 4 7 9 18
1 0 0 1 2 3 4 9
5 0 0 0 0 1 1 2
10 0 0 0 0 0 0 1
20 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0

Horizontal transition in pixels 1920 x 1080
Stability (uRad) 5 10 20 50 75 100 200
FOV ()
0,25 2 4 9 22 33 44 88
0,3 2 4 7 18 28 37 73
0,5 1 2 4 11 17 22 44
1 1 1 2 6 8 11 22
5 0 0 0 1 2 2 4
10 0 0 0 1 1 1 2
20 0 0 0 0 0 1 1
30 0 0 0 0 0 0 1
camera resolution
NTSC : 640 x 480
PAL : 768 x 576
HD : 1280 x 720
full HD : 1920 x 1080
Field of View
0,2 - 30
Stability = Minimal
mechanically stabilized
angular movement
10 - 100 microRadians
21
st
The rows in these tables represent different field of view (FOV) operations. The higher an operator zooms in
with the camera, the lower this FOV is and the smaller the observed region is. For an airborne platform such as a
UAV looking at specific targets on the ground, a very narrow FOV (0,25 to 5 degrees) is quite common. The
columns in these tables represent different stability specifications, the capability to mechanically compensate for
vibration. This is a gimbal specification. The calculated values represent the number of pixels a video image is
shifted as a result of the remaining uncompensated movement. Values of 0 represent a sub-pixel displacement.
This calculation is presented for two types of sensor resolution : a typical PAL camera and a high resolution HD
sensor. For overview purposes, the resulting image shifts are divided in three regions : sub-pixel vibration, small
displacements (less then 4 pixels) and larger displacements.

Per these calculations, a narrow FOV of 1 on a PAL camera and a common stability of 100 uRad results in a
picture shift of 4 pixels. In a market trend targeting higher resolution EO-sensors, the effect will increase. A full
high definition sensor would have a remaining vibration of 11 pixels under the same circumstances.

A second reason in favor of electronic image stabilization is cost. High performance gimbals can provide
stabilization as good as 10 micro-radians, however, this kind of optical system is extremely expensive. A more
affordable gimbal system typically has a mechanical stabilization specification of 100 micro-radians. Cost
considerations drive the need to add electronic stabilization.

Thirdly, the more advanced mechanical stabilization solutions come with a certain weight and volume impact.
On an airborne platform such as a UAV, payload weight is a critical factor that affects range and endurance. Not
only does electronic stabilization come at a lower weight when embedded in the platform, it can also be
integrated at a remote viewing position. This post-processing option eliminates all weight and volume impacts
on the moving platform itself.

A further advantage of electronic stabilization is its flexibility in control. As opposed to mechanical
stabilization, an operator at the observation console, or smart software, can dynamically adjust or even enable /
disable the stabilization process and its parameters. Software can detect operator interactions such as panning
and zooming and apply immediate changes in the movement compensation of the image. This avoids the typical
transition latency caused by the algorithm trying to compensate an intentional motion. Speed and change in
direction of the platform can be dynamically used to alter the motion compensation in selected directions or at
expected shift speeds.

In certain cases, operators may want to disable the stabilization processing because an unstabilized image may
provide more information. This is best explained by the example application of looking through a grid, such as a
gate with many vertical bars. When moving laterally in front of such an object, the human mind integrates the
different picture frames and 'memorizes' them, which will give the impression of what one sees behind the bars.
A stabilized picture will partially hide the details behind the intersections and may, as such, hide important
visual information.

One can conclude that mechanical stabilization does not exclude electronic stabilization; rather there are good
reasons, depending on the platform environment, to choose a combined solution where both methods
complement each other's weaknesses.

21
st

The Stabilization Process

Overview

Electronic video image stabilization is basically a three-step process. The first step is called image registration,
which is estimating the global motion parameters. The second step is applying a filter on these motion
parameters to filter out the motion of the vehicle. Finally, a warping function is applied to the input image with
the inverse of the previous motion vectors. After this process, a final video output formatting is performed to
handle the borders of the warped image for which no current information is available. The described solution
also includes an additional pre-processing warp function which can optionally be applied to deal with lens
distortions. The whole process needs to be controllable and tunable, preferably in a dynamic way to adapt to
changes in environment and to user interactions.

Registration

Displacement of one frame to the next is defined by a horizontal translation, a vertical translation, and a rotation
component. The task of the registration part of the algorithm is to find these 3 global motion parameters to
which a new incoming frame must be subjected to fit as closely as possible to the previous frame.
In fact, a perfect match will not be possible because in practice, when the camera is moving, the background is
never stable at all. There might even be objects in the scene that are physically moving. The goal of a good
stabilization algorithm is to follow the moving background as closely as possible but to filter local motion in the
output sequence that is not caused by the movement of the camera.

The registration algorithm starts from an estimate of the global motion parameters. The new frame is then pre-
warped and compared to a reference frame. The remaining motion will be small and provides correction factors
for the estimated values. After a few iterations of this process, the remaining motion parameter error should be
close to zero under conditions that most of the pixels (true for the background) have the same translation and
rotation.

It is important to perform the registration on the whole frame or on a large enough, configurable region of
interest inside the picture. Not doing so may cause the stabilization to operate erroneously. Imagine for example
what would happen if an object-oriented stabilization algorithm considers a drop of water or dust on the lens to
be the object of interest. This is especially true for rugged environmental conditions such as in the defense
market. Considering the whole image content can be a huge computational load for any device to accomplish on
moving video at 30 frames per second. This can be simplified by a smart selection of the registration algorithm
and working on an input frame with decreased resolution.
Filtering the Global Motion Parameters

From the calculated motion parameters, an apparent motion velocity and frequency results. The intention is to
follow the slow camera motion but compensate for fast mechanical vibration and unexpected changes. In this
global motion
parameters
Filter Registration
Warp
Input Frame Output Frame
Pre-
processing
Warp
21
st
step, a low-pass filter is applied that maintains the low-frequency motion in the picture and rejects the high-
frequency motion at a selected cutoff frequency.

The implementation of the algorithm must allow adjustment of the cutoff frequency to be able to adapt to
different types of platforms and different expected velocities. In fact, these frequencies can be changed
dynamically while the stabilization process is continuing, and can be changed separately for each translation and
rotation. This dynamic tuning can either be automatic, based on the average motion history of the picture or
manually forced by means of a controlling software application. At the same time, the compensation is
monitored for extreme corrections. This insures the end-result is never worse then the unstabilized picture,
avoiding a self-oscillating effect that can theoretically occur in any closed loop process.
Warping

Image warping is the geometrical mapping of an image from one spatial domain onto another. It looks like a
projection of an image to another surface in a 3D space. This is a common process in graphics boards to apply
textures (like a wallpaper) to a 3-dimensional object as illustrated below.
This process is used in the stabilization algorithm to map a new incoming frame back to the position of the
previous frame. It is also used, as described below to fill borders and to optionally compensate for lens
distortions.

Video Output Formatting

The video is now stabilized but the missing data in the borders must still be filled in. This can be accomplished
in several ways. The simplest solution is to fill this data with a predefined colour. By applying a scaling function
to the image, these borders can be kept to a minimum. A second method is to keep the old information of the
previous video frame(s). The third one is to warp the old information. The last method is preferred over the two
others since this one gives a more natural viewing result for the same computational effort as the second and a
much better result than the first.

Optionally, by displaying the stabilized video in a larger frame and preserving the information of previous
frames for a longer time, one can visualize a history trail. This kind of picture composition is commonly referred
Fill Color
Fill with
old frame
Warp old
frame
Warp
Transformation
21
st
to as mosaicing and is especially useful for UAV reconnaissance applications. While this format may add to the
comprehensiveness of the image, it should be noted that it may also introduce artifacts due to moving objects in
the scene. In some cases, displaying non real-time information on a real-time system may even be prohibited.

Optional Generic Warp Pre-processing

A typical camera lens is not geometrically perfect and can induce all kinds of distortions such as barrel or
pincushion distortion.

Application experience shows that the impact of those distortions on the stabilization process is generally so
small, that we can neglect them completely for most applications.

However, when the distortion becomes too big, the same warping process as described above can be used as a
pre-processing step to eliminate the distortions as illustrated below. This step is optional and can be enabled or
disabled on demand.

Object Pincushion distortion Barrel distortion
Warp
Transformation
21
st
Controlling the process

Controlling the full stabilization process involves controlling a number of parameters that apply to the individual
sub-processes described above :
- Controlling the location and size of a region of interest for the registration process in order to :
o force the stabilization to focus on objects of interest inside the picture frame
o ignore fixed parts of the input signal e.g. inserted overlay graphics
- Controlling the iteration process of the registration. More iteration may give a better result but increase
the processing time.
- Setting the cutoff frequency of the filter process either automatically or user-enforced. This can be of
interest when the nature of the vibration is known in advance, or when the vibration frequency is very
close to the motion of the vehicle
- Defining the maximum correction of the warping algorithm (which is essentially defining the
maximum size of the fill borders) in order to :
o ignore known intentional picture movement in either translation or rotation (e.g. turning of a
vehicle, climb or descend of a UAV)
o avoid over-compensation
- Enabling / disabling the process to avoid the processing delay e.g. when the vehicle is known to be
stationary or the vibration source is gone.

By implementing the stabilization so that it reads these parameters at each operation from modifiable registers,
the process can be controlled dynamically. A change in parameters will be effective at the next input frame.
21
st

Practical Implementation

Overview

The process described above is computationally quite intensive. To perform this in a real-time way, dedicated
video processing hardware is required.

Some additional functionality is required to implement electronic video image stabilization as a full solution :

- Acquiring the video input signal : While many digital camera interface standards are appearing in the
market, a lot of basic cameras are still analog composite output. An accurate digitization is the first
important step to a useful picture. If the video is from a remote platform and transmitted over a limited
bandwidth link, some decompression may be required to re-create the video signal.
- Visualizing the stabilized video : An operator console is typically composed of video and graphical
information such as maps, instrumental information, etc. In some cases, graphical information needs to
be superimposed on the video image such as target markers and geo-referencing information. Mixing
the stabilized video inside this graphical user interface and providing graphics overlay capabilities is
essential in this solution.
- External control of the process : In most cases, enabling stabilization is not always required. When a
camera is moved to a different view or while the lens is being zoomed in to a region of interest, the
stabilization process would inject an unwanted delay in the operator view. Furthermore, the process can
benefit from information provided by other sensors such as gyro or operator initiated tuning. A
complete system implementation should interpret different input signals and enable/disable the
stabilization as appropriate, or even dynamically control and tune the process.
Component Implementation : Computational Performance of
FPGAs

The stabilization scheme as described above involves a large number of pixel transformations to be applied on
the horizontal camera image grid, especially in the registration and warping phases. While the nature of the
transformations is relatively straightforward, the number of individual operations is large and proportional to the
resolution of the sensor. This kind of operation benefits from the massive parallel computational capability of
field programmable gate arrays (FPGAs).

An FPGA is essentially a silicon device that incorporates a large amount of programmable logic functions such
as AND, OR, XOR, NOT. Specific function blocks are generated by properly programming the interconnects of
these logic components. Different function blocks can be programmed to perform different types of operations
and a large number of equal function blocks can be programmed to perform parallel operations on large blocks
of input data.

The quick evolution of silicon technology has lead to a constant growth of the number of logic gates available in
one FPGA device that basically doubles every 18 months. An additional consequence of this is the affordability
of these devices, providing increasing performance characteristics.

Another recent trend in FPGAs is the occurrence of hybrid technologies which combine embedded processors
and related interface functionalities with FPGA architectures in a single device.

The programmable nature of an FPGA allows easy modification of the embedded algorithm to perform different
functions or easy tuning for specific requirements, without expensive hardware modifications or upgrades. This
property guarantees the value of the long-term investment typically associated with defense systems. It also
supports the trend for spiral development initiatives of many system integrators and defense departments.

The Xilinx Virtex-II Pro family of FPGAs provides a number of features that are well suited for this application
:
- a large capacity, capable of handling the large number of transformations required for high resolution
motion video stabilization
21
st
- a high speed serial interface to quickly move around the large amount of data associated with digital
video
- an embedded PowerPC to control the process and handle overall computational functions

Board Implementation : Acquisition and Visualization

A practical implementation of real time video stabilization is found in Barco's FlexiVision III product platform.
FlexiVision III is a single-board solution for real-time video and graphics mixing, display recording, real-time
video streaming and image processing such as stabilization.

Mixing Block
Control Block
(PowerPC)
8 bit parallel RIO
Gigabit Ethernet 1
Gigabit Ethernet 2
Master Graphics
In
Master Graphics
Out
Universal Frame
Grabber
Function Block II Function Block III
P
C
I

o
r

V
M
E
Video In Video In Video In
S
e
r
i
a
l

P
o
r
t
Universal
Frame
Grabber
(UFG)
JPEG2000
Codec
(J2K)
Programmable
Video
Processor
(PVP)

It is made up of four modular components that can be combined in a variety of ways to perform various
functions e.g. video acquisition, stabilization and windowing :

- The Carrier board is based on a RapidIO switch fabric architecture that is tightly coupled with a local
on-board PowerPC. The hardware is designed to support three optional mezzanine modules, one of
which is a fixed framegrabber. The control block contains the on-board PowerPC and runs an
embedded Linux operating system.
- The Universal Frame Grabber (UFG) module is a dual channel analog video frame grabber that is
capable of supporting virtually any standard analog video format. This part performs the front-end
sensor acquisition when dealing with directly connected analog sensors.
- The JPEG 2000 (J2K) module is a hardware JPEG2000 based codec module. With this wavelet
domain-based compression technique, it is possible to select the compression level from lossless to
very high, and transport video over industry standard TCP/IP networks with low latency. As this
encoding algorithm is quite computationally intensive, it should be performed in hardware for real-time
systems. This part allows for networked acquisition of remote sensors, rather then direct connectivity.
- The Programmable Video Processor (PVP) module accepts digitized video from any of the other
modules and processes it with its on-board FPGA. The PVP provides the core for the image processing
capabilities and is where the stabilization algorithm resides.

21
st
Operator Workstation
FlexiVision III carrier
PVP
Graphics Board
Master In
Master Out
UFG
Unstable Video
Source
Video In
Stabilized Video With
Graphics Overlay

For a video stabilization solution, the UFG module may capture any direct analog sensor input, or the J2K
module may uncompress a networked JPEG2000 video stream. A PVP module can be loaded with the
stabilization algorithm, and the outgoing video can be combined in real-time with computer-generated graphics
and overlay to provide a complete operator workstation solution.

Latency is extremely important for real-time systems, especially those that perform interactive control, based on
the visual information. E.g. a remote pilot of a UAV expects to see an immediate effect if he makes the vehicle
turn. A long processing delay causes the image to lag behind the actual maneuvering commands. This would
appear very confusing to a human operator and makes the system practically unusable for direct control. The
single board implementation described above allows the stabilization and mixing to occur within the time
window of 2 to 3 frames, which is practically unnoticeable to a human observer.

System Implementation : External Control & Tuning

Full system integration of video stabilization involves software for control and tuning. The figure below is a
sample high-level diagram of such a system for vehicle control, and the overall layout is applicable to most
applications.

Pilot Control
Vehicle Control
sub-system
Visualization
sub-system
Control
User Interface
sub-system
Control
Graphics
Video
GPS, Gyro, ...
Vehicle Platform
Control
GPS, Gyro, ...
Communication
sub-system

In this diagram, we consider 4 different processes :
21
st
- The vehicle control sub-system handles the driving of the vehicle. This can be an automatic system or
can be based on user-input through a piloting control interface. Often, this module loads a pre-
programmed mission into the vehicle for further autonomous operation. As a feedback, the vehicle
control module receives actual positional information from the vehicle.
- The communication module handles all communications with the vehicle in both directions. This could
be over radio-link, satellite link, or direct wired control in the case of small ground robots.
- The user interface generates graphics information to be presented to the operator and translates the
operator input to control operations of the other modules.
- The visualization sub-system is responsible for combining user-interface graphics and sensor
information to generate the actual operator picture.

In this diagram, the stabilization process is part of the visualization sub-system. It receives video through the
communication module, via either a direct analogue or digital signal or via a compressed video stream. The
stabilization can be controlled by inputs from the other sub-systems :

- An operator may decide to enable / disable the stabilization process by clicking an on-screen button. He
can indicate the region of interest on the screen by drawing a box over the video. In these cases, the
user interface sub-system controls the stabilization.
- If the vehicle control is aware of the intentional motion of the vehicle, either through pilot control or
through automatic guidance, this information can be used to change the way the stabilization process
works to optimize the result. E.g.: if the vehicle and its attached sensor move to the right, the
horizontal motion compensation can be temporarily minimized.
- Positional and movement information from the vehicle itself can be used to optimize the stabilization
process dynamically and in an automated way.

Video latency of the complete system now has to be considered over the whole transport chain. Each element
may contribute to the total latency of the video visualization. In fact, any of the subsystems described typically
contains input- and/or output-framebuffers. A video frame that enters the sub-system line by line is stored into
these memory blocks for processes that require a full field or frame on which to operate. As such, each of those
buffer memories may add an additional field or frame of delay. Even the display device itself typically contains
up to 3 buffers delaying the actual visualization of the picture. Hence the importance of keeping latency
minimized in each single block of the system.

If a direct connection to the sensor system exists, one typical method to minimize delays is synchronization of
all sources. A separate synchronization signal is distributed throughout the system that makes sure all sources
are generated at a common pace. A common approach is to use the refresh rate of the user interface graphics
and insure it happens at the exact same moment a new video frame is generated.

21
st
Future Impact of Current Trends
Sensor Resolution

A common trend in all EO-sensor devices is an increase of the picture resolution, mainly driven by the
commercial television world. While PAL and NTSC resolutions are still quite common today, new devices are
coming into the market with high-definition (HD) resolution. The term HD is commonly used for every sensor
resolution which is higher then the common PAL and NTSC, covering a wide range of different resolutions. Full
HD refers to the highest resolution (1920 x 1200), but most HD sensor devices are still limited to 1280 x 720.

As already indicated in the table on page 4 of this paper, an increase in resolution, without a change in
mechanical vibration tolerance, will result in more pixels being shifted. As such, the importance of stabilization
grows in order to take advantage of the higher level of detail that an HD sensor provides.

Increasing resolution also brings additional difficulty to the processing functions. Note that doubling both the
horizontal and vertical number of pixels results in four times the number of pixels to be processed. Full HD
results in 7.5 times the pixel quantity of NTSC. Not only does this exclude software PC-based processing, in
requires very high performance of dedicated hardware.

A high resolution sensor only brings a benefit if it is also visualized to its fullest extent. The trend for higher
resolution display devices has been ongoing for a number of years. This also places higher requirements on the
output stage of the visualization sub-system. Combining graphics with live sensor data now also requires higher
bandwidth mixers.

The input stage of the visualization sub-system is also subject to some changes. Higher resolution comes with
higher data bandwidth, and to fully exploit its advantage it typically is no longer transported in an analog
composite way. The composite PAL / NTSC signals have the disadvantage that some color bandwidth is
sacrificed to limit the total signal bandwidth. To avoid this loss, HD signals are typically transported as a high-
speed digital serial signal. The raw bandwidth of these signals often exceeds the 1Gb limit of typical
contemporary networks (e.g. SMPTE 292M standard defines a nominal data rate of 1,485 Gb per second). As a
result, input requirements of the video capturing hardware become more stringent, and high performance data
compression becomes a critical need.
Processing Components

As device performance keep going up, it becomes possible to embed more complex functionality, or multiple
functions, for real-time processing applications. These additional functionalities may serve to offload the
operator, automate the system, or simply improve efficiency and performance.

A basic motion detection system that highlights and/or tracks moving objects within video frames can help an
operator to more easily distinguish points of interest in the video information. This kind of detection processing
becomes more difficult as the video platform itself is subject to motion. In such applications, stabilization
becomes a pre-processing step for the motion detection algorithm.

Opposed to what one would expect, performance does not come with weight, cost and volume. The
miniaturization trend continues as more functionality gets integrated into the same package, e.g. hybrid FPGAs
that contain PowerPC cores. Higher performance at approximately the same price means a lower relative cost
for the same capability. As a consequence, vide processing features such as stabilization become available as
embedded functionality, either at the sensor side or at the display side. What used to be a dumb display is
available today as a smart device that includes advanced local processing functions and computing capabilities.

Too often, higher performance has come with higher power demands. Heat dissipation is also closely related
with power consumption. Today, one of the main challenges for system designers is thermal management.
Technologies such as liquid flow through cooling, spray cooling and Peltier elements enjoy renewed attention in
an effort to cope with the typical heat dissipation of modern processing and graphics devices. Luckily,
component manufacturers have also understood this integration issue and are now paying increasingly more
attention to power consumption, e.g. by having components operate at lower voltages and using more efficient
semi-conductor materials.

Realtime Video Stabilization For Moving Platforms

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Realtime Video Stabilization For Moving Platforms

Hochgeladen von

Copyright:

Verfügbare Formate

Paper : Real-time Video Stabilization For Moving Platforms

Das könnte Ihnen auch gefallen