GE Intelligent Platforms

Automated Target Tracking

Closing the Gap Between the Predictable and the Unpredictable

Automated Target Tracking

In the world of autonomous control of a UAV, there are many things that can be predicted and accounted for. The path that a vehicle will travel can be mapped from known navigable routes. Static objects that box that path can be identified and avoided using long-range imaging, plus non-real-time processing and adaptive path planning, which can be a combination of onboard and remote monitoring assets. Other moving vehicles can be made cooperative in their operation via rules of operation and telemetry transmitting devices. All of these things create an environment whereby location technology and rules execution can enable successful autonomous vehicle operation. All objects, however, cannot be made cooperative. For example, if an automated vehicle is a car and a soccer ball rolls into its path, and seconds later a child follows the ball, their sudden presence can neither be predicted nor cooperative in nature. For any autonomous vehicle, a similar uncooperative threat can be imagined. Real-time moving object tracking input will help to determine unexpected events, new potential threats, and the ability to project the correct course of action for safe operation and mission completion. This paper explores the ramifications of real-time autonomous vehicle performance in cluttered environments. It analyzes the gap between the predictable and the unpredictable, introduces real object tracing, explores the benefits and pitfalls of real-time moving object tracking and introduces possible solutions for uninterrupted operation of autonomous vehicles. Additionally, we demonstrate reliability improvement opportunities that can be derived when real-time tracking data is used in conjunction with predicted trajectory data to provide real-time performance assessments of vehicle operation.

Automated Target Tracking

Introduction
A person operating a motor vehicle uses sensory inputs, learned knowledge, experience and instincts to successfully navigate from point A to point B. Their learned knowledge includes the route traveled, the information necessary to identify objects and landmarks along their path and the ability to predict their future actions and reactions. From a drivers learned knowledge, objects such as rocks, cars, trees, animals and buildings can be distinguished from each other. Furthermore, identification of an object allows for a drivers expectation of an objects behavior which can be either predictable or unpredictable. For example, a building is predictably static and non-moving and an on-coming car is expected to stay in its lane of travel. On the other hand, the sometimes erratic behavior of a bird or a dog in the road cannot be accurately predicted. In addition to learned knowledge, a human operator will draw upon their past experience to make decisions about the environment that they are operating in. Having driven previously through an area may help to determine the behavior of the driver when they travel through the same or similar area again. Experience while potentially beneficial however, is not always necessary for successful navigation, and due to the dynamic nature of the world, may not always permit valid input. Sensory input including sight, sound, smell, and touch can help in the identification of objects and the recognition of unpredicted events. Unpredictable events come in many forms such as objects that have no predictable behavior (i.e. a bird), a predictable object that does the unexpected like an oncoming car that crosses the centerline, or in the form of an unforeseen event such as an object that suddenly enters a vehicles path with no prior warning. Sensory inputs allow for the tracking and closed loop avoidance* of unpredictable objects. The recognition, tracking and avoidance of unpredicted events are by far the most important and differentiating functions between a human operator and an autonomous control system. Consider the following example: A vehicle is traveling along a road and on the right-hand side of the road there is a school building. On the far side of the building there is a soccer field for the school. There is a low fence around the playing field and an open gate to allow access and the school building blocks the drivers view of the activities on the field. Suddenly, as the vehicle approaches the field, a soccer ball with a young child in pursuit rolls into the street directly into the path of the on-coming vehicle. The vehicle operator must make a quick decision to take an action quickly and correctly. Should the

vehicle suddenly stop or quickly swerve to the side to avoid the moving obstacles of the ball and child? Those are valid questions, but they cant be answered without more information. Which direction is the ball and child moving? Are they moving in the same direction? Will the drivers response guarantee avoidance? What will it take to make a good decision? The answer is timely, accurate and relevant inputs. In this case, actionable intelligence comes from collecting data about the dynamic environment around the vehicle. When the human is in the loop, either directly or remotely, the main source of this actionable input is the drivers vision. Human vision detects movement, and relative direction and velocity of the motion. This information is then used by the brain to decide the correct responsive action to take and transmit the proper signals to the rest of the body to carry out an avoidance maneuver. So, how do we translate this capability to an autonomous vehicle?

Case For Automated Target Tracking

While there are several methods of detecting movement and measuring the direction and the velocity of motion, the method that most resembles those human capabilities of visual input is the use of imaging sensors and automated target tracking hardware. Most autonomous or remotely piloted systems already have imaging sensors on-board, so the addition of an automated target tracking capability builds upon capabilities that are already resident in the platform.

Video Processing Architecture Overview

The block diagram in Figure 1 defines a potential video ingest system. This particular system is segmented into three processing areas: image, visualization, and post processing. The separation between these processing areas is defined by the timeliness in which the required computational load can be completed and the timing necessary for the computation result to be relevant. The image processor segment is where data extraction from raw video occurs. Image registration, detection of moving objects, and target tracking occur in this processing area. Data that can be used for real-time operation of hardware assets such as the steering gimbal, weapons system, and sense avoidance system is generated in the image processor. This area is where an automatic target tracker function would reside. The visualization processor takes data processed image frames and packages them for video consumption. The visualization processing activities typically include image enhancement, mosaicing, stitching, graphical overlay, image fusion, and image stabilization. These activities may also include display formatting or compression depending on the consumption method.

* Close loop avoidance is an avoidance maneuver that requires constant update of the avoidance path due to the dynamic nature of the obstacle that is being avoided.

Automated Target Tracking

Image Processor

Data Extraction

Control Data

Real-time Data Steering Gimbal Weapon System Sense and Avoid

Raw Video

Visualization Processor

Visualization

Enhanced Video

Enhanced Video

Video Consumption Monitor Communications Storage

Post Processor

Exploitation

Object Database

Figure 1 Potential Video Ingest System

The post processor uses all of the available metadata about the video to process the images to provide object identification, object recognition and activity-based intelligence. If these activities can be completed while the object is still within the field of view of the system, data such as object identification and recognition can be used to enrich the quality of the decision making capability of the autonomous system. The three processing regions are bounded by three measures of timeliness: real-time, near real-time, and non real-time. The definition of these three regions is soft, much debated, and subjective based on the processing environment and the application. The following are definitions that we will use regarding situation awareness systems: Real-Time Computational processing that is completed on the video stream within the amount of time that is required to generate the image frames. The frequency of this calculation will vary with the frame rate of the imaging device. The image processor provides real-time processing. Near Real-Time Computational processing that is completed within a defined number of video frames. This processing introduces a constant, consistent delay into the final image stream
4

causing a time lag from the capture of an event to the recognition of the event in the processed video. Acceptable near real-time performance is determined by the amount of delay that can be tolerated in the system while still providing usable performance. Near realtime activities are processed in the visualization processor. Non Real-Time Computational processing that requires a longer than acceptable delay for use as live video. This processing activity is often completed prior to video playback, but can be incorporated into real-time video taking advantage of the video tracking information. Post processing of video is a non-real-time activity.

Operation Scenario
Figure 2, Figure 3, and Figure 4 illustrate an UAV flyover scenario. The scenario incorporates air, ground moving and ground stationary targets into a forested ground terrain. Five imagers are used to create a field of regard which covers about 180 forward looking from the UAV. The automatic tracker provides the location of moving objects in the field of view of the imager. These moving targets are marked with solid white box outlined with another white box. While the boxes serve as a visual cue to the operator, they can also be used

Automated Target Tracking

to cue information into other systems, such as sense-and-avoid routines, or object classification and identification systems. Moving targets are the largest threat to the operation of the system. In Figure 2, the target tracker has identified five possible moving objects. Target one is an aircraft that just entering the field of view of the left most imager. Target two is a ground target which is an encampment. Target three is a ground vehicle. Targets four and five are another ground based encampment. The tracker has limited information about these targets. Are they threats, are they friendly, is there a potential for a collision with the UAV? These are all decisions that the tracker cannot make its own. However, the tracker can provide vital information to other systems which can make a determination about the targets and any potential threats to the UAV. If we focus on the Target 1 in Figure 2, this airborne target could be a potential threat to the safe operation of the UAV. The automated target tracker can provide input to a sense-and-avoid system on the UAV and the targets position, velocity and direction can be used to determine if the two aircraft are on a collision

course. If the aircrafts are on a collision course, the continuous update input feed from the video tracker could be used to steer the UAV away to avoid the other aircraft. Figure 3 and Figure 4 show the aircraft passing safely in front of the UAV flight path and out of the tracking field of view. A more sophisticated application of automated tracking technology, environmental monitoring, assumes that the UAVs sense-and-avoid system has access to information from other systems such as Identification Friend or Foe (IFF), Automatic Dependent Surveillance (ADS), Air-borne Warning and Control System (AWACS), or even mission planning, that would allow for the identification of an unknown aircraft and compare its expected flight path to its actual flight path to predict the expected behavior of the unknown aircraft and to determine its potential as a threat to the UAV. Figure 5 shows a scenario where an aircraft has deviated from its expected flight path. The aircraft is still a safe distance from the UAV and no avoidance maneuver is necessary. However, this type of environmental monitoring enables the UAV to act

Figure 2 Tracking Scene 1

Figure 3 Tracking Scene 2

Figure 4 Tracking Scene 3

Figure 5 Aircraft deviation from expected

5

Automated Target Tracking

more intelligently to the environment around it. There could be many reasons for an aircrafts flight path deviation, for example a maintenance issue, the aircraft (target 1) must take a defensive or offensive action, a change in the mission plan, or something more sinister such as jamming or spoofing. Any of these could be a reason for a UAV to alter its procedures to protect its assets or alert the mission manager for further action.

The second stage (target detection) processor analyzes the output from the pre-processor to identify objects within a selected area, or window, in the video image which meet user-defined target criteria. Target detection from among these identified objects may be prioritized using several different criteria, including the target nearest to the boresight or platform, and the largest target. The third stage (tracking) is where a selected object is identified as the target and its behavior (velocity, position and trajectory) is measured and predicted. Objects (potential targets) detected in the previous stage are sorted based upon automatic or manual means into a primary target or one of several secondary targets. One target (designated as the primary) may be used for closed loop control of a pan and tilt head or gimbal; here, the behavior data is fed to the gimbal control to automatically keep the target in the center of the field of view of the camera or at a fixed position offset from the center if desired. Target selection may be made automatically or without user intervention or the operator can choose which target to follow.

Video Tracker Technology

Target detection and tracking may be divided into three main stages: pre-processing, target acquisition, and target tracking. In the pre-processing stage, verified mathematical algorithms are employed to improve target contrast and suppress background clutter. The result of this enhancement is an accurate separation of the target image from noise in the scene for more robust target acquisition and tracking. Standard pre-processors use contrast threshold processes to separate the object from the background and have three main categories: A positive (white-hot) pre-processor enhances positive contrast targets. This function is performed using an adaptively calculated threshold where intensities below the threshold are transformed to a black intensity level and intensities above the threshold are unmodified. It is used when the target has a higher intensity than the local background and the points of interest are the bright parts of the target, such as when tracking hot objects with a thermal imager. A negative (black-hot) pre-processor enhances negative contrast targets. This function is the inverse of the positive contrast pre-processor in that it enhances low-intensity pixels and sup-presses high-intensity pixels. It is performed using an adaptively calculated threshold where intensities above the threshold are transformed to a low (black) intensity level and intensities below the threshold transformed to high (white) intensities. A black-hot pre-processor is used when the target has a lower intensity than the local background and the points of interest are the dark parts of the target, such as when tracking hot objects with a thermal imager with reversed polarity. A statistical pre-processor enhances targets with both positive and negative components. Here, intensities that are statistically likely to be target intensity (and not background intensity) are enhanced. This function is performed using a statistical measure of the video image inside and outside a settable Fieldof-Regard (FoR) to differentiate between target and background intensities. The target pixels are transformed to a white (high) intensity level and the background pixels transformed to a black (low) intensity level. A statistical preprocessor has the advantage in that it enhances both positive and negative contrast targets as well as those targets containing both positive and negative contrasts with respect to the local background. It also continues to function during a contrast polarity change.
6

Advantages Video Tracker Technology

Automated target tracker technology is inserted in the beginning of the image processing chain. It processes images at the frame rate of the imager. For objects that are moving relative to the image scene, the video tracker produces target location, direction and velocity information. The major advantage of this method is that the moving target data is available with low latency relative to the object movement or entry into the field of view of the imager that is being processed by the tracker. An additional benefit of an automated video tracker is that is it capable of detecting objects that would not be discernible to the human eye. Improved target detection provides the opportunity for quicker responses to potential threats to the system. The low light and contrast image in Figure 6 has a moving image in it. The moving object is obscured by clouds in the image.

Figure 6 Image has moving target

Automated Target Tracking

Figure 7 indicates the location that the moving target in the image. Without outside knowledge of the location of the target it would be impossible for a human to identify the moving target in the picture. The targets distance from the observer and its relative size make it difficult to detect either with the unaided eye or in a video image without further enhancement.

Figure 9 Movement detection and tracking

that can help enable the closing of the gap. Automated target trackers are only as good as the input that they are provided. Low contrast images and obscured targets are difficult environments for automated target tracker operation.
Figure 7 Blue circle indicates location of moving target

Figure 8 demonstrates the effects that contrast enhancement and preprocessing can have on the video image. In Figure 8, the object to be tracked becomes distinct, although it cannot be discerned from other high contrast objects. The image in Figure 9 is the result of the application of the movement detection. This final processing has identified the moving object from the non-

Automated target tracking algorithms often must be tuned for the environment that they are operating in. Nonoptimized solutions can lead to either missed recognition or high false alarm rates. Neither is desirable, both can cause autonomous to make bad decisions and recreate operational failure of the system. GE Intelligent Platforms line of automatic target trackers are at the heart of a wide range of commercial and military systems where they provide the highest performance solutions in the smallest, fully environmentally proven hardware packages. A range of proven high-performance algorithms, which can be tailored for the application, allow real-time detection and tracking of targets within the video image which meet user-determined selection criteria. Other functionalities include real-time video fusion and video stabilization. GEs SWaP (size, weight, power) optimized ADEPT3000 target tracker is designed for the cramped space of small unmanned aerial vehicles and man portable devices. Despite its extremely small size, GEs ADEPT3000 offers high performance single target tracking with any standard definition analog video signal. Approximately the size of a quarter, the commercial off-the-shelf (COTS) ADEPT3000 operates as a stand alone unit allowing great flexibility in board location. More information on the whole line of GE automatic target trackers can be found here: http://defense.ge-ip.com/products/ family/target-tracking

Figure 8 Contrast enhancement and pre processing

moving objects in the image.

Disadvantages Video Tracker Technology

Automated target tracking is not the panacea to solve all difficulties faced in dealing with unpredictable events during autonomous operations. Video target tracking is just one tool

Automated Target Tracking

Conclusion
Automated target tracking technology provides a method of closing the gap between predictable and unpredictable events for autonomous system operation. This tool offers an excellent opportunity to supplement the capability of an autonomous aerial or ground system. By taking advantage of imagers that are already available in a UAV, and providing critical information that can be used to improve the performance of the system, UAV designers can measurably increase the probability of mission success and survivability of the platform.

GE Intelligent Platforms Contact Information Americas: 1 800 433 2682 or 1 434 978 5100 Global regional phone numbers are listed by location on our web site at www.ge-ip.com/contact

defense.ge-ip.com
2012 GE Intelligent Platforms, Inc. All rights reserved. *Trademark GE Intelligent Platforms, Inc. All other brands or names are property of their respective holders. 07.12 GFT-842