National Aeronautics and Space Administration

Airport Traffic Conflict Detection and

Resolution Algorithm Evaluation

Denise Jones, Ryan Chartrand, Sara Wilson, and Sean Commo,

NASA Langley Research Center
Sharon Otero, Northrop Grumman
Glover Barker, Science Systems and Applications, Inc.

denise.r.jones@nasa.gov
www.nasa.gov

31st Digital Avionics Systems Conference

October 14 – 18, 2012
 Collision Avoidance for Airport Traffic Concept
Multi-staged On-board Collision Prevention Strategy

II. Know where others are I. Know where you are III. Know where to go
Traffic position awareness Own-ship position awareness Route awareness
(ADS-B or TIS-B data link) (GPS & airport database) (Taxi route from ATC)

HUD Guidance
Departure Surface Map “Caution, Traffic” Taxi Surface Map
“Warning, Traffic”
“Crossing Hold”
“Off Route”
IV. Know when a mistake occurs
Conflict detection
(Immediately alert flight crew)
2
Study Objectives

Evaluate:
• Performance of aircraft-based CD&R algorithm via fast-time
simulation
 Runway, taxiway, and low altitude operations
 Various levels of horizontal position accuracy
 Multiple levels of CD&R system equipage

3
Conflict Detection Algorithm

• Airport Traffic Collision Avoidance Monitor (ATCAM)

– Aircraft-based algorithms
– Runway, taxi, and low altitude air-to-air operations
– Developed for NASA by Lockheed Martin
• Alerts
 Caution – for conditions that require immediate flight crew awareness
and subsequent flight crew response
 Warning – for conditions that require immediate flight crew awareness
and immediate flight crew response

4
Simulation Description

• Desktop simulation application operating in fast-time mode

• Pilot model for required avoidance maneuvers
• Two B757 aircraft simulated per scenario – Aircraft A and Aircraft B
• ADS-B In and Out
 1 Hz data transmission rate
 Gauss-Markov process for modeling time correlation between
position measurements
 Latency, line-of-sight blockage not simulated
• Pilot reaction delay
 5 seconds when on approach
 3 seconds when rolling-out
 2 seconds during taxi

5
Test Method
• Scenarios – 7 runway, 3 taxi, 2 low altitude air-to-air
(Only 2 scenarios reported in paper)
• Horizontal position accuracy – NACp 8, 9, 10, 11 and truth
• CD&R equipage NACp 95% Horizontal Accuracy Bound (EPU)
1. Both aircraft equipped 8 EPU < 92.6 m (0.05 nm, 305.6 ft)
2. Aircraft A equipped 9 EPU < 30 m (99 ft)
10 EPU < 10 m (33 ft)
3. Aircraft B equipped 11 EPU < 3 m (9.9 ft)
4. Neither aircraft equipped
 Equipped aircraft take avoidance action after a warning alert and
pilot reaction delay
 Unequipped aircraft continue along planned flight path
• Number of replicates for each treatment combination varied according
to level of surveillance accuracy – NACp 8, 7 reps; NACp 9, 6 reps;
NACp 10, 4 reps; NACp 11, 3 reps; truth, 1 rep
• Varied location of aircraft, speed, and/or time aircraft started to proceed
along predefined route
 Not every test run resulted in conflict or collision
6
Arrival with Taxi Crossing Scenario

0 to 15 kt

•198,828 test runs

•Aircraft A maneuver – go around if > 70 ft AFL, otherwise, land and stop on runway
• Aircraft B maneuver – stop if nose had not reached runway shoulder (100 ft from
runway centerline) at projected stopping point, otherwise, continue taxi across runway
7
Test Metrics – Near Collision (NC) / Collision (C)

ƒ NC if CG’s < 300 ft laterally, < 200 ft vertically

ƒ C if CG’s < 150 ft laterally, < 100 ft vertically

8
Results – Arrival with Taxi Crossing Runway Scenario

Collision Avoidance
• Most collisions avoided when both aircraft equipped with CD&R
• Collision avoidance more effective when crossing traffic equipped than when
approach aircraft equipped
• Position accuracy had little effect

Collisions
10

8
CD&R
7
Equipage
Percent of runs

6
Neither
5
Aircraft A
4
Aircraft B
3 Both
2

0
8 9 10 11 Truth
Position Accuracy (NACp)

9
 Results – Arrival with Taxi Crossing Runway Scenario
Algorithm performance
• Warning alerts generated on ~ 50% of test runs, independent of NACp level
• Caution alerts – ~ 20% of Aircraft A runs, ~ 50% of Aircraft B runs – issued on
approach when 8,000 ft to 6,000 ft from runway threshold
• Alert toggling occurred when multiple instances of alert generated
 Distraction and could cause mistrust in the technology
 More frequent as position accuracy reduced, especially for NACp 8 & 9
 Aircraft B toggling for truth data – after crossed runway and entering nearby
runway – alerts should not have been issued
60 60

50 50
Caution Alerts Caution Alerts
Percent of runs

Percent of runs
40 40

30 Multiple Caution 30 Multiple Caution

Alerts Alerts
20 Warning Alerts 20 Warning Alerts

10 Multiple Warning 10 Multiple Warning

Alerts Alerts
0 0
8 9 10 11 Truth 8 9 10 11 Truth
Position Accuracy (NACp) Position Accuracy (NACp)

Aircraft A Aircraft B 10
Test Metrics - When Taxiing Across Runway

Nuisance Boundary Missed Boundary

* As defined in RTCA DO-323, SURF IA SPR

11
Test Metrics - When Traveling Along Runway

Nuisance Boundary

* As defined in RTCA DO-323, SURF IA SPR

Missed Boundary 12
Test Metrics - When On Approach

• NACp 8 approach corridor with 0.01 probability of missed alert,

as defined in SURF IA SPR (RTCA DO-323)
• True position tracked extended runway centerline, so never
entered nuisance boundary

* As defined in RTCA DO-323, SURF IA SPR

13
 Results – Arrival with Taxi Crossing Runway Scenario

Missed and Nuisance Boundary

• Aircraft can cross into missed and nuisance boundary multiple times for varying
lengths of time
• Number of runs, number of times, and amount of time entered missed and
nuisance boundary increased as accuracy decreased
• Aircraft A mostly entered missed boundary after crossing runway threshold,
entered nuisance boundary when exiting the runway
• Aircraft B entered missed boundary for high % of runs for all NACp because no
buffer between when inside or outside of boundary
Entered Missed Boundary Entered Nuisance Boundary
90 30
80
25
70
Percent of runs

60 Percent of runs 20
50
15
40 Aircraft A Aircraft A
30 Aircraft B 10 Aircraft B
20
5
10

0 0
8 9 10 11 Truth 8 9 10 11 Truth
Position Accuracy (NACp) 14
Position Accuracy (NACp)
Test Metrics - Nuisance / Missed Alert

• Nuisance Alert – alert generated by a properly functioning system that

is inappropriate or unnecessary for the particular situation
 Can distract flight crew unnecessarily, reduce confidence in
system, negatively affect safety and operational effectiveness
 Definition: Alert issued when aircraft was within a nuisance
boundary

• Missed Alert – failure to provide an alert when it is necessary, provided

aircraft are adequately equipped
 Represent reduction in CD&R benefits
 Definition: Alert issued when using truth data but alert not issued at
the same instance when using NACp data. This definition is
algorithm dependent.

15
 Results – Arrival with Taxi Crossing Runway Scenario

Missed and Nuisance Alerts

• Number of runs with missed and nuisance alerts relatively low overall
• Aircraft A – all missed alerts occurred after crossing runway threshold,
nuisance alerts occurred as exiting runway

7 7

6 6

5 Missed Caution 5
Missed Caution
Percent of runs

Percent of runs
Alerts
Alerts
4 4
Missed Warning Missed Warning
Alerts Alerts
3 3
Nuisance Caution Nuisance Caution
Alerts Alerts
2 2
Nuisance Warning
Nuisance Warning
1
Alerts
Alerts 1

0 0
8 9 10 11 Truth 8 9 10 11 Truth
Position Accuracy (NACp) Position Accuracy (NACp)

Aircraft A Aircraft B
16
Test Metrics - Unwanted Alert

• Determine how far alert zone should be from hold line so

unwanted (nuisance) alerts do not occur
• Alert considered unwanted if true position behind hold line but
detected position over hold line, causing alert
• Maximum distance detected aircraft nose crossed over hold line
used to approximate distance alert zone should be from hold line
17
 Results – Arrival with Taxi Crossing Runway Scenario
Unwanted Alert
• Frequency increased as data accuracy decreased
• Larger buffer required between hold line and alerting zone as position
accuracy decreases
• 99% of unwanted alerts could have been avoided by placing alert zone:
 390 ft from hold line for NACp 8 accuracy
 135 ft for NACp 9
 55 ft for NACp 10
• Maximum standard for separation between hold line and runway centerline
is 280 ft to accommodate largest aircraft Airport Design, AC No, 150/5300-13, Change 15
18

16

14
Unwanted Caution Unwanted Warning
Percent of runs

12
NACp Max dist over hold line Max dist over hold line
10 Unwanted (ft) (mean, st dev) (ft) (mean, st dev)
Caution
8 8 126.4, 75.3 147.5, 80.7
6 Unwanted 9 50.5, 26.4 55.4, 26.0
4
Warning 10 16.6, 11.2 24.7, 9.5
2 11 0.0, 0.0 0.0, 0.0
0 Truth 0.0, 0.0 0.0, 0.0
8 9 10 11 Truth
Position Accuracy (NACp) 18
 Summary
• Alert toggling occurred more frequently as position accuracy decreased,
especially for NACp 8 and 9
• Occurrence of missed alerts higher with NACp 8 accuracy
• Occurrence of nuisance alerts relatively low
• Collision avoidance affected by CD&R equipage level, the most collisions
avoided with both aircraft equipped
• Sometimes collision avoidance more effective depending on which aircraft
equipped
• When taxiing across a runway, buffer needed between hold line and
alerting zone to reduce nuisance alerts – buffer size increases as data
accuracy decreases

Future Research
• Validate results of fast-time study
• Human-in-the-loop study planned for FY2013 19
Backup Slides

20
Taxi Intersection Scenario

0 to 15 kt

• 6,384 test runs

• Maneuver – stop if nose > 100 ft from intersecting taxiway centerline at projected
stopping point, otherwise, continue taxi across taxiway

21
Results – Taxi Intersection Scenario

Collision Avoidance
• Collision rate cut in half when one aircraft equipped
• When both equipped, rate drops significantly, down to 1% with NACp 10
• Position accuracy had most effect when both equipped
• Collisions occurred when using truth data because after alert issued and pilot
delay over, too late to stop before reaching 100 ft from centerline

Collisions
50
45
40 CD&R
Percent of runs

35 Equipage
30
Neither
25
Aircraft A
20
15 Aircraft B
10 Both
5
0
8 9 10 11 Truth
Position Accuracy (NACp)

22
 Results – Taxi Intersection Scenario

Algorithm performance
• Caution alerts generated on 55% to 76% of test runs
• Warning alerts generated on 50% to 66% of test runs
• Alert toggling more frequent as position accuracy reduced, especially for
NACp 8 & 9

80 80

70 70

60 60
Caution Alerts Caution Alerts
Percent of runs

Percent of runs
50 50

40 Multiple Caution 40 Multiple Caution

Alerts Alerts
30 Warning Alerts 30 Warning Alerts
20 20
Multiple Warning Multiple Warning
10 Alerts 10 Alerts

0 0
8 9 10 11 Truth 8 9 10 11 Truth
Position Accuracy (NACp) Position Accuracy (NACp)

Aircraft A Aircraft B
23
 Results – Taxi Intersection Scenario
Missed and Nuisance Boundary
• Entered missed boundary for large percentage of runs for all surveillance
accuracies, except truth, because no buffer between when inside or outside of
boundary
• Number of times and length of time within missed boundary greater with less
accurate data
• Nuisance boundary only entered for NACp 8 and 9
• Only possible to enter nuisance boundary when crossing the intersecting taxiway
because path followed taxiway centerline
Entered Missed Boundary Entered Nuisance Boundary
100 35
90
30
80
70 25
Percent of runs

Percent of runs
60
20
50
Aircraft A 15 Aircraft A
40
Aircraft B Aircraft B
30 10
20
5
10
0 0
8 9 10 11 Truth 8 9 10 11 Truth
Position Accuracy (NACp) 24
Position Accuracy (NACp)
 Results – Taxi Intersection Scenario

Missed and Nuisance Alerts

• Number of runs with missed increased as position accuracy decreased
• Rate of missed alerts same for both aircraft since conducted identical
operations
• No nuisance alerts

7 7

6 6

5 5
Missed Caution Missed Caution
Percent of runs

Percent of runs
Alerts Alerts
4 4
Missed Warning Missed Warning
3
Alerts 3
Alerts
Nuisance Caution Nuisance Caution
2 Alerts 2 Alerts
Nuisance Warning Nuisance Warning
1 Alerts 1 Alerts

0 0
8 9 10 11 Truth 8 9 10 11 Truth
Position Accuracy (NACp) Position Accuracy (NACp)

Aircraft A Aircraft B
25