Insight

Industry Day
Dr. Michael Pagels
Program Manager
Information Innovation Office

National Rural Electric Cooperative Association Conference Center

21 & 22 September 2010

Approved for Public Release, Distribution Unlimited

 Problem: Limited Ability to Automatically Exploit and
Cross-Cue Multi-INT Sources

Operational example: 5 HUMINT Report:

High Value Individual (HVI)
Suspect 2 interviewed
tracking with if-and-then Insight: Cross-cue, initiate tracks,
constraints associate track to report

4
Comm Device Emission 6
Insight: Stimulate device to Suspects converge to same location
update state uncertainties Insight: Associate tracks, cue
2 video activity processing, alert
Post-Checkpoint: vehicle driving operator
Insight: Predicts track loss,
1 cross-cue alternative sensor
7 HUMINT: Conflicting information
Checkpoint: Suspect – Suspect 3 seen in Paris
1 in Vehicle A passed Insight: Update uncertainties and
Insight: Cross-cue sensor, initiate alert, cue source analyst for
tracks, associate track to report verification
3
0 Activity Alert: HUMINT: Verifies Suspect 3
Vehicle A stops,
8
Watchlist posted and seen leaving residence
Suspect 3 enters Insight: Update uncertainties
accepted by analyst
Insight: Associate tracks and and alert
Insight: Add watchlist threat network information,
to active tasks, initiate assess state uncertainties
data search
Approved for Public Release, Distribution Unlimited 2
 Problem: Limited Ability to Automatically Exploit and
Cross-Cue Multi-INT Sources

Existing and planned systems connect multiple real-time sources to tactical

operations centers for largely manual exploitation and cross-cueing via chat-
based operator interaction
• No automated correlation and uncertainty management
• Limits the number of threats to a fraction of the number of operators
• Limits scenario complexity to less than sensors observe
• No automated collection task assessment and management
• Limits the ability to prosecute agile threats
• Limits response times amid rapidly changing conditions
• No integrated human-machine reasoning
• Limits the ability to exploit operator knowledge of context
• Limits operator ability to reason over complex data
• Warfighters cannot select best asset mix to employ for a given mission
• Assets are underutilized or unavailable

Approved for Public Release, Distribution Unlimited 3

 Problem: Limited Ability to Automatically Exploit and
Cross-Cue Multi-INT Sources

Capability Objective Context and Mission Human Machine

Adaptability Interaction
HVI tracking with Detect and track multiple HVIs; • Deliberate decision making • Identity confusion
multiple conditional satisfy multiple alerting and collaboration management
constraints constraints • Activity analysis
• Resource management

Long term HVI Identify and track threat • Suburban domain • Track stitching
tracking dismounts and vehicles through • Rural domain • Track correlation
clutter and obscuration • Rural-to-Suburban-to-Urban • Resource management
Estimating the Identify threat relationships, • Compound area networks • Threat network
threat network roles, and associated physical • Urban area networks hypothesis generation
infrastructure • Dispersed rural networks • Resource management
Behavioral Estimate normal patterns of • Mobile force overwatch • Normalcy estimation
modeling, activity on roads, along borders, • Site monitoring • Pattern detection
prediction, and and in areas friendly forces will • Road monitoring • Pattern prediction
anomaly detection be operating • Border monitoring • Anomaly detection
Detect deviations from normalcy • Resource management

Approved for Public Release, Distribution Unlimited 4

 Solution: Next Generation ISR Exploitation System,
Evaluated At NTC Physical Test Bed

Insight Exploitation & Resource Management (E&RM) System

Adaptive multisource
exploitation & resource
management tools
Unified all-source ISR
workstation
Algorithm Command,
Control & Data Archive
• Build model-based correlation, behavioral modeling, and threat network analysis tools to automatically combine
data across sources and manage uncertainty.
• Build resource management tools to identify collection opportunities, and enable efficient use of multi-INT
sensors and platforms across missions.
• Build a unified data management and processing environment to support data and processing normalization,
and enable rapid integration of new data sources, exploitation algorithms, and analysis methodologies.
• Integrate human and machine processing, including visualization, hypothesis manipulation, crowd-sourcing,
and on-line learning.

• Increase the number of participants Insight Test Bed

• Lower the cost of entry for participants
• Test multiple concepts of operation (CONOPs)
• Test alternative sensor mixes
• Test multiple exploitation algorithm solutions
• System evaluation under realistic
operational conditions
• Supports Service transition

Development Incubator Theater

• Collected, simulated and
processed data
• Variable resolution
• Multiple security levels
• Support infrastructure
Virtual Environment
• Simulated sensor data
• Simulated world of various threats,
terrains, and terrain features Physical Test Bed
• Augmented with real-world • National Training Center
collected data • Realistic operational conditions
Approved for Public Release, Distribution Unlimited 5
 Insight E&RM System:
Three components

Notional Architecture
1: Analysis and Resource Management System
Information
Needs
Known Threats
Search
Extraction & 2: Unified All-source
Resource Link Discovery
Management Threat ISR Human-Machine Interface
Description Actionable
Collection Cues Intelligence
Threat and
Tasks Background Models Anomaly Recognition
• Leadership Tracks
ID of Individuals
Multi-Mode Track • Mission Planning
Behavior Patterns
Collection Multiple Futures
Prediction
•
•
Notional
Bombmakers
Supply Chain
Node and Link Analysis
(Sensing) • Financers
Roles
Background/Environment
Registered Data Layers: • Recruiting
• Open Source (OSINT) • IED Emplacers
Histories GIS
• Law Enforcement •
Threat Description
• Commercial Sources
• IMINT (Imagery, Video, Wide New Behaviors Information Needs Analysts
Area Motion Imagery (WAMI)) and Features
• GMTI/VMTI Classify
• SIGINT
Interpretation
• COMINT
• MASINT Forensic Studies
• CI/HUMINT
• Soldier Information
• SIGACTS 3: Integration, Processing and Data Archive Environment Archive for Forensic Studies
• Ground Based Sensors and Forensic Analysis Reach
back

Approved for Public Release, Distribution Unlimited 6

 Insight E&RM System
Component 1: Analysis & resource management system

Develop information exploitation analysis tools which

automatically combine and process data across Development Incubator
multiple sources to fulfill multiple ISR missions Virtual Environment
• model-based correlation Physical Test Bed
• behavioral modeling
• threat network analysis
Develop collection and sensor management tools smart
enough to know when and how to cooperatively
prosecute missions and targets
Provide the foundational research for enhancing
algorithmic performance by visualization of
intermediate results, human intervention in
hypothesis resolution, and interactive on-line learning
Technology push towards open, modular systems User interaction drives the
Unbundle exploitation system development stovepipes automated exploitation process

into minimal components to reduce the time and cost

to develop future ISR systems

Approved for Public Release, Distribution Unlimited 7

 Insight E&RM System
Component 2: Unified all-source ISR human-machine interface

User interaction drives the

automated exploitation process
Modern human-computer interface (HCI), which acts as
the visual and procedural gateway to Insight’s
management and exploitation capabilities
Provide foundational research into enhancing effective
interaction between human and E&RM system
Provide access to exploitation tools and products over a
wide range of interface devices to maximize multiple
echelon coverage
Insight Workstation
Adaptable, customizable, context-aware, unified
interface capable of supporting dynamic, interactive
missions
• Social network leveraged to maximize collaboration
• “Push” technology leveraged for information discovery
• Adapts to task and mission focus covering all training,
planning and exploitation phases
• Plug-in architecture supporting rapid integration and Insight Analyst
evaluation of innovative HCI technologies
• Standards-based implementation supporting dissemination
User interaction with both real
of HCI technologies to other systems and domains and virtual data supporting
multiple exploitation phases
Approved for Public Release, Distribution Unlimited 8
 Insight E&RM System
Component 3: Integration, Processing and Data Archive Environment

Information

Architected to maintain an adaptive and agile

Needs

Search

user experience by rapidly integrating new

Extraction &
Resource Link Discovery

Management Threat

algorithms via a standardized application Collection Cues

Description

programming interface Tasks

Threat and
Background Models
• Leadership
• Mission Planning

Manages and monitors the mapping of the

Multi-Mode Track • Bombmakers
Multiple Futures • Supply Chain
Collection Prediction • Financers

E&RM system’s dynamic algorithms on to a

(Sensing) • Recruiting
• IED Emplacers
• GIS

distributed heterogeneous execution Histories

environment of computing and visualization

New Behaviors
and Features
Classify
resources Interpretation

Captures the provenance of all products in the

processing chain to explain the reasons
behind algorithmic exploitation results Distributed Heterogeneous Execution Environment
Provides the unified data source for all Diverse platforms required for diverse missions

processing
• Common standards-based, spatial, temporal and error
measure registration techniques to unify layers across
pixel, text and other data representations
• Real sensor output and virtual/simulated data coexist
• Leverage existing diverse COTS database technologies

Approved for Public Release, Distribution Unlimited 9

 Solution: Next Generation ISR Exploitation
System, Evaluated At NTC Physical Test Bed

Insight Exploitation & Resource Management (E&RM) System

Adaptive multisource Global ISR solution has two elements
• Build model-based correlation, behavioral modeling, and threat network analysis tools to automatically combine
exploitation & resource data across sources and manage uncertainty.
management tools • Build resource management tools to identify collection opportunities, and enable efficient use of multi-INT
Unified all-source ISR sensors and platforms across missions.
workstation • Build a unified data management and processing environment to support data and processing normalization,
and enable rapid integration of new data sources, exploitation algorithms, and analysis methodologies.
Algorithm Command, • Integrate human and machine processing, including visualization, hypothesis manipulation, crowd-sourcing,
Control & Data Archive and on-line learning.

• Increase the number of participants Insight Test Bed

• Lower the cost of entry for participants
• Test multiple concepts of operation (CONOPs)
• Test alternative sensor mixes
• Test multiple exploitation algorithm solutions
• System evaluation under realistic
operational conditions
• Supports Service transition

Development Incubator Theater

• Collected, simulated and
processed data
• Variable resolution
• Multiple security levels
• Support infrastructure
Virtual Environment
• Simulated sensor data
• Simulated world of various threats,
terrains, and terrain features Physical Test Bed
• Augmented with real-world • National Training Center
collected data • Realistic operational conditions
Approved for Public Release, Distribution Unlimited 10
 Insight Test Bed
Component 1: Development incubator

Collection Tasks
Technology clearinghouse supporting development and
rapid evaluation of new capabilities for inclusion in Multi-Mode
Insight’s E&RM System Collection
(Sensing)
• Provide access to technical and military experts
Registered Data Layers:
• Provide operationally relevant mission descriptions and • Open Source (OSINT)
context • Law Enforcement
• Commercial Sources
• Incremental evaluation utilizing historical data, virtual • IMINT (Imagery, Video, Wide Area Motion
environment and physical test bed Imagery)
• GMTI/VMTI
Provides an unclassified data warehouse that lowers • SIGINT
the cost of participation via system evaluation and • COMINT
• MASINT
maximizes analysis of trade-offs for innovative • CI/HUMINT
development • Soldier Information
• SIGACTS
• Generates “dialable” individualized views into the data • Ground-based Sensors

• Captures processed data necessary for component

development and evaluation
Enables leveraging of other funding opportunities
which broadens participation to non-traditional
performers
Validates Insight’s Measure of Performance (MOP)
development and evaluation among performers
Coordinates transition outreach and demonstrations
Development Incubator
Approved for Public Release, Distribution Unlimited 11
 Insight Test Bed
Component 2: Virtual environment

Enables evaluation of ISR exploitation systems using

simulated sensor data, augmented with real-world
collected data, within a simulated world of various
threats, terrains and terrain features.
Coupling a virtual world to the real
• Test on sequestered data or data at higher classification levels world
not available in development incubator
• Test with data from simulated sensor and threat operating
conditions beyond those in development incubator
• Assess system-level performance impacts of novel component-
level algorithms
Enables system evaluation within multiple CONOPs,
alternative sensor mixes, and multiple exploitation
algorithm solutions
• Provide data-rich environment to evaluate system-level Virtual environment
• Simulated sensor data
tradeoffs of sensors with different performance points • Simulated real world
• Experiment with human-machine interaction in realistic • Augmented collected data
simulated world environment
• Assess against varying terrain and threat contexts
Enables closed-loop system tests prior to deployment to Physical Test Bed
reduce risk
• Adapt high fidelity sensor simulations to Insight system
• Merge recorded, live, and simulated data to enable realistic
system test and performance analysis

Approved for Public Release, Distribution Unlimited 12

 Insight Test Bed
Component 3: Physical test bed

Populates the development incubator with data & is

used to derive parameters for the virtual
environment, from multiple data collections in a
militarily relevant environment
Validates the development incubator and virtual
environment results against validated scenarios
executed in a militarily relevant environment
Used to develop and evaluate CONOPs and CONEMPs Physical test bed
• National Training Center
necessary for successful Service transition • Realistic operational conditions
Leverage physical test bed
• Facilities to house, operate, exercise, and compute within
an operational infrastructure
• Physical environment reflective of operational settings
• Sources of clutter reflective of operational settings
• Civilian and military personnel, and equipment sufficient to
execute validated scenarios
• Infrastructure to capture all “truth” required for evaluation
Capability to electronically incorporate other remote
facilities that provide unique capabilities via existing Defense Research Enterprise Network
• Secure links to other facilities
Secret/Defense Research Enterprise Network • Extended operational conditions

Approved for Public Release, Distribution Unlimited 13

 The National Training Center
Fort Irwin, CA
Large training area located in the high Mojave Desert
US Army’s premiere Combat Training Center
Dedicated to training Brigade Combat Teams prior to deployment
Extensive existing instrumentation and infrastructure
Realistic OPFOR and clutter environment
Cooperative leadership permits experimentation without training interference

Dahuk
Dahuk
Bashur
Mosul

Environment
Al Hasakah Arbil

Scale
Tall Afar Arbil
SYRIA
Brigade: ~6000 Dayr az Zawr
MND-N
Ninawa Kirkuk Sulaymaniyah IRAN

Officer Controllers: ~400

As Sulaymaniyah
At Ta’mim

Salah ad Din Hamadan

10km
Tactical Vehicles: ~2,000
Al Mayadin
Hadithah Tikrit
Al Qaim Dam
MND-NC
Role Players Population: ~3,000 H1
Hadithah
Samarra Diyala

Baqubah

Civilian Vehicles: ~300 H2

Ramadi

Fallujah Baghdad
Kirkush

Insurgents: ~300
Baghdad
10 Ar Rutbah
MND-W MND-B
H3 W asit
Babil

Civilian Main Post Population: ~10,000

Karbala Dezful
Al Anbar Karbala Al Hillah  Al Kut
Ad Diwaniyah

Main Post Military Population: ~ 3,000

Maysan

An Najaf Al Qadisiyah
Al Amarah

Commuting Workforce: ~2,000 As Samawah

Dhiqar
Ahvaz Major military exercises conducted monthly
Area: ~300 sq kilometers
An Nasiriyah Al Basrah
An Najaf Al Basrah

SAUDI
ARABIA
MND-S
Al Muthanna Az Zubayr Robust operational environment, tailored to both
Iraq and Afghanistan
Kuwait City
FOR OFFICIAL USE ONLY - Distribution authorized to U.S. Government Agencies Source: NTC Command Brief 5
only

Realistic training environment used by ground forces to prepare for Iraq and Afghanistan rotations
Approved for Public Release, Distribution Unlimited 14
 Insight BAA Technical Areas of Interest

1. Integration, Processing
and Data Archive
Environment
2. Adaptive Multisource
Exploitation System
3. Collection and Resource
Management System
4. Unified All-source ISR
Human-Machine
Interface
5. Physical Test Bed
6. Virtual Environment
7. **Development
Incubator
8. **Transition Discovery
and Oversight
9. Accelerated Innovation

** Technical Areas 7 & 8 will not be solicited under the Insight BAA
Approved for Public Release, Distribution Unlimited 15
 Notional Insight Program Organization

Collaborating DARPA
Program Managers Insight Program “Unity of Effort” via ACAs
Manager
and contract incentives

Development Transition Discovery Contracting Science & Engineering Senior Advisory Group
Incubator & Oversight Team Contracting Officers’ Technical Advisors Service Stakeholders
Single Prime Single Prime Technical Representatives IC Stakeholders
Government or FFRDC Government or FFRDC Emeritus Scientists

Evaluation Leadership Architecture Leadership

Group Group
All Contractor All Contractor
Evaluation Leads Principle Investigators

Physical Test Bed Accelerated Adaptive Multisource Unified All-source ISR

& Test Integration Innovation Exploitation System HMI
Single Prime Multiple Contracts Single Prime Single Prime
Commercial & University

Virtual Environment Collection & Resource Integration,

Single Prime Management Processing & Data
Single Prime Archive
Single Prime

Approved for Public Release, Distribution Unlimited 16

 Phased Development Plan

FY10 FY11 FY12 FY13 FY14 FY15 FY16

Collaborations

Field Test 3‐9 Apr 2012 HART PERSEAS

WAND TAILWIND
ARGUS FITT
Field Test 2 Mar 2011 VADER GuardDog
VIRAT RC2

Field Test1 23 Oct through 02 Nov 2010 TRANSTION TRANSTION TRANSTION

Phase 1 leverages FT 1
Release Phase 1 BAA
Q4FY10 24 month phase 1 30 month phase 2
TRANSTION TRANSTION

Release Phase 2 BAA
Q4FY12

Capabilities
• HVI tracking with multiple
conditional constraints
• Long term HVI tracking in heavy
clutter
• Estimating & predicting the enemy
network
Phase 1 Characteristics
Demonstrations Exploitation System Test Bed
• Ability to correlate across multiple • Unified workstation that integrates • Assemble collected, simulated and
sensor sources; WAMI, Radar, human and machine processing, processed data supporting tasks
COMINT, OSINT, crowd-sourcing, including visualization, hypothesis • Assemble metrics & test harness
text-based, and NFOV FMV manipulation, crowd-sourcing, and • Evaluate collected, simulated and
sensors on-line learning processed data supporting new
• Incorporate automated resource • Automated correlation, behavioral algorithms and CONOPs/
management modeling, and threat network CONEMPs integration
• Provide user interaction though an analysis across multiple sources • Evaluate correlation, control,
advanced human-computer • Baseline control algorithms human-machine algorithms on
interface test + variants of each sensor
Approved for Public Release, Distribution Unlimited 17
 www.darpa.mil

Approved for Public Release, Distribution Unlimited

 Insight
Industry Day
Global ISR Data Collection #1
Dr. Michael Pagels
Program Manager
Information Innovation Office

National Rural Electric Cooperative Association Conference Center

21 & 22 September 2010

Approved for Public Release, Distribution Unlimited

 GISR-DC 1
Sensor Systems & Processed Products to Insight
JSTARS
(GMTI) SENSOR PROCESSED
SYSTEM PRODUCT TO Insight

VADER • Target detections,

SIGINT locations & time tags
ARGUS-IS (WAMI)
• STANAG 4607 MTI &
MX-14 / EO
VEGS Tracks

VADER (GMTI) TAILWIND (EO/IR) ARGUS-IS • Full FOV Mosaic (in

JPEG 2k format)
• Archival imagery: JP2K
SIGINT frames + headers
SIGINT • Video window imagery:
JP2K frames + headers

TAILWIND • Stitched LWIR imagery

• Local area contrast
enhancement
• De-noise
• Temporal filter
• Sharpening
• Tracks
• 3-D models
• Alerts
• Video summary

SIGINT Deanonymonized
geolocations

SIGINT Geolocations

SIGINT • Device geolocations

with IDs
• Device tracks

MX-14 / EO • Single FMV stream

• EO imagery nadir to
MX-14
• Surface Model
• Orthos

JSTARS Tracks

Approved for Public Release, Distribution Unlimited 2

 GISR-DC 1
Airborne Systems

Airborne Systems
Typical collection Operating
flight paths Altitude
(MSL)
23000 ft

ARGUS
18000 ft
SIGINT

14000 ft
VADER

10000 ft
VADER flight
path
(15 km offset) SIGINT 9000 ft
MX-14/EO

8000 ft
Antennas at ground
station shelters @ TAILWIND
Building 6223 7000 ft
SIGINT
Approved for Public Release, Distribution Unlimited 3
 GISR-DC 1 Data Collection #1
GISR
AOIs &Focus
Scenarios
Areas/Scenarios
Scenario3
Small
urban/Rural

FOB Detroit
Citadel

Medina Wasl

Medina Jabal

Scenario 4
Small
urban/urban/Rural
Medina Irwin
Scenario 2
Scenario1 Urban/Rural
Urban

Calibration Box

Approved for Public Release, Distribution Unlimited 4

GISR-DC 1
Insurgent Scenarios

Approved for Public Release, Distribution Unlimited 5

GISR-DC 1
Scenario 1: Urban – Medina Irwin

Approved for Public Release, Distribution Unlimited 6

 GISR-DC 1
Final Product Delivery
System Onsite Data Post? Final Product & Delivery Mechanism
VADER • STANAG 4607 data; Raw IQ data; “Quicklook” data No Duplicate HDs delivered to VDL
• VEGS track & shape files

ARGUS-IS Imagery organized by focal plane array (when No Duplicate HDs delivered to VDL
archived, organized by time); Video streams sent to
ground
TAILWIND Imagery from 4 IR cameras; Imagery from 1 VNIR Yes Stitched LWIR imagery
HSI; Imagery from 1 EO Duplicate HDs delivered to VDL
SIGINT • Binary signal data broken in time sliced files Yes • Geolocations with error ellipses & IDs (temporary)
• ASCII event metadata • Temporary->permanent ID mappings
• Event metadata merged with above data to report
permanent IDs
• Labeled geolocations delivered to VDL

SIGINT RF data, tracking data, receiver metadata, tipping Yes Labeled geolocations delivered to VDL
data
SIGINT Ground Live geolocations; raw IQ data No Labeled geolocations delivered to VDL
SIGINT DF data & INS Yes Geolocations & tracks; updated geolocations with
IDs delivered to VDL
MX-14 Single FMV stream (from 6 EO cameras) No Copied and delivered to VDL
EO imagery nadir to MX-14 DLTV & spotter video No Copied and delivered to VDL
files
Truth Vehicle track files & personnel/dismount track files; Yes Merged annotated tracks delivered to VDL
event logs
National SIGINT TBD No TBD
National IMINT TBD No TBD

Approved for Public Release, Distribution Unlimited 7

 GISR-DC 1
Schedule

GISR DC 1 Schedule

NTC Schedule

GISR DC 1
Collection
Schedule
Approved for Public Release, Distribution Unlimited 8