Beruflich Dokumente
Kultur Dokumente
David Gunning
DARPA/I2O
Proposers Day
11 AUG 2016
A. Introduction
B. Program Scope
1. Explainable Models
2. Explanation Interface
3. Psychology of Explanation
4. Emphasis and Scope of XAI Research
C. Challenge Problems and Evaluation
1. Overview
2. Data Analysis
3. Autonomy
4. Evaluation
D. Technical Areas
1. Explainable Learners
2. Psychological Model of Explanation
E. Schedule and Milestones
F. Deliverables
Distribution Statement "A" (Approved for Public Release, Distribution Unlimited) 2
Questions
• The current generation of AI systems offer tremendous benefits, but their effectiveness will
be limited by the machine’s inability to explain its decisions and actions to users.
• Explainable AI will be essential if users are to understand, appropriately trust, and
effectively manage this incoming generation of artificially intelligent partners.
Today Task
• Why did you do that?
Decision or • Why not something else?
Machine Recommendation • When do you succeed?
Training Learned
Learning • When do you fail?
Data Function
Process • When can I trust you?
• How do I correct an error?
User
XAI Task
• I understand why
New • I understand why not
Training Machine Explainable Explanation • I know when you succeed
Data Learning Model Interface • I know when you fail
• I know when to trust you
Process
• I know why you erred
User
XAI Task
• I understand why
New • I understand why not
Training Machine Explainable Explanation • I know when you succeed
Data Learning Model Interface • I know when you fail
• I know when to trust you
Process
• I know why you erred
User
XAI Task
• I understand why
New • I understand why not
Training Machine Explainable Explanation • I know when you succeed
Data Learning Model Interface • I know when you fail
• I know when to trust you
Process
• I know why you erred
User
XAI Task
• I understand why
New • I understand why not
Training Machine Explainable Explanation • I know when you succeed
Data Learning Model Interface • I know when you fail
• I know when to trust you
Process
• I know why you erred
User
Prediction Accuracy
machine learning Models
Deep
techniques that Learning Ensemble
Bayesian Methods
produce more Belief Nets
explainable models, SRL Random
while maintaining a CRFs HBNs Forests
AOGs
high level of Statistical MLNs
Models Decision
learning Markov Trees
performance SVMs Models Explainability
Prediction Accuracy
machine learning Models
Deep
techniques that Learning Ensemble
Bayesian Methods
produce more Belief Nets
explainable models, SRL Random
while maintaining a CRFs HBNs Forests
AOGs
high level of Statistical MLNs
Models Decision
learning Markov Trees
performance SVMs Models Explainability
Deep Explanation
Modified deep learning
techniques to learn
explainable features
Prediction Accuracy
machine learning Models
Deep
techniques that Learning Ensemble
Bayesian Methods
produce more Belief Nets
explainable models, SRL Random
while maintaining a CRFs HBNs Forests
AOGs
high level of Statistical MLNs
Models Decision
learning Markov Trees
performance SVMs Models Explainability
Prediction Accuracy
machine learning Models
Deep
techniques that Learning Ensemble
Bayesian Methods
produce more Belief Nets
explainable models, SRL Random
while maintaining a CRFs HBNs Forests
AOGs
high level of Statistical MLNs
Models Decision
learning Markov Trees
performance SVMs Models Explainability
End User
Explanation
Question
Visual
Answering
Analytics
Dialogs
XAI
Emphasis
Human
Machine
Computer
Learning Interactive
Interaction
ML
Category Definition
Basic Research Systematic study directed toward greater
(6.1) knowledge or understanding of the
fundamental aspects of phenomena and/or
observable facts without specific
applications in mind.
Multimedia Data
Classifies items of Explains why/why not Analyst decides which
interest in large data set for recommended items items to report, pursue
An operator is
Actions
Autonomy Explainable Explanation directing autonomous
Model Interface systems to accomplish
Reinforcement Explanation
Learning Task a series of missions
©ArduPikot.org
©US Army
ArduPilot & SITL Simulation
Measure of Explanation
Effectiveness
User Satisfaction
Explanation Framework
• Clarity of the explanation (user rating)
Task • Utility of the explanation (user rating)
Recommendation, Mental Model
Decision or
• Understanding individual decisions
Action
• Understanding the overall model
Explainable Explanation • Strength/weakness assessment
Decision • ‘What will it do’ prediction
Model Interface
The user • ‘How do I intervene’ prediction
makes a
XAI System Explanation decision Task Performance
The system takes The system provides based on the • Does the explanation improve the
input from the current an explanation to the explanation user’s decision, task performance?
task and makes a user that justifies its
• Artificial decision tasks introduced to
recommendation, recommendation,
diagnose the user’s understanding
decision, or action decision, or action
Trust Assessment
• Appropriate future use and trust
Performance
Teams that provide
Learning
prototype systems Interpretable
with both components: Model
• Explainable Teams
Model
Data Analytics • Psych. Theory Explanation
• Explanation Effectiveness
Multimedia Data Model of Explanation
Interface
Induction • Computational
Model Explanation
Teams Measures
• Consulting
• User Satisfaction
• Mental Model
• Task Performance
Autonomy
ArduPilot & Evaluator • Trust Assessment
SITL Simulation • Correctability
• Theories of Explanation
o Describe how you will summarize the current psychological theories of
explanation
o Describe how this work will inform the development of the TA1 XAI
systems
o Describe how this work will inform the definition of the evaluation
framework for measuring explanation effectiveness by the XAI evaluator
• Computational Model
o Describe how you will develop and implement a computational model of
explanation
o Identify predictions that might be tested with the computational model
o Explain how you will test and refine the model
• Model Validation
o Describe how you will validate the computational model against the TA1
evaluation results in Phase 2 of the XAI program
o The government evaluator will not conduct evaluation of TA2 models
Analysze Results
Prep for Eval Analyze Eval Analyze Eval
Evaluator Define Evaluation Framework Prep for Eval 2 Prep for Eval 3 &
Eval 1 1 Results 2 Results 3
Accept Toolkits
Deliver
Summarize Current Psychological Develop Computational Model of Refine & Test
TA 2 Computational
Theories of Explanation Explanation Computational Model
Model
Meetings
KickOff Progress Report Tech Demos Eval 1 Results Eval 2 Results Final
2017 2018
APR MAY JUN JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV
Develop
Summarize Current Psychological
TA 2 Computational
Theories of Explanation
Model
Meetings
KickOff Progress Report Tech Demos
Analysze Results
Analyze Eval Analyze Eval
Evaluator Prep for Eval 2 Prep for Eval 3 &
Results 2 Results 3
Accept Toolkits
Develop Deliver
Refine & Test
TA 2 Computational Computational
Computational Model
Model Model
Meetings
Eval 1 Results Eval 2 Results Final
29
Distribution Statement "A" (Approved for Public Release, Distribution Unlimited)
F. Deliverables
• Slide Presentations
• XAI Project Webpage
• Monthly Coordination Reports
• Monthly expenditure reports in TFIMS
• Software
• Software Documentation
• Final Technical Report