System Safety and Risk Management

System Safety and Risk Management
Sunday, October 06, 2013
Presented by AFS-900
Amer M. Younossi
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
Module 6 Challenges Summary
Objectives/Functions
Modules
1 Why Flight Standards Is Implementing
2 3 4 5 6
Risk Concepts System and Organizational Principles Human Factors Data and Analysis Risk Management
2
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
Module 1 Outline
How did we get here?

Why do system safety? AFS-900 objectives and functions Where is our authority? System safety challenges
Summary
3
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
How Did The FAA Get Here?

90-Day Safety Review White house and congressional mandates to reduce the accident rate by 80% in 10 years
4
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
90-Day Safety Review
The task force categorized issues into six general areas: certification policy and process resource targeting to address safety risks newly certificated air carrier operations and growth outsourcing and varied fleet mix inspector and air carrier guidance material inspector resources
5
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
Policy Statement of the FAA
Aggressive attitude with respect to aviation safety Pro-active approach Air carrier to maintain the highest possible degree of safety Achieve this in the most economical and efficient manner to the government and the carrier
Source: FAA Order 1000.1A
6
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
FAA Mission Statement
Setting regulations and standards for: air carriers Flight Standards Vision air commerce To be recognized and respected worldwide as the foremost authority on system air agencies safety approaches for regulation and airmen, etc. certification
Accomplishing certification, surveillance, inspection, investigation and enforcement activities Managing the systems for registry of ... aircraft airmen
7
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
Why Do System Safety
Air traffic is increasing Aviation operations are becoming more complex FAA oversight staff and resources are constrained We can no longer afford to function as a direct source of QC Systems approach is proactive Stresses process improvements System safety is good business
8
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
System Safety Defined

The application of special technical and managerial skills in a systematic, forward-looking manner to identify and control hazards throughout the life cycle of a project, program, or activity.
Source: System Safety Course, FAA Academy
9
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
System Safety Philosophy
System safety is: Validation of regulatory compliance Review and validation of systems and entire programs, not portions Application of safety attributes Application of risk management
10
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
AFS-900 Objectives
Air carrier certificates are our gold standard to measure how well we are accomplishing our responsibilities.
Dave Gilliom ~ AFS-900
Standardize air carrier certification Standardize systems and targeting risks Centralize information management functions Six Processes are ISO-9000 Certified
11
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
ATOS Model
Policy --------------------------------------------------
Module 2
Certificate Management
Module 3
Surveillance Resource Mgmt
Module 1
System Configuration
Module 4
Surveillance
Module 8
Implementation
Module 5
Reporting
Module 7
Analysis
Module 6
Evaluation
12
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
AFS-900 Functions
Certification, Standardization, and Evaluation Team (CSET)

Certification Surveillance/Evaluation Planning Aircraft conformity team Technical support program Improve the certification and surveillance processes for air carriers Ensure regulatory compliance and incorporate a systems approach Enhance the surveillance process Standardize the surveillance processes
Air Transportation Oversight System (ATOS)

Flight Standards Safety Analysis Information Center (FSAIC)

Provide analytical support to Flight Standards Identify industry-wide risk areas Partnership with aviation safety research institutions
13
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
AFS-800 Function
System Safety Approach for General Aviation (SAGA)

Improve the oversight process for general aviation Open communications with air operators. Identify focus areas to target, and provide feedback to operators.
14
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
Integrated System Safety Efforts

SASO ~ Systems Approach for Safety Oversight
Business process procedures, guidelines, and automation tools Data quality enhancement and database standardization Risk management/analysis procedures and automation Interconnectivity of agency and industry databases
15
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
Where Is Our Authority?
Title 49 44701, 44702 Current 121 Rules
16
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
USC Title 49
Sec. 44702. Issuance of certificates
... (B) considerations. - When issuing a certificate under this chapter, the administrator shall (1) consider (a) the duty of an air carrier to provide service with the highest possible degree of safety in the public interest. ...
17
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
USC Title 49
Sec. 44701. General requirements
(A) promoting safety. The administrator of the FAA shall promote safe flight of civil aircraft in air commerce by prescribing ... (2) regulations and minimum standards in the interest of safety
18
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
14 CFR Part 121 Rules

14 CFR part 121.135 (a) (1) each manual required by 121.133 must (1) include instructions and information necessary to allow the personnel concerned to perform their duties and responsibilities with a high degree of safety; ...
19
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
System Safety Challenges
Operators organizational structures Systemic orientation and completeness of regulations Defining and measuring safety FAA and industry cultures Human factors Availability of valid, reliable data that are representative of the system Risk management application
20
Module 1 Outline
Module 2 How
Module 3 Why
Module 4
Module 5 Authority
Summary

90 day safety review System Safety AFS-900
21

Module 2
Risk Concepts
Module 1 Outline
Module 2 Fundamentals
Module 3 Definitions
Module 4 Concepts
Module 5
Module 6 System Safety Summary
Risk Factors
Module 2 Outline
Fundamentals of Risk
Definitions of Risk Common Concepts of Risks Risk Factors/Controls System Safety
Summary
23
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
Risk Management
Risk management is a core process to system safety.

Risk management can be practiced at all levels and in all types of aviation.
Safety is not equivalent to risk free
U.S. Supreme Court, 1972
Risk management is a more realistic term than safety

Jerome Lederer, 1928
24
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
Basic Risk Concepts
Event Accident, Incident, Occurrence Undesired event resulting in loss

Hazard Condition that can cause event Risk Likelihood / severity of event associated with the hazard
Exposure to hazard...

Likelihood of occurrence Severity of consequences
25
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
Schools of Thought
Product Development (Traditional SS)

Organizational Psych (e.g. James Reason) Project Management (Risk Management e.g. SEI) AFS approach is a blend
26
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
Definition(s) of Risk
Several definitions exist, based upon the context for which they were developed
More than one are used in AFS programs
All have a common underlying set of concepts
27
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
Common Concepts
What is it that I will run into, hit, trip on, etc.? Hazard, condition, failure, threat Something bad can happen to us Undesired event, mishap, consequence
How likely is it to happen? Likelihood, probability (impact)1 How bad will it hurt if it does? Severity, impact (impact)1
28
1
Source: Product school
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
AFS Accepted Definition

Safety risk is an expression of the impact of an undesired event in terms of event severity and event likelihood.
Source: FAA Order 8040.4, Safety Risk Management - Product School
29
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
Risk Factors and Control
Most system risk is incurred through factors involving severity or likelihood

Most deal with factors increasing likelihood of a hazardous condition Risk controls typically: Help detect likelihood-increasing factors Lower the likelihood Eliminate the hazard
30
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
Risk Factors
Things about the inherent hazard, system design, or organizational performance that affect risk likelihood or severity.
Task complexity
Environmental factors
Difficult procedures Poor or missing supervisory controls
Poor Communications
31
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
Risk Controls (Mitigation)
Serve to reduce likelihood or severity

Be wary of substitution risk Controls must recognize human factors and culture
32
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
Risk Involving Organizations
Inherent risk Cause of the problem

Design deficiencies Failure provide controls on the problem Performance Deficiency Failure to comply with design failure to the design to control risk
33
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
Inherent Hazards and Risk

?
Environment
Operation
Hazard
34
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
System Design and Risk

System
Procedures Interfaces Resources Process Controls Process Measures
35
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
System Performance
Conformance/compliance
Are we complying with the design of the system?
Effectiveness
Does the system accomplish its intended function (control the risk factors)?
36
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
Regulations as Risk Controls
Based upon public law (49 USC 447)

Regulations and approved programs may be risk controls.
Effectiveness is bounded by specificity of regulations, programs, manuals, etc.

Proper application is essential!
37
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
System Safety Model
The FAA will use the System Safety Process Model as a basis.
Model was adapted from Nicholas J. Bahr by AFS-900 and ASY-300 for use by Flight Standards
38
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
System Safety Process

Define Objectives
Risk Analysis: Analyze Hazards & Identify Risks
Risk Management
Hazard Identification: Identify Hazards & Consequences
System Descriptions
Risk Assessment: Consolidate & Prioritize Risks
Decision Making: Develop an Action Plan
Modify System/Process
System/Process Review
Validation of Control: Results for Further Action
39
Documentation
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors

1 - Identify the task(s)/functions
2 - Describe the environment 3 - Identify the hazards 4 - Evaluate the risk ...
40
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors

5 - Identify and evaluate mitigating strategies
6 - Decide on a course of action
7 - Evaluate effectiveness of controls
41
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
Risk Perceptions
Carelessness and overconfidence are more dangerous than deliberately accepted risk
- Wilbur Wright, 1901
Wilbur Wright gliding, 1901 Photographs: Library of Congress
42
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
Summary
43

Module 3
System and Organizational Principles
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Operator Objectives
Regulators Objectives
Module 3 Outline
What is a system?
Tasks, functions and environments
System attributes Operator's Objectives Regulators Objectives Summary
45
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Operator Objectives
System and Organizational Principles

Accidents Are Not Due to Lack of Knowledge, but Failure to Use the Knowledge We Have.
- Trevor Kletz, What Went Wrong?
July 17, 1996 8:31:10 PM

TWA Flight 800 Boeing 747-131, N93119 153rd aircraft built (1971)
46
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Operator Objectives
Whats a System?
A system is a composite of people, tools, procedures, materials, equipment, facilities, and software operating in a specific environment to perform a specific task or achieve a specific purpose, support, or mission requirement.
- Roland and Moriarty, System Safety Engineering and Management
Systems
Procedures
People Software Tools Equipment

Facilities Materials
...a set of components that act together as a whole to achieve a common goal
47
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Operator Objectives
Situation - Environment
What Are the Conditions?
Physical environment Facilities Airports Weather patterns Operational environment Type of operation Type of equipment Employee demographics Financial factors
48
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Operator Objectives
Situation - Tasks and Functions

What Is Being Done?
Tasks - accomplished by humans Pilot tasks - Advanced Qualification Program (AQP) Maintenance tasks - Work cards Functions - organizational structures to accomplish tasks
49
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Operator Objectives
System Safety Attributes

Responsibility Interfaces Authority
Process Measurement
System Safety Attributes

Procedures Controls
50
What is a Process?
A Process is a set of tasks, work-flows and information flows that produce a desired result.
Source: System Safety Course, FAA Academy
If you can't describe what you are doing as a process, you don't know what you're doing.
W. Edward Deming
US business advisor & author (1900 - 1993)
51
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Operator Objectives
System Based Process

Controls
Inputs
Process
Outputs
Resources
52
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Responsibility
Operator Objectives
Authority
Jobs and Tasks

Responsibility / Authority
Procedures
System Safety Attributes Procedures
Function Job Input

Task Task Task Task
Output
53
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Interfaces Authority Responsibility
Operator Objectives
Interfaces Between Tasks

Procedures Interfaces Controls
Controls
Function Job Input

Task Task Task Task
Job
Task Task Task Task
Output
54
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Operator Objectives
And More Tasks

Controls
Function Job Input

Task Task Task Task
Job
Task Task Task Task
Output
Job Input
Task Task Task Task
55
Resources Process Controls System Safety and Risk Management
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Operator Objectives
Feedback and Information

Process Measurement
Controls
Function Job Input

Task Task Task Task
Job
Task Task Task Task
Status Job Input

Task Task Task Task
Process Measures
56
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Operator Objectives
Completed Function
Process Measurement
Controls
Function Job Input

Task Task Task Task
Job
Task Task Task Task
Output
Status
Job Input
Task Task Task Task
Process Measures
57
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Operator Objectives
Operators Objectives
What regulatory requirements are trying to achieve

Designing systems that ensure regulatory compliance and safety
58
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Operator Objectives
Organizational Decision Factors

Outcome relatively uncertain Measured by absence of negative events
Delicate and Complex Balance
Outcome relatively certain Measured in positive terms
Resources Available Money Equipment Personnel/Expertise Available Time
Safety Goals
Decision Makers
Production Goals
Protection
Defensive Filters
Production
59
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Operator Objectives
Understand what the system is designed to do

Assure the operators are complying with applicable regulations Assure that operators follow its approved or accepted programs
60
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Operator Objectives
System Safety Philosophy

System safety is:

Validation of regulatory compliance Review and validation of systems and entire programs, not portions Application safety attributes Application of risk management
61
Module 1 Outline
Module 2 System
Module 3 Attributes
Module 4
Module 5
Module 6 Summary
Operator Objectives
Summary
Importance of system design

Delicate balance within an organization Understanding the objectives The problems are usually related to human error
62

Module 4
Human Factors
Module 1 Outline
Module 2 Human Error
Module 3
Module 4
Module 5
Module 6 Inspections Summary
Understand Causes
Assessing Organizational Health
Module 4 Outline
Human Performance Human Error Understand Causes Summary
64
Human Performance
Basic
motor reflexes Skill-based performance Rule-based performance Knowledge-based performance
65
Human Factors in the Decision Process
Information processing Situation awareness Cognitive (Mental) workload CRM (Cognitive Resource Management)
Situation Awareness
Situation awareness is the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future.
-Mica Endsley, 1993
67
Human Thought Process
Mental Model
Perception
Comprehension
Projection
Source: Dr. Mica Endsley
68
Situation Awareness Errors
Level 1 - Failure to correctly perceive the situation. Level 2 - Failure to comprehend the situation. Level 3 - Failure to comprehend the situation into the future.
69
Situation Awareness Analysis
ASRS data. 169 errors in 113 incidents. 80.2% Level 1 Errors 16.9% Level 2 Errors 2.9% Level 3 Errors
70
Cognitive (Mental) Workload

Human working memory is limited The magic 7 plus or minus 2 (Miller, 1956) Fatigue can reduce this Interruptions can bump material in working memory Information needed for a typical instrument approach can exceed the capacity available
Fatigue and Workload
Activities late in flight have high workload Capacity is diminished by fatigue High accident potential (Boeing studies)
Module 1 Outline
Module 3
Module 4
Module 5
Understand Causes
Human Error
Most accidents are human related Accident Rates 1977 1992

Mechanical/Environmental Factors Human Error
2002 - What can we do?
73
Human Error
Error is ubiquitous - its a human thing. Most accidents are human related. Rules and attention to detail are essential in a systematic approach to safety, but... Regulations and exhortations often have limited effectiveness in error prevention. Be careful only works so far.
Human Error
Slips/lapses are most often associated with psychomotor or automatic skills.
Skill or action-type problems most often responsible for frequent, low-severity events.
Mistakes are often associated with cognitive tasks such as decision-making.
Decision-type problems more often responsible for highseverity events (fatal accidents).
Many inadvertent rule violations are really decision or knowledge problems.

New Zealand GA Study
200 GA accidents analyzed Decision errors: 31% of minor/non-injury accidents 63% of fatal accidents Action (skill) errors: 48% of non-injury accidents 16% of fatal accidents
76
Error Mitigation
Skill-based errors
Rule-based errors Knowledge-based
77
Module 1 Outline
Module 3
Module 4
Module 5
Understand Causes
Understand Causes
Active Failures
Latent Failures Air Ontario - Dryden Pilot Error Organizational Failures
78
Module 1 Outline
Module 3
Module 4
Module 5
Understand Causes
Latent Failures
Operational Personnel
Diminishing Returns Redirect Safety Efforts Judging Safety Health
79
Module 1 Outline
Module 3
Module 4
Module 5
Understand Causes
James Reason Model

Organization
Organizational Factors
Communication Management Structure Goals Policy & Procedure
Workplace
Local Factors
Physical Environment Operational Environment Task Design
People/Team
Active Failures
Performance Errors - Mistakes - Slips - Lapses Violations
Controls Outcome
Accidents
Active Error Path
Latent Error Path

Source: The organizational connections to hurt bodies and bent metal, by: Dr. James Reason
80
Module 1 Outline
Module 3
Module 4
Module 5
Understand Causes
James Reason Model

Organization
Workplace
Local Factors
People/Team
Active Failures
Controls Outcome
Accidents
Active Error Path
Latent Error Path

81
Module 1 Outline
Module 3
Module 4
Module 5
Understand Causes
James Reason Model

Organization
Workplace
Local Factors
People/Team
Active Failures
Controls Outcome
Accidents
Active Error Path
Latent Error Path

82
Individual Differences - Attitudes
Antiauthority (dont tell me!). Impulsivity (do something quickly!). Invulnerability (it wont happen to me). Macho (I can do it). Resignation (whats the use?).
83
Individual Differences Personality (Big 5)

Openness
to experience Conscientiousness Extraversion/Introversion Agreeableness Neuroticism (emotional stability)
84
Group Differences - Culture

Uncertainty
Avoidance Masculinity/Femininity Individualism/Collectivism Power Distance Short/Long-Term Orientation
85
Module 1 Outline
Module 3
Module 4
Module 5
Understand Causes
Managing Human Factors
Safety Culture
Incident Investigation Feedback Increase Tolerance Limit Error Potential
Closed Loop
86
Module 1 Outline
Module 3
Module 4
Module 5
Understand Causes
Organizational Response
Denial
Repair Reform
87
Module 1 Outline
Module 3
Module 4
Module 5
Understand Causes
Addressing Human Factors
Humans Will Make Mistakes

Controls Increase Tolerance Lessen Potential
Using the checklist is statistically safer than thinking.

- British Airways Check Airman
88
Module 1 Outline
Module 3
Module 4
Module 5
Understand Causes
Summary
Why Study Human Factors

Active and Latent Failures Errors and Violations Controls Managing Human Factors
89

Module 5
Data and Analysis
Module 1 Outline
Module 2
Module 3
Module 4 Question
Module 5 Analysis Data
Module 6 Contribution of Data Summary
Decision-Making Hierarchy
Module 5 Outline
Decision-Making Hierarchy Question Analysis Data Contribution of Data to System Safety Summary
91
Module 1 Outline
Module 2
Module 3
Module 4 Question
Action
Decision
Knowledge
Information Data Observation
92
Module 1 Outline
Module 2
Module 3
Module 4 Question
Biases
Humans tend to use simple rules of thumb called heuristics. These heuristics simplify mental processes but they can also lead to over simplification and error. Typical biases include: Representativeness heuristic Salience - most attention-getting rather than most useful Availability heuristic - recency Confirmation bias - tendency to focus on initial hypothesis As if weighting (all info assumed of same quality) 93 Conservatism System Safety and Risk Management
Module 1 Outline
Module 2
Module 3
Module 4 Question
Action
Decision
Question/Problem
Knowledge
Information Data Observation
Analysis
Data
94
Module 1 Outline
Module 2
Module 3
Module 4 Question
Counteracting Biases
Question
Bias can be reduced through the careful use of data Clear Empirical Question
Data
Good Quality Data
Analysis
Appropriate Data Analysis
95
Module 1 Outline
Module 2
Module 3
Module 4 Question
Question
Problem Definition - What do we want to know? System definition and analysis Description of the situation Risk or Problem statement Operational definitions are important
96
Module 1 Outline
Module 2
Module 3
Module 4 Question
Analysis
Types of information

Records Inferences Judgments
Important to separate reports (facts) from inferences and judgments

Recording only judgments or conclusions will usually decrease data reliability
Data must fit:

Analysis needs Nature of observation Observation situation
97
Module 1 Outline
Module 2
Module 3
Module 4 Question
Analysis
Aggregation
Cannot create system level analyses by rolling up unrelated data
Result will depend more on mix of data than performance of any element
Trends are especially sensitive
Cant mix apples and oranges and compare fruit baskets
98
Module 1 Outline
Module 2
Module 3
Module 4 Question
Analysis
Comparisons
Many comparisons are legitimate:

Norms / thresholds Association with factors Comparisons between subgroups (e.g. location, aircraft fleet) Trends
Comparisons should be avoided in uses to rate offices or people
99
Module 1 Outline
Module 2
Module 3
Module 4 Question
Analysis
Trend Analysis

A special type of comparison Often confused with generalization Must be based upon a reliable, quantifiable metric at progressive time points Must have good internal validity Must have data available in adequate quantity, regular intervals
100
Module 1 Outline
Module 2
Module 3
Module 4 Question
Data
Data Torturing
Inappropriate use of data
Inappropriate comparisons to averages, specifications, targets and percentages Smoothing Limited point trends Use of raw counts
Source: Dr. Marcey Abate, Sandia National Labs, 1998
101
Module 1 Outline
Module 2
Module 3
Module 4 Question
Data
What is data?
Information organized for analysis or used as a basis for decision making
Factual data is not a record of conclusions, concerns, or opinion

Factual data is a record of facts upon which to base conclusions
102
Module 1 Outline
Module 2
Module 3
Module 4 Question
Data
Automation and Data
Good quality automation is essential to manage the workload of providing data, but
Neither automation nor analysis can add information which is not in the data
Computers do not produce new sorts of errors. They merely provide new and easier opportunities for making old errors.
Source: Trevor Kletz, Wise After the Event
103
Module 1 Outline
Module 2
Module 3
Module 4 Question
Data
Sampling
Sample data represents a larger group with data from a small group Distribution of observations is important Must identify factors that can effect performance Problems can be created in rollups
Normal trends (e.g. season) may be involved

Statistical confidence is unlikely. Confidence in what?
104
Module 1 Outline
Module 2
Module 3
Module 4 Question
Data
Text and Narratives
Text is very flexible, however, very unreliable as as a data source
Narratives are valuable adjuncts to explain observations

Text may be coded into quantitative form using an appropriate taxonomy; however, this procedure is very time-consuming and adds another source of error Anecdotes may not be representative of any larger group
105
Module 1 Outline
Module 2
Module 3
Module 4 Question
Data
Data Quality Attributes

Validity Reliability Sensitivity Representativeness
106
Module 1 Outline
Module 2
Module 3
Module 4 Question
Data
Data Validity
Validity means that the data measure what they are intended to measure
Validity is weakened or reduced by:

Vague Definitions Insufficient Evaluator Training Casual Data Collection Methods Operator Discretion
Source: AQP Data Management Guide, 1998
107
Module 1 Outline
Module 2
Module 3
Module 4 Question
Data
Data Reliability

Consistency between raters Consistency among items Stability in measurements over time Factors effecting data reliability

Definition of item(s) to be observed Standards of evaluation Training of inspectors Type of measurement used
108
Module 1 Outline
Module 2
Module 3
Module 4 Question
Data
Data Sensitivity
Sensitivity means that small gradations or variations in the parameter being measured are reflected in some variation in the measurement
Multi-point scales allow more sensitivity than pass/fail grading

Using multiple items to measure individual components of performance and then combining them into a composite score or index could increase sensitivity
109
Module 1 Outline
Module 2
Module 3
Module 4 Question
Data
Data Problems Confounding
Confounding occurs when two different factors can explain the same result When research is confounded, it is impossible to determine what variable is responsible for any obtained difference in performance When no confounding is present, a research study is said to have internal validity
Contamination
Behavior changes as a result of evaluation process Observer effects Announced inspections
110
Module 1 Outline
Module 2
Module 3
Module 4 Question
Data
Improving Data Collection

Develop focused factual information requirements Document the requirements Seek answers that are: System configuration information System performance information
Data must be factual and representative
111
Module 1 Outline
Module 2
Module 3
Module 4 Question
Contribution of Data to System Safety
System Validation
Hazard Identification Risk Analysis/Assessment
112
Module 1 Outline
Module 2
Module 3
Module 4 Question
Summary
Collection of pertinent and quality data, and appropriate analysis will enhance current and future programs Data-driven surveillance programs will result in enhanced System Safety, whereby issues and hazards are identified early and addressed adequately
113

Module 6
Risk Management
Module 1 Outline
Module 2
Module 3
Module 4 Practical Application
Module 5
Module 6 Summary
Case Study
Module 6 Outline

Practical Application Case Study Summary
115
Module 1 Outline
Module 2
Module 3
Module 5
Module 6 Summary
Case Study
What, How, Why
System Safetys primary objective

Consider the entire system Understanding the operators accident prevention program
117
Module 1 Outline
Module 2
Module 3
Module 5
Module 6 Summary
Case Study

Define Objectives Objectives Define Risk Analysis: Analysis: Risk Analyze Hazards Hazards & Analyze & Identify Identify Risks Risks
Risk Management
HazardIdentification: Identification: Hazard Identify Hazards Hazards & Consequences Identify Consequences
System Descriptions System Descriptions
Risk Assessment: Assessment: Risk Consolidate & Prioritize Risks
Decision Making: Decision Making: Develop an Action Plan
Modify Modify System/Process System/Process
System/Process Review System/Process Review

Yes
Validation of Control: Validation Control:

Results for Further Action
No
118
Documentation
Module 1 Outline
Module 4 Concepts
Module 5
Risk Factors
Risk Assessment Matrix

SEVERITY LIKELIHOOD Frequent Probable Occasional Remote Improbable Negligible Marginal Critical Catastrophic
High Serious Medium Low
124
Outline
Practical Application
Case Study
Summary
April 28, 1988 Near Maui, Hawaii Flight 243 - Boeing 737-200 (N7371I)
128
Puzzle
Module 1 Outline
Module 2
Module 3
Module 5
Module 6 Summary
Case Study

Define Objectives
Risk Analysis: Analyze Hazards & Identify Risks
Risk Management
Hazard Identification: Identify Hazards & Consequences
System Descriptions
Risk Assessment: Consolidate & Prioritize Risks
Decision Making: Develop an Action Plan
Modify System/Process
System/Process Review
Validation of Control: Results for Further Action
129
Documentation
Module 1 Outline
Module 2
Module 3
Module 5
Module 6 Summary
Case Study
Summary
Understand... the System Safety Process how It Will Assist You in Your Job
130
Additional Sources
Training Available at FAA Academy in Oklahoma City
ATOS Overview Course Length: 24 hours Course Description: This course is designed to familiarize industry personnel, international students, and government agencies other than FAA with ATOS. It will provide instruction on the ATOS access, ATOS system, including the tools, the policy and guidance, that make the ATOS system work.
System Safety Course Length: 24 hours Course Description: The purpose of this course is to train FAA aviation safety inspectors on system safety and risk management processes and the importance of both as they relate to systems management. This will be accomplished by demonstrating how system safety and risk management are used throughout the FAA environment.This course is available to industry personnel based on class availability
131
Additional Sources
System Safety and ATOS Overview Course Length: 32 hours Course Description: This seminar is designed to provide an introduction to system safety and risk management processes as they are used by the FAA for air carrier oversight. The seminar also familiarizes industry personnel with the FAA's Air Transportation Oversight System (ATOS). The four-day seminar is conducted on location by special arrangement with the sponsoring organization for up to 50 students.
Contact: Mr. Joseph Souza 405-954-4321.
http://www.faa.gov/avr/afs/ Don Arendt - AFS-900 FSAIC Manager 703-661-0516
132

System Safety and Risk Management

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

System Safety and Risk Management

Hochgeladen von

Copyright:

Verfügbare Formate

System Safety and Risk Management

Sunday, October 06, 2013

Module 6 Challenges Summary

Module 6 Challenges Summary

How did we get here?

Module 6 Challenges Summary

How Did The FAA Get Here?

Module 6 Challenges Summary

90-Day Safety Review

Module 6 Challenges Summary

Policy Statement of the FAA

Module 6 Challenges Summary

FAA Mission Statement

Module 6 Challenges Summary

Why Do System Safety

Module 6 Challenges Summary

System Safety Defined

Module 6 Challenges Summary

System Safety Philosophy

Module 6 Challenges Summary

Module 6 Challenges Summary

Module 6 Challenges Summary

Certification, Standardization, and Evaluation Team (CSET)

Air Transportation Oversight System (ATOS)

Flight Standards Safety Analysis Information Center (FSAIC)