Engineering The Architecture Of

Distributed Control Systems

January 2002

Eric Runnerstrom
MPR Associates
erunnerstrom@mpr.com
703-519-0200
Distributed Systems Architecture

This methodology was developed by MPR to enable the rigorous

design of distributed control systems that will perform as required.

The methodology was developed and implemented as part of the

Navy Research Laboratory’s Damage Control Architecture for
Reduced Manning (DC-ARM) Program.

The methodology ensures that the technology demonstrated by the

DC-ARM program can be effectively applied to other platforms.
It also could be be applied to other types of distributed systems,
such as distributed sensor systems.

Jan. 2002
2
Distributed Systems Architecture

Outline
• Definition Of Supervisory Control System
• Premise
• The Process
• Firemain Example
• Further Development

Jan. 2002
3
Distributed Systems Architecture

Definition Of Supervisory Control System

A system, automated to some extent, that:

• Monitors and controls multiple, integrated sub-systems

• Enables a human supervisor to interact with the sub-systems

• Provides the information necessary so that the responses of

sub-systems and the actions of personnel can complement one
another.

Jan. 2002
4
Distributed Systems Architecture

Premise
• Projects that implement complex control systems
often experience:
– Cost overruns
– Schedule delays
– Performance that does not meet expectations, even after
additional delays and costs, and substantial changes to
the system

Performance
Performanceproblems
problemsare
areexacerbated
exacerbatedwhen
when
systems
systemsoperate
operateoff
offdesign
designor
orequipment
equipmentfails.
fails.

Jan. 2002
5
Distributed Systems Architecture

Premise
• The principles for designing distributed control systems
are evolving
– Industry is able to build distributed control systems, but is still
learning how to engineer their design
• Engineering the architecture of distributed control systems
will
– Provide a basis for preventing the cost, schedule, and performance
problems often experienced during development
– Enable optimizing the architecture of the control system with
respect to acquisition program criteria
– Enable effective human-systems integration

AAmethodology
methodologyfor
forengineering
engineeringthe
thearchitecture
architectureof
of
distributed
distributedcontrol
controlsystems
systemsadvances
advancesthe
thestate-of-the-art.
state-of-the-art.
Jan. 2002
6
Distributed Systems Architecture

The Process
 Define Control Decisions
 Develop Control Decision Logical Architecture
 Define Candidate Hardware Architectures
 Evaluate Candidate Hardware Architectures &
Select Optimum
 Develop Software Architecture

Jan. 2002
7
Distributed Systems Architecture

1. Define Control Decisions

Functional Analysis to define control decisions
FIRE Characterize Fire
LOCATION No Boundary
FIRE SEVERITY

Fire Severity?
NO FIRE

"SMALL" FIRE
Needed
Functional Analysis Enables:
Characterize Damage -
Top Level

Compartments
"MEDIUM" FIRE Water Mist
YES • System functional requirements driven
"LARGE" OR Functioning In
Containing Damage "FULLY DEVELOPED"
FIRE
Fire Space? Control Water
Mist System
by top level program requirements
• Fundamental integration of:
Compartment Boundary
Characteristics NO
Needed
Maintain Fire

– Sub-systems with one another

Determine Boundary With
Boundary Water Mist Maintain
Functioning In Water Mist
Compartments YES Boundary With
Boundary Spaces? Water Mist

Calculate Time For

– Sub-systems and controls
NO Fire Space To Reach
Predict
Personnel ** TV = Time To ** TH = Time To
500 C = T(500)
– Humans and sub-systems
Performance Vertical Spread = Horizontal Spread
T(500) + 5 Min. = T(500) + 9 Min. Characterize
Fire • Database of attributes for each control
Can Personnel Set
Boundary Before Fire
Spreads?
FIRE PREDICTED
IN COMPARTMENT decision
NO
Define
Operational YES
Priorities
Is Boundary Space High
Compartment Enough Priority To Apply NO
Characteristics Personnel?

YES
Maintain Maintain Fire
Boundary With Boundary With
Personnel Personnel

Jan. 2002
8
 Distributed Systems Architecture

2. Develop Control Decision Logical Architecture

Three levels of control logic for systems or compartments:
• Device Level Logic – Device Level Control Logic requires only inputs
available at the controlled device itself
– Similarly, Compartment Level Control Logic requires
only inputs available at the compartment itself

• System Level Logic – System Level Control Logic requires inputs from
more than one device in the system
– Similarly, Zone Level Control Logic requires inputs
from more than one compartment in the zone

• Ship Level Logic – Ship Level Control Logic requires inputs from more
than one system or zone
Other applications could have different boundaries for the control decision logic levels.

Jan. 2002
9
Distributed Systems Architecture

2. Develop Control Decision Logical Architecture

Ship Level Logic requires Example: Correlation of firemain
Ship Level
input from more than one system. status & compartment status is
Logic
plant level logic.

System Level Logic requires

inputs from more than one Example: Firemain flow
device in the system. System balance is system level logic. System
A B
Logic Logic

Device Level Logic requires Example: DC-ARM

only inputs available at the Smart Valve uses
device itself. device level logic.
Smart Smart Device #3
Device #1 Device #2
Device Logic Device Logic Sensors/Input

Sensors/Input Actuator Sensors/Input Actuator

Jan. 2002
10
Distributed Systems Architecture

2. Develop Control Decision Logical Architecture

• Functional requirements for the control system are
defined by:
– Definitions and attributes of control decisions
– Control decision logical architecture
• Consider providing redundant logic for the same
function to enhance reliability and survivability
– Recognize trade-off of increased complexity and cost

Control
Controldecision
decisionlogical
logicalarchitecture
architectureprovides
providesaabasis
basisfor
for
the
thesynthesis
synthesisof
ofcandidate
candidatehardware
hardwarearchitectures
architecturesthat
that
meet
meetthe
thesame
samefunctional
functionalrequirements.
requirements.
Jan. 2002
11
Distributed Systems Architecture

3. Define Candidate Hardware Architectures

• Hardware Architecture =
– Control functions that will be performed on each processor
– Where processors will be located that collect data and
execute control decisions
– Redundancy and separation of processors
• Redundant processors can improve reliability
• Separated, redundant processors can improve
survivability
– Consider need for communications among data sources and
processors
• Communications load typically increases significantly
during a casualty

Jan. 2002
12
Distributed Systems Architecture

3. Define Candidate Hardware Architectures

• Processors can be located at: • Decision logic can be
– Device Level; e.g. valves, pumps, executed in processors at
fans, dampers, A/C plants almost any level; for example,
– Compartment Level extremes include:
– System Level; e.g. controllers for – All decisions executed at device
firemain, ventilation, chilled level processors
water, electrical distribution – All decisions executed at a ship
– Zone Level level processor
– Ship Level • Hardware architecture that
– Other locations mirrors decision logical
architecture may provide best
performance
Communications
Communicationsload,
load,cost,
cost,reliability
reliability&&survivability
survivabilitycan
canvary
vary
significantly
significantlydepending
dependinguponuponthe
thehardware
hardwarearchitecture.
architecture.
Jan. 2002
13
Distributed Systems Architecture

4. Evaluate Candidate Hardware Architectures

• Criteria are determined by the acquisition program
• Typical criteria might include:
– Acquisition Cost
– Life-Cycle Cost
– Reliability
– Survivability
– Ease of Troubleshooting and Maintenance
– Affordability of Likely Upgrades
Select
Selectthe
thehardware
hardwarearchitecture
architecturethat
thatprovides
providesthe
thebest
best
balance
balanceamong
amongacquisition
acquisitionprogram
programcriteria
criteriaand
andpriorities.
priorities.
Jan. 2002
14
Distributed Systems Architecture

5. Develop Software Architecture

• Modular software architecture that mirrors control
decision logical architecture probably is a good
baseline
– Provides logically consistent basis for:
• Development
• Troubleshooting
• Maintenance
– Maximizes “plug and play” capability for upgrades
• Adjust software architecture considering:
– Hardware architecture
– Communications

Jan. 2002
15
Distributed Systems Architecture

DC-ARM
SHADWELL Firemain Example
Following slides are an example of applying
the architecture engineering process to the
DC-ARM firemain aboard SHADWELL.

Jan. 2002
16
 Distributed Systems Architecture

Firemain Example - Functional Analysis

Firemain Firemain Self Monitor

**Monitoring Data includes data from sensors for component material condition,
Monitoring Data**
Pump, Valve, & Pipe Segment Availability component operating data, and system hydraulic data, as needed.

Pump Status & Valve Position

Human Supervisor's Evaluation
Valve & Pump Response To Commands Assess Material Condition & Readiness of
Pressures Firemain

Commands To Operate Component Tag-Out

PM & CM Performed
Pumps & Valves
Preventive & Corrective Component Material
Maintenance Requirements Condition & Readiness

Control Firemain &

Provide Information To Perform Firemain Maintenance
Isolate Ruptures &
Maintain Vital Services
Operating Status of Firemain: Pump & Valve Status;
Does Condition Fluid Pressures, Flows, Temp., Etc., As Needed.
Indicate Probable
NO
Damage? Readiness Information for Each Firemain Service;
Fire Plug Availability E.G. Fireplugs, Sprinkling, AFFF, Etc.
Continue
Fire Plugs Vital To Assigned Tasks
YES
Location/Compartment of Damage Evaluate Readiness of
Description of Damage Firemain

Typically, data is passed to links without waiting for human

supervisor's assessment. Supervisor's assessment, when
available, updates and may override previous assessments.

Characterize Predict Maintain Attack Small, Link To Monitor Ship Systems, Link To Attack Link To Link To
Compartments, & Pre-Damage Major Fire
Damage - Top Personnel Boundary With Medium, Large, Or Predictions
Restore Firemain
Firemain Reflexive
Level Performance Personnel Fully Developed Operation
Link To Respond To
Fire Link To Set Probable Fire & Extinguish
Boundaries Minor Fire

Control Firemain Maintain Firemain

Jan. 2002
17
Distributed Systems Architecture

Firemain Example - Functional Analysis

• Examples of attributes defined for each action in the functional
analysis
– Level in control logical hierarchy
• Device, system, or ship
– Primary or back-up means of accomplishing the action
– Discrete or continuous action
• For discrete actions, need an initiating event
– Intended effect of the action
– Effects if the action is not performed when it should be
– Effects if the action is performed when it should not be
– Allocated to machine or human
– Inputs required and locations of input sources
– Outputs and locations of users of the output

Jan. 2002
18
 Distributed Systems Architecture

Firemain Example - Decision Logical Architecture

• Ship Level Control Decisions
– Provide information to the human supervisor to isolate ruptures and maintain vital services
– Provide information to the human supervisor about the operational status of firemain
components
– Start other pump if operating pump is in rupture segment

• System Level Control Decisions

– Isolate ruptures by closing valves closest to the rupture
• Redundant with device level rupture isolation
• Flow balance logic
– Inputs required from multiple valves
– Logic is different from device level to minimize common mode failure
– Assess material condition and readiness of firemain
• Inputs required from multiple devices
• Device Level Control Decisions
– Isolate ruptures by closing valves closest to the rupture
• Rupture path logic
– Inputs and actuator are at the valve
Jan. 2002
19
Distributed Systems Architecture

Firemain Example - Candidate HW Architectures

• Device Level Processors
– Rupture path device level logic - YES
• Maximum survivability
• Excellent graceful degradation
• Simple
• Minimum impact on data communications
• Acceptable cost
– Flow balance system level logic - NO
• Survivability not enhanced beyond rupture path device level logic
• Logic to achieve graceful degradation likely to become unmanageably
complex
• Increases data communications load
– Provide information to the human supervisor to isolate ruptures and about
firemain operability - NO
• HCI is not applicable to device level processor

Jan. 2002
20
Distributed Systems Architecture

Firemain Example - Candidate HW Architectures

• System Level Processor(s)
– Rupture path device level logic - NO
• Execute at device level
– Flow balance system level logic - CANDIDATE
• May enhance survivability if:
– Separated from ship level processor
– Ship level processor executes redundant logic
– Dynamic application reallocation is provided
• Some increase in data communications load
• Increased cost for: additional computer, associated communications, more complex software
for dynamic application reallocation
• Additional redundant computers may increase survivability
– Start other pump if operating pump is in rupture segment - CANDIDATE
• May be a candidate for system processor if automated logic can be developed
– Provide information to the human supervisor to isolate ruptures and about firemain operability -
NO
• HCI is not applicable to system level processor
• Automated assessment of firemain operability could be done at a system level processor if applicable logic
is developed

Jan. 2002
21
Distributed Systems Architecture

Firemain Example - Candidate HW Architectures

• Ship Level Processor
– Rupture path device level logic - NO
• Execute at device level
– Flow balance system level logic - CANDIDATE
• May enhance survivability if:
– Ship level processor is redundant and separated from system level computer
• Some increase in data communications load
– Start other pump if operating pump is in rupture segment - CANDIDATE
• Without logic for automation, a human decision is needed - HCI is at the ship level
processor
– Provide information to the human supervisor to isolate ruptures and about
firemain operability - YES
• HCI is at ship level processor

Jan. 2002
22
Distributed Systems Architecture

Firemain Example - HW Architecture Decision

• Execute system level logic in
redundant, separated computer
OR • Execute system level logic in ship
level computer
– More survivable – Less survivable
– Less processor usage – Greater processor usage
– More complex – Less complex
– Higher data communications load – Lower data communications load
– More hardware cost – Less hardware cost
– Higher development cost – Lower development cost
– More equipment to maintain – Less equipment to maintain

• Could eliminate redundant system level logic for rupture isolation

– No automatic back-up for device level control
– Minimum cost and complexity
• Quantitative analyses could be performed to assess options
• Communications have a significant influence on survivability

Jan. 2002
23
Distributed Systems Architecture

Firemain Example - Software Architecture

Firemain Device Level Software Modules
Rupture Isolation - Rupture Path Logic

Firemain System Level Software

Module
Redundant Rupture Isolation - Flow Balance Logic
Firemain Readiness

Firemain Ship Level Software Module

Display Information To Isolate Ruptures & Maintain Vital Services
Display Information To Enable Assessment Of Firemain Operability
Remote Control Of Valves & Pumps

Jan. 2002
24
Distributed Systems Architecture

Further Development
• Extend to compartment-zone structured architecture
– Fire detection
– Access closure status
– Watermist actuation
• Investigate integration of device-system structured
architecture and compartment-zone structured architecture

Jan. 2002
25