Thinking systemically about complex

systems and decision making

Patrick T. Hester, Ph.D.

NASA Engineering and Safety

Center (NESC)
March 7, 2014
Outline
 Introduction
 Systems Errors
 The Systems Age
 Current State in Systems Engineering
 Systemic Thinking
 Decision Analysis
 A Way Ahead

2
The omnipotent nature of problems
 Think about a problem that keeps you up at night, one that
really eats at you
 Too easy, right?
 You’ve got countless, I’m sure
 Work/home/life, etc. all present numerous problems at any
given time

 So, why haven’t you solved it? What makes it hard?

3
What makes problems hard? (1/2)

 Significant uncertainty
Both“known unknowns” and “unknown
unknowns”
 Competition
Between individuals, objectives, resources
 Unpredictability
They exhibit emergent behavior

4
What makes problems hard? (2/2)

 Potential for loss

 Conjures up images of the road not taken
 Bias for action
 We are an instant gratification society
 The right now answer vs. the right answer

 Humans-in-the-loop
 Designing a mechanical system is straightforward,
although still complicated
 Designing the same system while accounting for
ergonomics, fatigue, and operator error prevention is
exponentially more complex

5
What should we do?
 Avoid all problems exhibiting one or more of these criteria
 Boring and unrealistic
 Our suggestion is adopting a novel way to reason about these
persistent, complex problems
 As engineers, we tend to think we have all the answers
 In reality, truly complex problems require a discipline-agnostic
approach
 i.e., a purely systems engineering approach may be good for an engineering problem,
but the real, interesting problems aren’t engineering problems only
 They require us to embrace concepts of many disciplines that may be advantageous
to our effort
 Simply, they require us to think systemically about our problem
 You should think and reason about complex problems using insight
from engineering, psychology, mathematics, biology, management,
and many other fields

6
Outline
 Introduction
 Systems Errors
 The Systems Age
 Current State in Systems Engineering
 Systemic Thinking
 Decision Analysis
 A Way Ahead

7
Our postulation
 Most complex problems can be characterized as having:
(1) significant uncertainty, (2) competition, (3)
unpredictability, (4) potential for loss, (5) bias for action
among stakeholders, and (6) humans-in-the-loop
 The way in which a problem is thought about, acted on,
and observed is a major determinant of the effect of these
factors.
 Independent of the construct or rigor used to address a
complex problem is the opportunity to commit a number of
errors.
 There is no agreed-upon taxonomy of these errors; thus,
we’ve developed one.
8
Error typology

Error Definition Issue

Type III (γ) Solving the wrong problem precisely. Wrong Problem
Type IV (δ) Inappropriate action is taken to resolve a problem as Wrong Action
the result of a correct analysis.
Type V (ε) Failure to act when the results of analysis indicate Inaction
action is required.
Type I (α) Rejecting the null-hypothesis when the null-hypothesis False Positive
is true.
Type II (β) Failing to reject the null-hypothesis when the null- False Negative
hypothesis is false.
Type VI (q) Inferring causation when only correlation exists. Unsubstantiated
Inference
Type VII (ζ) An error that results from a combination of the other System of Errors
five error types, often resulting in a more complex
problem than initially encountered.

Adapted from Adams and Hester (2013)

9
Error tree and probability

Problem to Solve

No Type III Error Thinking

Type III Error (P(g))
(1-P(g))

No Type IV or No Type IV or
Type IV Error Type IV Error Acting
Type V Error (P(e)) Type V Error Type V Error (P(e)) Type V Error
(P(d)) (P(d))
(1-[P(d)+P(e)]) (1-[P(d)+P(e)])

Type I Error (P(a)) Type I Error (P(a)) Type I Error (P(a)) Type I Error (P(a)) Type I Error (P(a)) Type I Error (P(a))

Type II Error (P(b)) Type II Error (P(b)) Type II Error (P(b)) Type II Error (P(b)) Type II Error (P(b)) Type II Error (P(b))

Observing

Type VI Error Type VI Error Type VI Error Type VI Error Type VI Error Type VI Error
(P(q)) (P(q)) (P(q)) (P(q)) (P(q)) (P(q))

No Type I, Type II, No Type I, Type II, No Type I, Type II, No Type I, Type II, No Type I, Type II, No Type I, Type II,
or Type VI Error or Type VI Error or Type VI Error or Type VI Error or Type VI Error or Type VI Error
(1- (1- (1- (1- (1- (1-
[P(a)+P(b)+P(q)]) [P(a)+P(b)+P(q)]) [P(a)+P(b)+P(q)]) [P(a)+P(b)+P(q)]) [P(a)+P(b)+P(q)]) [P(a)+P(b)+P(q)])

10
Adapted from Adams and Hester (2013)
Outline
 Introduction
 Systems Errors
 The Systems Age
 Current State in Systems Engineering
 Systemic Thinking
 Decision Analysis
 A Way Ahead

11
Historical background for complex
problems (1/2)
 Complexity of ill-structured, wicked, or messy problems has led
to a need for more sophisticated methods
 Problem solvers have been approaching complex problems
using usually a predominantly technical perspective since the
advent of large-scale telecommunications systems in the United
States during the 1930s
 Studies of these systems were interrupted by WWII, when many
scientists and engineers worked with the US military to help
solve complex logistical and strategic bombing problems
 Many of these efforts made significant contributions to the
philosophy and techniques of what was then called Operations
Research

12
Historical background for complex
problems (2/2)
 At the same time, the need for many novel types of electronic gear for
airborne use gave rise to a wide variety of component devices,
popularly known as black boxes
 “These were ingenious devices, but their application in terms of the entire system of
which they were merely parts was a matter of improvisation” (Engstrom, 1957, p. 113)
 Those working on these systems were required to think about the
system of which they were a part
 After WWII, many companies (notably the RAND Corporation, the Bell
Telephone Laboratories, and RCA) hired many of these scientists and
engineers to provide services to both the government and the
telecommunications industry
 Much early work in modern telecommunications system development
spurned systems approaches (Goode & Machol, 1957; Hall, 1962)
 In many ways, this was the beginning of the systems age

13
Systems and complexity
Russell Ackoff (1974) used the terms machine-age and
systems-age to refer to eras that were concerned with two
different types of systems. The machine-age was concerned
with simple systems, and the systems-age was concerned
with complex systems.

Characteristic Machine Age Systems Age

Simple Systems Complex Systems
Boundary Closed Open
Elements Passive parts Purposeful parts
Observable Fully Partially
Method of Scientific method of Cannot use
Understanding reductionism reductionism

14
The Machine Age and the Systems Age (1/2)
 Ackoff (1979) coined the concept of a mess and messes:
Because messes are systems of problems, the sum of the optimal
solutions to each component problem taken separately is not an
optimal solution to the mess. The behavior of the mess depends
more on how the solutions to its parts interact than on how they
interact independently of each other. But the unit in OR is a problem,
not a mess. Managers do not solve problems, they manage messes.
(p. 100)
 Presence of human activity necessitated more than a
purely technical perspective (Ackoff, 1979)
 Humans must be accounted for (and cannot be using a
machine age paradigm)

15
The Machine Age and the Systems Age (2/2)
 Burrell & Morgan (1979) support Ackoff’s assessment,
stating:
Mechanical models of social systems, therefore, tend to be
characterized by a number of theoretical considerations and are thus
of very limited value as methods of analysis in situations where the
environment of the subject is of any real significance. (p. 61)
 In short, the methods and techniques of traditional
operations research are “ . . . mathematically sophisticated
but contextually naïve and value free” (Hughes & Hughes,
2000, p. 10)

16
Messes at the intersection between hard and soft
systems

Soft
Perspective
Hard
(organizational,
Perspective
Messes managerial,
(technical
policy, political,
factors)
and human
factors)

Adapted from Hester and Adams (2014)

17
System type versus system treatment

Complex
Mismatch MATCH
System is

Simple MATCH Mismatch

Simple Complex

System Treated as

18
Outline
 Introduction
 Systems Errors
 The Systems Age
 Current State in Systems Engineering
 Systemic Thinking
 Decision Analysis
 A Way Ahead

19
Current state in systems engineering (1/2)
 Systems engineering has moved toward a more
processed-focused, acquisition-based discipline; it
became procedularized problem solving
 Popular current SE texts include Blanchard and Fabrycky;
Kossiakoff, Sweet, Seymour and Biemer; and Sage and
Armstrong
 Each of these texts expends substantial intellectual
resources discussing the process of systems engineering
 Indeed, SE had lost its roots; it became systematic
engineering, where systematic connotes the methodical,
process-based nature of standards for systems
engineering

20
Current state in systems engineering (2/2)
 Practiced by many organizations such as the DoD and
NASA
 SE, as currently practiced, is by and large the practical
application of procedural problem solving (most
traditionally problems concerning acquisition)
 Their thought process may be described as systematic
thinking
 Of course, many other systems approaches exist

21
 Systems-based approaches to complex problem
solving (1/2)
Systems Major Theme Primary Author(s)
Approach
Viable System Diagnosis of structural system functions, relationships, and (Beer, 1979, 1981, 1985)
Model communications channels necessary for any system to maintain
existence.
Sociotechnical Work system analysis and redesign based on joint optimization of the (Cherns, 1976; Pasmore, 1988;
Systems social and technical subsystems for performing work. Taylor & Felten, 1993)
Systems Structured formulation, analysis and interpretation of the technical, (Blanchard & Fabrycky, 2011;
Engineering human, and organizational aspects of complex systems to address Sage, 1992)
needs or resolve problems subject to cost, schedule, and operational
performance constraints.
System Computer modeling and simulation approach to understand the (Forrester, 1961, 1969, 1971;
Dynamics relationships and underlying behavior of complex systems. Maani & Cavana, 2000)
Soft Systems A process of inquiry focused on formulation of ill-structured problems (Checkland, 1993)
Methodology appreciative of multiple perspectives.
Total Systems A system problem solving approach based on creative thinking, (Flood & Jackson, 1991)
Intervention appropriate method selection, and implementation of method based
change proposals to resolve complex issues.
Gibson's Systems Provides six iterative phases to study complex systems problems, (Gibson, Scherer, & Gibson,
Analysis including System Goals, Ranking Criteria, Alternative Development, 2007)
Methodology Alternative Ranking, Iteration, and Action.
22
adapted from Keating (2012, p. 212)
Systems-based approaches to complex problem
solving (2/2)

 These methodologies are certainly successful, but

they have not been universally adopted
 Many of them are focused on systematic
approaches to gaining understanding
 Not appropriate for systems age messes
 A new paradigm of systemic thinking,
conceptually founded in systems theory, is
necessary
 New paradigm must be discipline-agnostic and
theoretically-driven

23
Historical Roots of Systems Theory
Stream of Thought Major Contributor(s) with Selected References
1. General Systems Bertalanffy (1949, 1950, 1968), Boulding (1956)
Theory
2. Living Systems Theory Miller (1978)
3. Mathematical Systems Mesarovic (1967), Wymore (1967), Klir (1968)
Theory
4. Cybernetics Rosenblueth, Wiener & Bigelow (1943), Wiener
(1948), Ashby (1947a, 1952, 1956), Forrester
(1961, 1969, 1971)
5. Social Systems Theory Parsons (1970, 1979, 1991), Buckley (1967, 1998),
Luhmann (1995, 2012)
6. Philosophical Systems Laszlo (1972, 1973, 1996), Bunge (1979, 1997,
Theory 1999, 2004)
These six systems theory streams of thought do not provide a generally accepted
canon of general theory that applies to all systems (i.e., is discipline-agnostic).

Table adapted from Adams, Hester, & Bradley (2013)

24
Our View of Systems Theory

Figure adapted from Adams, Hester, et al. (2014)

25
Outline
 Introduction
 Systems Errors
 The Systems Age
 Current State in Systems Engineering
 Systemic Thinking
 Decision Analysis
 A Way Ahead

26
What is systemic thinking?
 As a term, it has been gaining traction in recent literature
(e.g., Boardman & Sauser, 2013; Hester & Adams, 2013;
Hester & Adams, 2014; Midgley, 2012; Mingers, 2010)
 Term has been used without specificity or universality
 Goal is to articulate a unique perspective on systemic
thinking to differentiate it from traditional systems
approaches
 Demonstrate utility in helping individuals to increase their
understanding about problems and messes of any size,
complexity, or discipline
 Founded on systems theory

27
 Systematic vs. Systemic Thinking
Systematic Thinking Systemic Thinking
Age Machine Systems
Unit of Analysis Problem Mess (System of problems)
Stopping
Optimization Satisficing
Criteria
Goal Problem Solution Increased Understanding
Underlying Constructivism and
Reductionism
Philosophy Reductionism
Epistemology Analysis Synthesis and Analysis
Multidisciplinary
Discipline Scope Transdisciplinary
and Interdisciplinary
Approach Prescriptive Exploratory
adapted from Hester and Adams (2013)

28
A methodology for systemic thinking (1/3)
 Key is consideration of the “5 W's and How?”
 Who is relevant to understanding our mess?
 Stakeholders
 What are we trying to achieve in understanding our mess further?
 Outputs, outcomes
 Why are we interested in this mess?
 Motivation
 Where does our situation reside?
 Context and boundaries
 How do we achieve improved understanding of our mess?
 Mechanisms for achieving understanding
 When do we want to have increased mess understanding by?
 Stability and maturity

29
A methodology for systemic thinking (2/3)

Who? What?

How? Mess Why?

When? Where?

Adapted from Hester and Adams (2014)

30
A methodology for systemic thinking (3/3)

Adapted from Hester and Adams (2014)

31
Outline
 Introduction
 Systems Errors
 The Systems Age
 Current State in Systems Engineering
 Systemic Thinking
 Decision Analysis
 A Way Ahead

32
Decision Analysis

Decision analysis directly supports the what

and how of systemic thinking (among others):
 What are we trying to achieve?
 How are we going to increase our understanding?

33
What is decision making?

A process where:
1) There is more than one possibility (a choice)
2) The decision maker can form expectations concerning
future events and outcomes associated with each choice
3) Consequences contingent on the choice can be
evaluated on a subjective scale with respect to the
decision maker’s values and goals

34
Sub-disciplines

 Decision Theory—You vs. nature

 Normative
(Prescriptive) Decision Theory: How humans
should make decisions
• Expected utility theory
 Descriptive Decision Theory: How humans actually make
decisions
• Prospect theory, biases and heuristics

 Game Theory—You vs. others

 Howhumans make decisions in time of competition and
cooperation

35
Why does decision making matter?
 As practitioners in engineering (and in life) we are
forced to make decisions every day
 So, we must know how to make these decisions
and how to differentiate between alternatives
 In some cases, we must be able to defend our
choices

36
Why is rigorous decision making important?
 If we make the right decision, it’s not
 Atleast, to a point
 No one asks questions (unless they want to repeat our
successes)
 If we make the wrong decision, there are huge
consequences
 We must be able to answer for our decisions
• Hyatt Regency Walkway collapse
• Ford Pinto recall
• Tacoma Narrows Bridge
• Etc….

37
So what makes a good decision?

 If we reflect back on the circumstances under

which we made the decision a day, week, month,
etc. later, would we make the same decision?
 If so, it was a good decision.
 If not, it was not.

 Remember, this is independent of what you now

know
 Not fair to say “I wouldn’t have bought that lottery ticket”
after you know the numbers didn’t win
• After all, who wouldn’t say this?

38
Methods for decision making
Utility
Theory
Keeney, 1974

Classical Probability
Decision & Statistics
Making (CDM) Helton & Burmaster, 1996
Oberkampf & Helton, 2005

Von Neumann & Morgenstern, 1944

Savage, 1954 Expert
Judgment
Klir & Yuan, 1995; Zadeh, 1995
Dempster, 1967; Shafer, 1976
Dubois & Prade, 1988 How do we pick a
Naturalistic Situation method to increase
Decision Awareness
Making (NDM) Endsley, 1988, 1995a, 1995b, 2000 our understanding?
Klein, et al., 1993
Zsambok & Klein, 1997 Recognition
Flin et al., 1997 Primed Decision
Salas & Klein, 2001 Klein, et al, 1986
Brehmer, Lipshitz & Montgomery, 2005 NDM Warwick, et al, 2002
Schraagen, Militello, Omeron, & Lipshitz, 2008
Methods Santamaria & Warwick, 2009
Dynamic Model of
Situated Cognition
Organizational Miller & Shattuck, 2005
Decision Shattuck & Miller, 2006
Making (ODM)
Bounded
Simon, 1945
March & Simon, 1958 Rationality
Cyert & March, 1963
Hodgkinson & Starbuck, 2008 Simon, 1955, 1956
Kahneman, 2003a, 2003b

Judgment &
Prospect
Decision
Theory
Making (JDM)
Kahneman & Tversky, 1979

Edwards, 1954 Heuristics

Thrall, Coombs & Davis, 1954
Meehl, 1954 & Bias Adapted from Hester
Koehler & Harvey, 2004 Tverksy & Kahneman, 1971, 1974 and Adams (2014)

39
Choosing a method
 Recall our earlier graphic:

Complex
Mismatch MATCH
System is

Simple MATCH Mismatch

Simple Complex

System Treated as

 We need a more detailed framework for classifying

systems and matching up appropriate techniques for them:
1. Could choose based on popularity of usage
2. Can use a sensemaking framework such as Cynefin

40
Identify the appropriate technique (1/2)
Method Advantages Disadvantages Areas of Application
Multi-Attribute Takes uncertainty into account; can Needs a lot of input; preferences need to Economics, finance, actuarial, water
Utility Theory incorporate preferences. be precise. management, energy management,
(MAUT) agriculture

Analytic Hierarchy Easy to use; scalable; hierarchy Problems due to interdependence Performance-type problems, resource
Process (AHP) structure can easily adjust to fit between criteria and alternatives; can lead management, corporate policy and strategy,
many sized problems; not data to inconsistencies between judgment and public policy, political strategy, and
intensive. ranking criteria; rank reversal. planning.

Case-Based Not data intensive; requires little Sensitive to inconsistent data; requires Businesses, vehicle insurance, medicine, and
Reasoning (CBR) maintenance; can improve over many cases. engineering design.
time; can adapt to changes in
environment.
Data Envelopment Capable of handling multiple inputs Does not deal with imprecise data; Economics, medicine, utilities, road safety,
Analysis (DEA) and outputs; efficiency can be assumes that all input and output are agriculture, retail, and business problems.
analyzed and quantified. exactly known.

Fuzzy Set Theory Allows for imprecise input; takes Difficult to develop; can require Engineering, economics, environmental,
into account insufficient numerous simulations before use. social, medical, and management.
information.
Simple Multi- Simple; allows for any type of Procedure may not be convenient Environmental, construction, transportation
Attribute Rating weight assignment technique; less considering the framework. and logistics, military, manufacturing and
Technique effort by decision makers. assembly problems.
(SMART)
Adapted from Velasquez and Hester (2013)

41
Identify the appropriate technique (2/2)
Method Advantages Disadvantages Areas of Application
Goal Capable of handling large-scale It’s ability to weight coefficients; typically Production planning, scheduling, health
Programming problems; can produce infinite needs to be used in combination with care, portfolio selection, distribution
(GP) alternatives. other MCDM methods to weight systems, energy planning, water reservoir
coefficients. management, scheduling, wildlife
management.
ELECTRE Takes uncertainty and vagueness Its process and outcome can be difficult Energy, economics, environmental, water
into account. to explain in layman’s terms; outranking management, and transportation problems.
causes the strengths and weaknesses of
the alternatives to not be directly
identified.
PROMETHEE Easy to use; does not require Does not provide a clear method by Environmental, hydrology, water
assumption that criteria are which to assign weights. management, business and finance,
proportionate. chemistry, logistics and transportation,
manufacturing and assembly, energy,
agriculture.
Simple Additive Ability to compensate among Estimates revealed do not always reflect Water management, business, and financial
Weighting (SAW) criteria; intuitive to decision makers; the real situation; result obtained may not management.
calculation is simple does not be logical.
require complex computer
programs.
Technique for Has a simple process; easy to use Its use of Euclidean Distance does not Supply chain management and logistics,
Order Preferences and program; the number of steps consider the correlation of attributes; engineering, manufacturing systems,
by Similarity to remains the same regardless of the difficult to weight and keep consistency business and marketing, environmental,
Ideal Solutions number of attributes. of judgment. human resources, and water resources
(TOPSIS) management.

Adapted from Velasquez and Hester (2013)

42
Cynefin domains

Complex Complicated

Disorder

Chaotic Simple

Adapted from Kurtz and Snowden (2003)

43
Outline
 Introduction
 Systems Errors
 The Systems Age
 Current State in Systems Engineering
 Systemic Thinking
 Decision Analysis
 A Way Ahead

44
A way ahead

Systems engineering needs additional

research which appreciates the complexity
of modern messes and provides guidance
for choosing a supporting decision analysis
method appropriately
We are working on an appropriate mapping
between the Cynefin sensemaking
framework and appropriate decision
techniques

45
Contact information

Patrick Hester
Old Dominion University
Department of Engineering Management and
Systems Engineering
Office: (757) 683-5205
Email: pthester@odu.edu

46
References (1/4)
 Ackoff, R. L. (1974). The systems revolution. Long Range Planning, 7, 2-20.
 Ackoff, R. L. (1979). The future of operational research Is past. Journal of Operational Research Society, 30(2), 93-
104.
 Adams, K.M. and Hester, P.T. (2013). Accounting for errors when using systems approaches. Procedia Computer
Science, 20, 318-324.
 Adams, K.M., Hester, P.T., and Bradley, J.M. (2013). A historical perspective of systems theory. In A. Krishnamurthy
and W.K.V. Chan (Eds.), Proceedings of the 2013 Industrial and Systems Engineering Research Conference (pp.
4102-4109). Norcross, GA: Institute of Industrial Engineers.
 Adams, K.M., Hester, P.T., Bradley, J.M., Meyers, T.J., and Keating, C.B. (2014). Systems theory as the foundation
for understanding systems. Systems Engineering, 17(1), 112-123.
 Ashby, W. R. (1947). Principles of the Self-Organizing Dynamic System. Journal of General Psychology, 37(1), 125-
128.
 Ashby, W. R. (1952). Design for a Brain London: Chapman & Hall, Ltd.
 Ashby, W. R. (1956). An Introduction to Cybernetics. London: Chapman & Hall, Ltd.
 Beer, S. (1979). The heart of the enterprise. New York: John Wiley and Sons.
 Beer, S. (1981). Brain of the Firm. Chichester, UK: John Wiley & Sons.
 Beer, S. (1985). Diagnosing the system for organizations. Oxford, UK: Oxford University Press.
 Bertalanffy, L. v. (1949). General Systems Theory. Biologia Generalis, 19(1), 114-129.
 Bertalanffy, L. v. (1950). An Outline of General Systems Theory. The British Journal for the Philosophy of Science,
1(2), 134-165.
 Bertalanffy, L. v. (1968). General System Theory: Foundations, Development, Applications (Revised ed.). New York:
George Braziller.
 Blanchard, B. S., & Fabrycky, W. J. (2011). Systems engineering and analysis, 5th Ed. Upper Saddle River, NJ:
Prentice Hall.

47
References (2/4)
 Boardman, J., & Sauser, B. (2013). Systemic thinking: Building maps for worlds of systems. New York: John Wiley &
Sons.
 Boulding, K. E. (1956). General Systems Theory – The Skeleton of Science. Management Science, 2(3), 197-208.
 Buckley, W. (1967). Sociology and Modern Systems Theory. Englewood Cliffs: Prentice-Hall.
 Buckley, W. (1998). Society - A Complex Adaptive System: Essays in Social Theory. Amsterdam: Overseas
Publishers Association.
 Bunge, M. (1979). A systems concept of society: Beyond individualism and holism. Theory and Decision, 10(1-4), 13-
30.
 Bunge, M. (1997). Mechanism and Explanation. Philosophy of the Social Sciences, 27(4), 410-465.
 Bunge, M. (1999). The Sociology-philosophy Connection. New Brunswick, NJ: Transaction Publishers.
 Bunge, M. (2004). How Does It Work?: The Search for Explanatory Mechanisms. Philosophy of the Social Sciences,
34(2), 182-210.
 Burrell, G., & Morgan, G. (1979). Sociological Paradigms and Organizational Analysis: Elements of the Sociology of
Corporate Life. London: Heinemann.
 Checkland, P. B. (1993). Systems Thinking, Systems Practice. New York John Wiley & Sons.
 Cherns, A. (1976). The Principles of Sociotechnical Design. Human Relations, 29(8), 783-792.
 Engstrom, E. W. (1957). System Engineering – A Growing Concept. Electrical Engineering, 76(2), 113-116.
 Flood, R. L., & Jackson, M. C. (1991). Creative problem solving: Total systems intervention. New York: John Wiley &
Sons.
 Forrester, J. W. (1961). Industrial Dynamics. Cambridge, MA: MIT Press.
 Forrester, J. W. (1969). Urban Dynamics. Cambridge, MA: MIT Press.
 Forrester, J. W. (1971). World Dynamics. Cambridge, MA: MIT Press.
 Gibson, J. E., Scherer, W. T., & Gibson, W. E. (2007). How to do systems analysis. New York: John Wiley & Sons.
 Goode, H., & Machol, R. (1957). Systems engineering: An introduction to the design of large-scale systems. New
York: McGraw-Hill Book Company.
48
References (3/4)
 Hall, A. D. (1962). A methodology for systems engineering. Princeton: D. Van Nostrand Company, Inc.
 Hester, P.T., and Adams, K. MacG. (2014). Systemic thinking: Fundamentals for understanding problems and
messes. Manuscript in preparation for publication by Springer-Verlag.
 Hester, P. T., & Adams, K. M. (2013). Thinking systemically about complex systems. Procedia Computer Science, 20,
312-317.
 Hughes, A. C., & Hughes, T. P. (Eds.). (2000). Systems, Experts, and Computers: The Systems Approach in
Management and Engineering, World War II and After. Cambridge, MA: MIT Press.
 Keating, C. (2012). Perspective 2 of the SoSE methodology: Designing the unique methodology. International Journal
of System of Systems Engineering, 2(2/3), 208-225.
 Klir, G. J. (1968). An approach to general systems theory. Princeton, NJ: Nostrand.
 Kurtz, C. F., & Snowden, D. J. (2003). The New Dynamics of Strategy: Sense-making in a Complex-Complicated
World. IBM Systems Journal, 42(3), p. 462-483.
 Laszlo, E. (1972). Introduction to Systems Philosophy: Toward a New Paradigm of Contemporary Thought. New York:
Harper Torchbooks.
 Laszlo, E. (1973). The Rise of General Theories in Contemporary Science. Journal for General Philosophy of
Science, 4(2), 335-344.
 Laszlo, E. (1996). The Systems View of the World: A Holistic Vision for Our Time. Creskill, NJ: Hampton Press.
 Luhmann, N. (1995). Social Systems (J. Bednarz & D. Beacker, Trans.). Stanford, CA: Stanford University Press.
 Luhmann, N. (2012). Theory of Society (R. Barrett, Trans. Vol. 1). Stanford, CA: Stanford University Press.
 Maani, K. E., & Cavana, R. Y. (2000). Systems thinking, systems dynamics: Understanding change and complexity.
Auckland, NZ: Prentice-Hall.
 Mesarovic, M. D. (1967). General Systems Theory and Its Mathematical Foundation. Paper presented at the IEEE
Systems Science and Cybernetics Conference.
 Midgley, G. (2012). Systemic intervention: Philosophy, methodology, and practice. New York: Springer.
 Miller, J. G. (1978). Living Systems. New York: McGraw Hill.

49
References (4/4)
 Mingers, J. (2010). Realising systems thinking: Knowledge and action in management science. New York: Springer.
 Parsons, T. (1970). On Building Social System Theory: A Personal History. Daedalus, 99(4), 826-881.
 Parsons, T. (1979). Concrete Systems and "Abstracted Systems" Contemporary Sociology, 8(5), 696-705.
 Parsons, T. (1991). The Social System (New ed.). London: Routledge.
 Pasmore, W. A. (1988). Designing effective organizations: The sociotechnical systems perspective. New York: John
Wiley & Sons.
 Rosenblueth, A., Wiener, N., & Bigelow, J. (1943). Behavior, Purpose and Telelogy. Philosophy of Science, 10(1), 18-
24.
 Sage, A. P. (1992). Systems engineering. New York: John Wiley & Sons.
 Taylor, J. C., & Felten, D. F. (1993). Performance by design: Sociotechnical systems in North America. Englewood
Cliffs, NJ: Prentice-Hall.
 Velasquez, M., and Hester, P.T. (2013). An analysis of multi-criteria decision making methods. International Journal
of Operations Research, 10(2), 56-66.
 Wiener, N. (1948). Cybernetics: Or Control and Communication in the Animal and the Machine. Cambridge: MIT
Press.
 Wymore, A. W. (1967). A Mathematical Theory of Systems Engineering - The Elements. New York: John Wiley &
Sons.

50