Evaluating Complex System

Interventions

Evaluation 2009
Professional Development Workshop

Beverly Parsons and Meg Hargreaves

1
 What is a System?

 A group of interacting, interrelated,

and interdependent elements
forming a complex whole
 A configuration of parts connected
and joined together by a web of
relationships
 The whole is different from, and
greater than, the sum of its parts
2
Parts of an Elephant

3
 Systems Thinking
 A way of understanding reality that
emphasizes the relationships among
a system’s parts, rather than the
parts themselves.
 Concerned about interrelationships
among parts and their relationship to
a functioning whole
 Sees underlying patterns and
structures
4
Foundations of Systems Theory
 Cybernetics: system feedback,
information; differences (that make
a difference); human – machine
analogy; inclusion of the observer
and the observed in the system
 General systems theory: open
systems; system integrity; nested
system hierarchy, boundaries, webs,
emergence (sum greater than parts)
5
 Systems Theories

 Soft and critical systems: human

systems - multiple perspectives,
power issues, intractable problems
without simple solutions
 Systems dynamics: systems have
reinforcing and balancing feedback
loops, circularity, system archetypes,
mental models, unintended
consequences
6
 More Systems Theories

 Complexity theory: complex adaptive

systems; semi-independent, interacting
agents; self-organization; emergence;
nonlinearity; co-evolution; past is
irreversible; future is unpredictable
 Learning systems: the way that people
learn and the systems in which they learn

7
 System Boundaries

 Shows what is inside and outside of the

system
 Geographical (location)
 Organization (department, unit or function)

 Physical (money, material, information)

 Conceptual (goals, mission, purpose, rules)

 Intangibles (perceptions, awareness, models)

 Natural or man-made

8
 System Relationships
(Interconnections)
 Connections and exchanges among system
parts, parts and the whole, and the whole
and its environment
 Flows of information
 Flows of funding

 Client referrals

 Collaborative partnerships

 Family, community, and social networks

9
 System Perspectives

 Stakeholders’ worldviews and

purposes
 System agents who have different
perspectives may pursue different
purposes within a given situation
 Patterns of (mis)alignment of
purposes and processes within and
across system levels
10
 System Change

 System differences generate creative tension

or energy within a system
 Positive or negative, energy provides potential
for system change
 System change: shifts in patterns (similarities
and differences) of system relationships,
boundaries, focus, timing, events and
behaviors over time and space

11
 System Dynamics

 Random (unorganized)
 Organized (simple or complicated)
 Adaptive (organic, self-organizing)
 All three system dynamics can be
present in a complex situation

12
Random System Attributes

 Random activity – no pattern

 Unconnected collection of parts
 No cause-effect relationships
 Turbulence – no stability or equilibrium
 Answers are unknowable
 No purpose or direction – people react
blindly in a war zone or natural disaster

13
Random System: Hurricane
Katrina

14
Organized (Simple) System
Attributes
 Stable, static pattern
 Parts tightly connected machines
 Predictable cause-effect relationships
 System can be reduced to parts and
processes and replicated
 Directive leadership, designed change
 Answers are knowable, with recipes or
prescriptions for action

15
Single Organized System:
Ring-Around the Rosie

16
Simple Organized System:
Riding a Bicycle

17
Organized (Complicated) System
Attributes
 Dynamic patterns of feedback loops with
many interrelated parts within and across
subsystem levels
 Recursive, non-linear cause-effect
relationships; reinforcing and balancing
feedback loops maintain equilibrium
 Expert analysis can identify causal loops, deep
structural causes to actions

18
Insider Trading: A Tangled Web
of
Tips and Trades

19
Adaptive (Complex) System
Attributes
 Dynamical patterns – parts adapting, co-
evolving with each other and environment
 Parts are massively entangled and
interdependent; nested webs, networks
 Parts self-organize, learn, and change
 Equilibrium in flux, sensitive to initial
conditions; system change emerges through
interactions among parts

20
Ecological View of an
Elephant

21
Complex Interdependencies

22
Alignment of Context, Program,
and Evaluation Dynamics
 Context can be random, organized,
adaptive, or combination of dynamics
 Program design uses random, organized
(entity-based), or adaptive (paradigm-based)
or a combination of dynamics
 Evaluation design (content and process) can
be entity-focused (organized), paradigm-
focused (adaptive) or a combination of both

23
System Dynamics of
Family Nutrition

24
Simple Organized Dynamics of
Family Nutrition
 Context: hungry family
 Intervention: buy ingredients, bake a
cake, serve family at dinner
 Evaluation: quality of cake, family
satisfaction

25
Complicated Organized Dynamics
of Family Nutrition
 Context: hungry family with different tastes
and preferences
 Intervention: ask for family preferences,
create optional dishes, serve family multiple
dishes at dinner
 Evaluation: quality and variety of dinner
options, matching of dishes to tastes

26
Complex, Adaptive Dynamics of
Family Nutrition
 Context: hungry family with different tastes,
schedules, and cooking ability
 Intervention: Buy and store meal options,
make dishes for non-cooks, agree on dinner
schedule, adapt shopping patterns to use of
food and supplies
 Evaluation: trends, patterns of food use,
meals, family nutrition, overall health

27
System Dynamics of H1N1 Flu

28
Simple Organized Dynamics of
H1N1 Flu
 Context – everyone should be
protected through vaccination
 Program design – universal flu shot
clinics
 Evaluation design - How many clinics
were conducted, how many people
were vaccinated, how many people
contracted the H1N1 flu virus
29
Complicated Organized Dynamics
of H1N1 Flu
 Context – people are at different risk levels for
contracting the H1N1 flu
 Program design – allocate, administer flu shots by
risk level, triage patients by level of risk
 Evaluation design - What proportion of people
with high/medium/low risk receive the vaccine?
What proportion of people at each risk level
contract the H1N1 flu? How many deaths and
hospitalizations are avoided as result of shots?

30
Complex Adaptive Dynamics of
H1N1 Flu
 Context – Timing of two interacting epidemics
(H1N1 and seasonal flu) is ahead of current
vaccine production
 Program design – Multi-level intervention:
national media messages, provider triage by
risk, populations self-organize multiple
responses
 Evaluation design – What are changing
patterns of twin epidemics? How are
governments, providers, populations reacting
and interacting in response to situation?
Population health impacts?
31
 System Dynamics Discussion

 What are the situations’ boundaries, focus,

interconnections, perspectives, power, timing,
and dynamics?
 What are the risks of not understanding the
system attributes and dynamics?
 What are the benefits of understanding the
system attributes and dynamics?

32
System Dynamics of Child
Abuse Prevention – Home
Visiting

33
U.S. Child Abuse and
Neglect Trends

34
 Context, Program Design of
Child Abuse Prevention
 Context: Many programs exist but child abuse
and neglect rates are increasing
 Program design: AFC funding for 17 grants for
the adaptation, implementation, spread, and
sustainability of evidence-based home visiting
programs through infrastructure development
and system change

35
Evaluation Design of Evidence-
based Home Visiting Initiative
 Program evaluation – tracking of cross-site cost,
implementation, fidelity, and child and family
outcomes of 17 EBHV programs
 System evaluation – tracking of cross-site and
grantee-specific system infrastructure, theories
of action, measures of system change, partner
collaboration and network analysis; system unit
of analysis

36
 Open Space Technology:
System Dynamics Exercise
 What are the dynamics (i.e., the
nature and balance of types of
system dynamics) of the situation
as a whole?
 What are the system dynamics of
the intervention?
 What are the implications for the
evaluation design and process?

37
Three Dynamics of a Social
System and its Context

38
 Match of Evaluation Designs to
Dynamics
of Social Systems and Their Context
Exploratory
Design
Initiative Renewal
Design

Organic
Design

Predictive
Design

39
Complex Adaptive Systems and
Adaptive (Self-organizing)
Dynamics
1. Self-organizing/adaptive/organic
2. Sensitivity to initial conditions
3. Emergence
4. Macro pattern

40
Complex Adaptive Systems and
Adaptive (Self-organizing)
Dynamics (cont.)
5. Feedback
6. Co-evolution
7. Pattern formation and points of
influence

41
Implications for Evaluation and
Action
1. Small differences can create large
effects.
2. The past influences but does not
predict the future.
3. Many points of influence exist.
4. Boundaries, differences, and
relationships are levers of influence
toward a purpose.
42
Implications for Evaluation and
Action
5. Simple rules underlie patterns.
6. Pattern-based feedback and
actions are iterative.
7. Tensions are not resolved.
8. Patterns are outcomes.

43
Four Stages of Evaluation

Design
Evaluation

Shape Collect
Practice Data

Make
Meaning from
Data

44
45
 Example: LEAP

Learning through Engineering

Design and Practice

46
Example: LEAP Research Design
 Quasi-experimental design embedded in
curriculum development process
 Pre-post assessments of
 Content knowledge
 Perceptions of engineers at work
 Tinkering
 Self-efficacy
 Engineering notebooks
 Career behaviors survey

47
External Evaluation Design
The external evaluation focused on:
 Confirmation of effectiveness
 Scale-up
 Sustainability

48
 Conceptual Shifts
The fundamental conceptual shift in this project
was from:
 teacher-directed de-contextualized learning to
student-engaged project-based learning
 fixed skills and knowledge as learning outcomes
to the desired outcomes being that students are
actively engaged; develop the capacity to
explore and figure things out; and act like an
engineer.

49
Confirmation of Effectiveness
 Knowledge and skills related to project
topics and STEM concepts
 Enjoyment and pride in project work
 Development of teamwork,
collaboration and workplace skills
 Interest in STEM courses and pursuit of
STEM career and educational pathways

50
 Scale-Up

Scale-up involved two tracks:

 Greater use of the curriculum within
the school system
 Transfer of the curriculum to Boys
and Girls Clubs

51
 Sustainability
 Program sustainability
 Sustainability of learning of
participating students
 Sustainability of collaborations
 Sustainability of teaching capacity

52
 Program Sustainability
 Maintain relationships with the district
 Professional development for teachers
 Shape research related to the project
 Explore ways to continue project at 9th grade
 Track STEM course selection of project students
in high school
 Collaborate on additional community
dissemination and funding

53
Sustainability of Learning of
Students
 Classroom
 Extracurricular activities
 Career-related activities
 Focused attention through high school
 Continued involvement of university faculty
and students
 Continued contact with science center

54
 Sustainability of
Collaborations
 Use current collaborations to spur
others over time

55
Sustainability of Teaching
Capacity
 Build capacity through formal and
informal professional development
approaches
 Training for new teachers
 Coaching by master teachers

 Summer professional development

activities
 Technology enhanced training

56
 Example 2

Communities of Learning,
Inquiry, and Practice
(CLIPs)

(video at www.insites.org/clip)

57
 Example 3

Strengthening Families

58
 Strengthening Families
Protective Factors
 Parental resilience
 Social connections
 Concrete support in time of need
 Knowledge of parenting and child
development
 Social and emotional competence of
child
59
60
 World Café Exercise
 What data gathering and/or analysis tools
have you found helpful in gaining a deeper
understanding of complex systems or
interventions?
 What practices help you develop your
capacity to recognize patterns?
 Do certain practices seem more related to
finding surface patterns and others more
related to finding deep patterns?

61
 Contact Information

 Beverly Parsons, Ph.D.

bparsons@insites.org
(360) 638-1442

 Meg Hargreaves, Ph.D.

mhargreaves@mathematica-mpr.com
(617) 301-8994

62