Riccio, G. (2010). A Programmatic Inquiry into Outcomes Based Training & Education. In: Riccio, G., Diedrich, F.

, & Cortes, M. (Eds.). An

Initiative in Outcomes-Based Training and Education: Implications for an Integrated Approach to Values-Based Requirements (Prologue). Fort
Meade, MD: U.S. Army Asymmetric Warfare Group. [Cover art by Wordle.net represents word frequency in text.]

Riccio
Prologue:
A Programmatic View of the Inquiry into Outcomes-Based Training & Education
Gary Riccio
The Wexford Group International
Historicity of our Research on OBTE

This unique investigation resulted from an unusual confluence of events that brought key
personnel from the Asymmetric Warfare Group (AWG) together with key personnel in the
science team that ultimately was engaged by the AWG. In September 2007, Fred Diedrich and
I were involved in a project with the Fort Benning research unit of the Army Research Institute
for the Behavioral and Social Sciences (ARI). One task in that project was directed toward
identification of formative measures for instructors in the Basic Noncommissioned Officer
Course (BNCOC) at Fort Benning. In particular, we were focusing on the Small Arms
Proficiency Training Module (SAPT) of BNCOC. Scott Flanagan was invited to participate in the
measure development process known as COMPASS (described in Chapter 2). Scott was working
on Outcomes Based Training and Education (OBTE) with the AWG at that time.
The most important criteria imposed on the COMPASS process for SAPT were the Pentathlete
Characteristics (i.e., Warrior Leader, an Ambassador, a Critical and Creative Thinker, a Leader
Developer, and a Resource Manager). This introduced an exigency to map the measures of
observable instructional behavior to something more abstract, to something more like cultural
values. While the Pentathlete Characteristics in particular ceased to become a priority over the
course of the project, we consider this kind of mapping to be an important general source of
external validity for a set of instructional measures (Sidman, Riccio, Semmens, et al., 2009).
In a prior project, we addressed a set of values-based concepts, those embodied in Warrior Ethos
(Riccio, Sullivan, Klein, Salter, & Kinnison, 2004; see Appendix D). We found that it is possible
to identify relationships between abstract values and concrete behavior of Soldiers in an
operational context or training context. These relationships led to a deeper understanding of
Warrior Ethos in terms of scientifically traceable concepts and in terms of specific actionable
recommendations for the planning and execution of training. This is important because there is a
natural skepticism about the meaning of values-based terminology that changes from time to
time. In general, we suspect that persistence of a relatively small number of core values can be
identified amid such changes in terminology through their common connections to a scientifically
and philosophically meaningful foundation of enduring concepts (see Chapters 3, 4, and 5).
Given the original intent of the ARI project, it was natural to consider common and convergent
themes across projects in which we had first-hand exposure to various programs of instruction in
the institutional Army. Our involvement with such programs was through research, thus we were
more likely to be exposed to programs in which change was taking place or was being
considered. A significant convergence occurred during this project between the SAPT module of
BNCOC and an initiative of the AWG to introduce a different approach to Army training and
education (Outcomes-Based Training & Education [OBTE]) through its initial application to
marksmanship training (see Chapter 1). The AWG initiative in OBTE became important to
consider because our initial assessment of instruction in BNCOC, using the measures developed
for SAPT module, showed gaps between what BNCOC leadership expected instructors and
students to be doing and what was actually occurring in SAPT. At the same time, OBTE was

Asymmetric Warfare Group

Prologue

receiving increasing attention in several programs of instruction (see Chapters 1 and 11). Both
our team and BNCOC leadership began to inquire into the lessons learned by the AWG.
By December of 2007, BNCOC leadership began to inquire into the possibility of BNCOC
instructors attending the AWGs marksmanship course. The intent of this course is to familiarize
trainers with OBTE. At the same time, Scott Flanagan brought information back to the AWG
about the COMPASS process and its potential value in bringing scientific rigorousness to OBTE.
By January, LTC Michael Richardson (Baker Squadron Commander at the AWG) began to
inquire into the possibility of developing measures for OBTE by employing the COMPASS
process and the team that ARI was using to develop measures for BNCOC. After some
deliberation about intent and purpose, in April 2008, the team began a three-month effort with the
AWG to define and measure the practice of OBTE and to begin the development of theoretical
underpinnings for the approach. LTC Richardson monitored the effort with the assistance of CSM
Michael Cortes, with the approval of COL Robert Shaw, and with visibility to many key
personnel in the AWG. Scott Flanagan, Morgan Darwin and Blaise Cornell dEchert (CD) were
key informants about OBTE to the science team on behalf of the AWG. The essential activities
and findings of the initial phase of the investigation are described in Chapters 2, 6, 7, 8, 10, and
11.
At the end of October 2008, the science team was re-engaged for further study of OBTE. In
addition to data collection, COL Shaw wanted the team to produce a monograph that would
reveal the current and potential scholarly breadth and depth of OBTE. By this time, LTC
Richardson had transitioned to a new assignment outside of the AWG and was replaced by LTC
Richard Thewes. CSM Cortes became a critical adviser and technical monitor across this change
of command. CD and Morgan continued in their influential roles as informants about the history,
cultural implications, and ongoing exploration of OBTE in the Institutional Army. The essential
activities and findings of the second phase of the AWGs internal scientific inquiry are reported
in Chapters 9, 12, 13, and 14. The engagement of the science team by the AWG ended in June
2009 as planned, consistent with the AWGs Operations Order for OBTE. Chapters 3, 4, 5, 15,
and the Epilogue were written over several months following the end of the investigation. These
chapters reflect the continuing dialog among a variety of stakeholders about the vision, purpose,
decentralized adaptation, and programmatic implications of OBTE.
The science team was co-led by Fred Diedrich and me. Fred was responsible for all data
collection on the ground and for managing the associated personnel, the most extensive part of
the investigation (see e.g., Chapters 8, 9, and 13). I was responsible for development of the
grounded theory for OBTE (see e.g., Chapter 3, 4, and 5) through continuous interactions with the
progenitors of OBTE about the assumptions, history, and expectations of the approach as well as
about their ongoing dialog with stakeholders. Fred also was primarily responsible for ensuring
that observations of OBTE were as concrete and verifiable as possible by emphasize its
manifestations and effects in the behavior of instructors and students. I was primarily responsible
for conducting due diligence on the conscious experience of instructors and students in OBTE to
achieve a deeper level of understanding of the causes and effects of good instruction. We believe
that the balance between behavior and conscious experience is one of the most noteworthy
achievements in this investigation. It helped us identify and stay focused on behavior that is
meaningful and on conscious experience that is grounded in reality.
Amid the division of labor between Fred and me, there was frequent substantial communication
between us on all aspects of the project. The purpose and outcome of this association was a
reciprocal influence between the evolving theory of OBTE and the findings about the practice of
OBTE to move us systematically toward a theory of practice. In particular, our interactions with
Asymmetric Warfare Group

Riccio

progenitors of OBTE directly influenced the development of measures utilized in the

investigation (see Chapters 2, 7, and 12) as well as our understanding of the limits of the
investigation and associated gaps in validation of OBTE (see e.g., Chapter 11 and Epilogue). The
findings from the AWGs multifaceted investigation, in turn, influenced the concurrent dialectic
with OBTE progenitors and stakeholders by providing increasingly clear and relevant questions
centered on concrete issues and opportunities in the ongoing implementation of OBTE. The
current level of understanding of OBTE would not have been possible without the empirical
components of investigation and the theory development that it enabled.
The Approach and Lessons Learned from the Research
The research design is consistent with a state-of-the-art Service System Development (SSD)
process area of the Capability Maturity Model Integration for Services (CMMI-SVC) and also
reflects best practices in scientific support of social and programmatic decision-making (Chapters
1 and 11). Accordingly, the investigation addressed development and analysis of stakeholder
requirements (Specific Goal 1 of SSD), development of OBTE as a service system that meets
such requirements (Specific Goal 2 of SSD), and verification and validation of OBTE as a service
system (Specific Goal 3 of SSD). Specific practices associated with SG 3 include performance of
peer reviews, verification of selected service system components, and validation of the service
system. Peer review was addressed primarily in the context of surveys administered by the AWG.
Verification was conducted primarily through field-based observations in Initial Entry Training
(IET) at Fort Jackson, Fort Benning, and Fort Sill that employed measures specifically developed
to enable formative assessment of instructor and student behavior. The empirical approach to
validation focused on surveys of Soldiers who had been deployed to a combat environment after
being exposed to an early version of a field course in marksmanship that demonstrated
instructional principles of OBTE. Validation also was addressed through development of
grounded theory for OBTE based on established lines of scholarship in the behavioral and social
sciences.
Collectively, the evidence suggests considerable potential for OBTE in the Army, especially if it
becomes explicitly more directed toward staff and faculty development and quality assurance (see
Epilogue). It is noteworthy that our findings suggest that OBTE can motivate individuals to take
ownership of their own learning and development, and that it can increase self-efficacy for
teaching and developing others. The approach motivates a greater interest in instructor-student
interactions and how these interactions affect progress toward developmental outcomes while
satisfying course-specific learning objectives. While the findings show that there is both an
impact of OBTE and considerable room for improvement in Initial Entry Training, the measures
of OBTE provide actionable information about the gaps and shortfalls. The findings and
associated research have implications for all the factors that influence institutional learning
including student diversity and background, capacity for self development, instructional design
and development, instructor education, reward and recognition, individual assessment and
program evaluation, leadership in an institution of learning, leader development, and the relevant
science of measurement.
Given our approach to development of OBTE as an integrated instructional service system, it is
critical to understand what instructors should do and what they believe they should influence in
particular programs of instruction. Toward that end, we developed a set of measures for instructor
behavior that exemplifies OBTE and that can be applied to any program of instruction (Chapters
2, Appendix A). We also developed measures for student behavior that complement the instructor
measures and that help reveal the efficacy of instructor-student interactions (Chapters 7 and 12,
Appendix B). We thus defined OBTE as a verifiable and replicable instructional service system,
Asymmetric Warfare Group

Prologue

and we explored its links to established theories. We emphasized the utility of these measures in
formative assessments for instructors and instructional program evaluation. Moving forward, we
stress that definition and measurement of instructor-student interactions and verification of
instructional practices is critical to sustain or improve the quality of any instructional service
system and to transfer any new approach to particular programs of instruction (Chapters 1 and
11). In essence, definition and measurement of OBTE can help turn conventional wisdom into
conventional practice.
We believe that tacit knowledge about good training can be made concrete through programmatic
scientific inquiry. Of necessity, this also reveals connections with domains of training and
education outside the Army. Thus, there is an opportunity to realize the potential for improvement
in virtually everything that influences Army training and education (e.g., see Epilogue) by
adapting specific practices within and beyond Army training and education and by utilizing the
more general scientific underpinnings. The research cited in this monograph provides a
theoretical and empirical basis for shared understanding and continuous coordinated improvement
across multiple organizations and organizational levels in training and education (see Chapters 14
and 15). The most immediate result of this is the ability to develop and promulgate standards for
existing values-based requirements in Army doctrine. The longer-term result is that Army
training and education will be aligned more closely and more completely with Army doctrine and
the operational exigencies it represents.
Documentation of the Research
The purpose of the investigation was to define and measure OBTE to confirm that it satisfies its
intended requirements and that it will satisfy end-user expectations during actual implementation
of the approach. More specifically, we utilized multiple sources of evidence to understand the
implementation of OBTE and its effects on thought and behavior. The structure of the monograph
reflects these means and ends (Figure 1). As is evident in this structure, there was not a linear
progression through the chapters with respect to development of OBTE as an integrated
instructional service system. Consistent with spiral development, verification and validation both
informs and is informed by the ongoing refinement of stakeholder requirements and continuous
improvement of OBTE as a mature service system. There are reciprocal relationships among
these concurrent systems engineering activities. The AWGs intent is to develop OBTE into a
mature service system that can be transitioned to the institutional Army.
Some of the chapters will have straightforward utility to instructors and their chains of command
(e.g., Chapters 2, 12, and the Appendices); that is, they are actionable without undue
interpretation. It is not, however, a users manual for Soldiers and instructors. The primary
purpose of this document is to support programmatic decision-making (Chapters 1 and 11). In
this respect, the most important contribution of this document is that it reveals the depth and
breadth of evidence necessary to support decisions about instruction pursuant to requirements in
existing Army doctrine. This includes both methods of assessment and grounded theory that are
well established in the scientific literature. We also point to scholarship beyond science that may
be necessary to understand the inevitable interactions with the unknown that presumably are the
reason for emphasis on adaptability and values in preparing Soldiers for Full Spectrum
Operations. With grounding in the science that is relevant to values-based adaptability, and
recognition that science also has its limits in this regard, this monograph lays the foundation for
development of materials specifically for Soldiers and instructors (see e.g., the Epilogue).

Asymmetric Warfare Group

Riccio

Figure 1. Evolution of the investigation as reflected in the chapters of this monograph.

References
Bruny, T., Riccio, G., Sidman, J., Darowski, A., & Diedrich, F. (2006). Enhancing warrior ethos
in initial entry training. Proceedings of the 50th Annual Meeting of the Human Factors and
Ergonomics Society, San Francisco, CA.
Freeman, J., Jason, J., Aten, T., Diedrich, F., Cooke, N., Winner, J., Rowe. L., & Riccio, G.
(2008). Shared Interpretation of Commander's Intent (SICI). Final Report to the Army
Research Institute for the Behavior and Social Sciences, contract number W74V8H-06-C0004.
Riccio, G., dEchert, B.C., Lerario, M., Pound, D., Bruny, T., & Diedrich, F. (2006). Enhancing
Joint Task Force Cognitive Leadership Skills. Report to the Army Research Institute for the
Behavioral and Social Sciences, contract number Army contract no. W74V8H-06-P-0186.
Vienna, VA: The Wexford Group International.
Riccio, G., Lerario, M., Cornell dEchert, B., Pound, D., Bruny, T., & Diedrich, F. (2006).
Training a Joint and expeditionary mindset. Report to the Army Research Institute for the
Behavioral and Social Sciences, contract number W74V8H-06-P-0189. Vienna, VA: The
Wexford Group International.
Riccio, G., Sullivan, R., Klein, G., Salter, M., & Kinnison, H. (2004). Warrior ethos: Analysis of
the concept and initial development of applications. ARI Research Report 1827. Arlington,
VA: US Army Research Institute for the Behavioral and Social Sciences.
Sidman, J., Riccio, G., Semmens, R., Geyer, A., Dean, C., & Diedrich, F. (2009). Reshaping
Army institutional training: Current training. Final Report to the Army Research Institute for
the Behavior and Social Sciences, contract number W74V8H-04-D-0047 DO 0010.

Asymmetric Warfare Group

ix

Riccio, Diedrich, & Cortes (Eds.)

TABLE OF CONTENTS
.

page
Prologue: A Programmatic View of the Inquiry into Outcomes-Based Training & Education.......1
Historicity of our Research on OBTE ..........................................................................................1
The Approach and Lessons Learned from the Research..............................................................3
Documentation of the Research ...................................................................................................4
Section I. Development of Stakeholder Requirements for OBTE..............................................6
Chapter 1. Preparation for Full Spectrum Operations ......................................................................7
1.1 Requirements of Full Spectrum Operations ...........................................................................8
1.2 Outcomes-Based Training and Education (OBTE)..............................................................10
1.2.1 Exemplar of OBTE: Combat Applications Training Course........................................11
1.2.2 OBTE as a Multifaceted Instructional System .............................................................12
1.3 An Appraisal of Instruction with Respect to OBTE ............................................................13
1.3.1 A Systems Engineering Framework for Integration and Development of OBTE ........13
1.3.2 Preparation for Validation and Verification .................................................................14
1.4 References ............................................................................................................................17
Chapter 2. Formative Measures for Instructors ..............................................................................20
2.1 Development of Formative Measures ..................................................................................20
2.1.1 The COMPASS Methodology ......................................................................................20
2.1.2 Development of Measures for OBTE ...........................................................................21
2.2 Description of Formative Measures .....................................................................................21
2.2.1 Results of the COMPASS Process................................................................................21
2.2.2 Elaboration on the Description of Measures.................................................................23
2.3 OBTE Performance Measures: Planning for Training.........................................................23
2.3.1 Define Outcomes ..........................................................................................................23
2.3.2 Create a Positive Learning Environment ......................................................................25
2.3.3 Create the Parameters of Learning................................................................................27
2.4 OBTE Performance Indicators: Training Execution............................................................28
2.4.1 Communicate the Parameters of Learning....................................................................28
2.4.2 Training Emphasizes Broad Combat or Mission Success ............................................29
2.4.3 Customize Instruction When Possible Based on Constraints/Conditions ....................31
2.4.4 Facilitates Learning of Concepts ..................................................................................32
2.4.5 Creates a positive learning environment.......................................................................34
2.4.6 Instructors Utilize Measures of Effectiveness & Self-Evaluation ................................36
2.4.7 Uses scenarios to facilitate learning..............................................................................38
2.4.8 Instructors exhibit intangible attributes in own actions ................................................40
2.4.9 Hotwashes and Mini-AAR............................................................................................42
2.5 Uses of the Measures ...........................................................................................................43
2.5.1 Formative Measures for Instructors ..............................................................................44
2.5.2 Quality Assurance and Instructor Education ................................................................44
2.5.3 Continuous Improvement of Assessments....................................................................45
2.5.4 Program Evaluation and Organizational Change..........................................................46
2.6 References ............................................................................................................................46

Asymmetric Warfare Group

Table of Contents

Chapter 3. Principles and Practices of Outcomes Based Training & Education............................50

3.1 Multifaceted Inquiry.............................................................................................................50
3.1.1 Interaction with Progenitors of OBTE..........................................................................51
3.1.2 AWG Documents on OBTE .........................................................................................52
3.1.3 Collaborative Reflection on Participant Observation in CATC ...................................52
3.1.4 Interaction with Stakeholders .......................................................................................53
3.2 Essential Characteristics of OBTE.......................................................................................53
3.2.1 The Meaning of Developmental is a Critical Difference..............................................53
3.2.2 The Definition of Outcomes is a Critical Difference....................................................56
3.2.3 The Emphasis on Values and Causally Potent Intangibles is a Critical Difference .....58
3.2.4 The Meaning of Experience is a Critical Difference ....................................................61
3.2.5 The Emphasis on Instructor-Student Interactions is a Critical Difference ...................62
3.2.6 The Emphasis on Learning to Learn is a Critical Difference .......................................63
3.2.7 The Emphasis on Collaborative Design and Development is a Critical Difference.....65
3.3 Toward a Grounded Theory for OBTE ................................................................................66
3.3.1 Need for an Integrated Interdisciplinary Framework ...................................................66
3.3.2 Formative Measures of Instructor Behavior as Evolving Best Practices of OBTE......67
3.4 Emerging Best Practices in OBTE for a Community-Centered Environment.....................68
3.4.1 Leadership and Enculturation of Soldiers.....................................................................68
3.4.2 Robust and Adaptable Plan...........................................................................................70
3.4.3 Instructors as Role Models ...........................................................................................70
3.4.4 Collaborative Identification of Outcomes and Measures .............................................71
3.5 Emerging Best Practices in OBTE for a Knowledge-Centered Environment .....................71
3.5.1 Integrated Understanding of Basic Soldier Skills in Full Spectrum Operations ..........72
3.5.2 Task Relevance of Planned Instructional Events..........................................................72
3.5.3 Reveal Operational Relevance of Training...................................................................73
3.5.4 Incorporate Stress into Instructional Events .................................................................73
3.5.5 Identify General Lessons Learned and Extrapolate to New Situations ........................74
3.6 Emerging Best Practices in OBTE for an Assessment-Centered Environment ...................74
3.6.1 Collaborative Reflection and Problem Solving ............................................................75
3.6.2 Communication.............................................................................................................75
3.6.3 Nature and Extent of Guidance.....................................................................................76
3.6.4 Establish a Pervasive Mindset of Collaborative Reflection..........................................76
3.7 Emerging Best Practices in OBTE for a Learner-Centered Environment ...........................77
3.7.1 Soldier Motivation and Development of Intangibles....................................................77
3.7.2 Plan for Development of the Individual .......................................................................78
3.7.3 Get Students to Take Ownership ..................................................................................78
3.7.4 Collaborative Reflection as a Means to Develop Self Efficacy....................................79
3.8 References ............................................................................................................................79
Chapter 4. Grounded Theory for Values-Based Training & Education .........................................86
4.1 Exploration of Holistic and Functionalistic Underpinnings for OBTE ...............................86
4.1.1 Fundamental Units of Analysis.....................................................................................87
4.1.2 Nested Time Scales and Adaptability ...........................................................................88
4.1.3 Adaptability and Ambiguity .........................................................................................90
4.1.4 Mechanistic Analogies and Predominant Experimental Paradigms .............................92
4.2 Three Pillars for the Scientific Foundation of OBTE ..........................................................93
4.2.1 Ecological Psychology..................................................................................................93
4.2.2 Self-Efficacy Theory.....................................................................................................97
4.2.3 Positive psychology ......................................................................................................98
4.3 A More Integrated Scientific Infrastructure .......................................................................101
Asymmetric Warfare Group

xi

Riccio, Diedrich, & Cortes (Eds.)

4.3.1 Self Determination Theory .........................................................................................101
4.3.2 Situated Learning Theory ...........................................................................................103
4.3.3 Existential Psychology................................................................................................105
4.4 Building on the Scientific Infrastructure for OBTE...........................................................109
4.4.1 Triadic Frameworks ....................................................................................................109
4.4.2 Further Development ..................................................................................................112
4.5 References ..........................................................................................................................112

Chapter 5. Passion and Reason in Values-Based Learning & Development ...............................118

5.1 The Nested Self ..................................................................................................................118
5.1.1 An Alternative to Individual versus Collective ..........................................................118
5.1.2 Cognition and Reality .................................................................................................119
5.2 Conscious Experience and the Dynamics of Thinking ......................................................122
5.3 Emotion, Information, and Engagement ............................................................................125
5.3.1 Ecological Perspective on Emotion ............................................................................125
5.3.2 Emotion as Engagement .............................................................................................126
5.3.3 Implications for Training and Education ....................................................................129
5.4 Emotion, Decision-Making, and Inter-Temporal Choice...................................................129
5.4.1 Toward a More Integrated Theory..............................................................................129
5.4.2 Emotion and Decision-Making ...................................................................................130
5.4.3 Emotion and Nested Time Scales ...............................................................................131
5.4.4 Neuroeconomics and Inter-Temporal Reasoning .......................................................132
5.5.5 Inter-Temporal Reasoning and Adaptive Dynamical Systems...................................133
5.5 Beyond Science ..................................................................................................................134
5.5.1 Existentialism..............................................................................................................134
5.5.2 The Soldier-Scholar as an Emergent Property of a Collective Pursuit.......................135
5.6 References ..........................................................................................................................137
Section II. Verification and Validation of OBTE as a Service System ..................................142
Chapter 6. Initial Impressions of Participation in CATC .............................................................143
6.1 Methods..............................................................................................................................143
6.1.1 Participants..................................................................................................................143
6.1.2 Procedure ....................................................................................................................143
6.1.3 Analyses......................................................................................................................144
6.2 Results ................................................................................................................................144
6.3 Implications for Service System Development: Peer Review ...........................................146
6.4 References ..........................................................................................................................147
Chapter 7. Local Development of Measures of Effectiveness .....................................................149
7.1 What do Instructors Believe Soldiers Should Learn in Initial Entry Training? .................149
7.2 Measure Development Process ..........................................................................................150
7.3 What do OBTE-Trained DS Believe is Important to Assess in BRM/ARM? ...................151
7.4 Implications........................................................................................................................156
7.5 Conclusions ........................................................................................................................158
7.6 References ..........................................................................................................................159

Asymmetric Warfare Group

Table of Contents

xii

Chapter 8. Observations of Behavior and Communication in Rifle Marksmanship Training .....160

8.1 Methods..............................................................................................................................160
8.1.1 Participants..................................................................................................................160
8.1.2 Procedure ....................................................................................................................160
8.1.3 Analyses......................................................................................................................161
8.2 Results ................................................................................................................................163
8.2.1 Behavior of DS ...........................................................................................................163
8.2.2 Behavior and Performance of Privates .......................................................................165
8.2.3 Patterns of Communication ........................................................................................168
8.2.4 Potential Influence of Instructor Behavior on Performance of Privates .....................170
8.3 Implications for Service System Development..................................................................171
8.3.1 Verification of OBTE .................................................................................................171
8.3.2 Validation of OBTE....................................................................................................172
8.4 References ..........................................................................................................................173
Chapter 9. Impact on Rifle Marksmanship Training....................................................................174
9.1 Behavioral Data Collection During Basic Rifle Marksmanship ........................................174
9.1.1 Method ........................................................................................................................174
9.1.2 Assessment..................................................................................................................175
9.1.3 Results An Overview ...............................................................................................177
9.1.4 Evidence for Influence of OBTE ................................................................................178
9.1.5 Behavior of Drill Sergeants after Exposure to OBTE ................................................180
9.1.6 Behavior of Privates....................................................................................................182
9.1.7 Patterns of Communication ........................................................................................186
9.1.8 Summary .....................................................................................................................186
9.2 Attitudes Toward an OBTE in Basic Training...................................................................187
9.2.1 Method ........................................................................................................................187
9.2.2 Results.........................................................................................................................187
9.4 References ..........................................................................................................................191
Chapter 10. Influence of CATC in an Operational Setting ..........................................................192
10.1 Methods............................................................................................................................192
10.1.1 Participants................................................................................................................192
10.1.2 Procedure ..................................................................................................................192
10.1.3 Analyses....................................................................................................................193
10.2 Results ..............................................................................................................................193
10.2.1 Downstream Impact on Marksmanship ....................................................................193
10.2.2 Downstream Impact on Training in the Units ..........................................................194
10.2.3 Downstream Impact on Self Efficacy .......................................................................195
10.3 Implications for Service System Development: Validation.............................................196
10.4 References ........................................................................................................................197
Chapter 11. Implications for Service System Development.........................................................198
11.1 Lessons Learned about Transfer of OBTE.......................................................................198
11.2 Implications for Service System Development................................................................199
11.2.1 Further Development and Analysis of Stakeholder Requirements for OBTE..........199
11.2.2 Further Development of OBTE as a Service System ...............................................199
11.2.3 Further Verification and Validation of OBTE ..........................................................201
11.3 References ........................................................................................................................203

Asymmetric Warfare Group

xiii

Riccio, Diedrich, & Cortes (Eds.)

Section III. Further Development of OBTE as a Service System ..........................................206

Chapter 12. Development of General Measures for Students ......................................................207
12.1 Intent ................................................................................................................................207
12.2 Performance Measure Development Process...................................................................207
12.2.1 Phase One: Define Performance Indicators (PI).......................................................207
12.2.2 Phase Two: Translate PI into performance measures ...............................................208
12.2.3 Phase Three: Measure refinement.............................................................................208
12.2.4 Phase Four: Retranslation of Measures ....................................................................208
12.3 Product of Measure Development....................................................................................209
12.3.1 Learner Perception of the Instructor and Course ......................................................209
12.3.2 Learner Engagement .................................................................................................211
12.3.3 Student Relationship with Teacher ...........................................................................212
12.3.4 Student Results .........................................................................................................214
12.3.5 Self-Report Measures ...............................................................................................216
12.4 Conclusion........................................................................................................................217
12.5 References ........................................................................................................................217
Chapter 13. Adapting OBTE in a Classroom Environment .........................................................219
13.1 Intent ................................................................................................................................219
13.2 Observing OBTE in the Classroom Environment............................................................219
13.2.1. Participants...............................................................................................................219
13.2.2. Procedure .................................................................................................................220
13.2.3. Measures ..................................................................................................................220
13.3 Utility of OBTE Measures in a Classroom Environment ................................................220
13.3.1 Generality of Measures .............................................................................................220
13.3.2. Implications for Improvement of Measures.............................................................221
13.3.3 Implications for improvement of course design .......................................................222
13.4 Use of 360 Reviews for Collaborative Reflection..........................................................223
13.4.1 The Role of a 360 Review in OBTE .......................................................................223
13.4.2 Narrative of a Participant Observer ..........................................................................225
13.5 Learning, cognitive load and motivation..........................................................................228
13.5.1 The NASA Task Load Index as a subjective measure of student workload.............228
13.5.2 Results.......................................................................................................................229
13.5.3 Implications ..............................................................................................................230
13.6 Conclusions ......................................................................................................................230
13.7 References ........................................................................................................................231
Chapter 14. Organizational Climate and Creation of Durable Change ........................................233
14.1 The Need ..........................................................................................................................233
14.2 Initial Indications of Possible Resistance to Change .......................................................234
14.3 Models and Considerations for Sustainable Change........................................................235
14.3.1 The Change Transition Period ..................................................................................235
14.3.2 Organizational Culture..............................................................................................237
14.3.3 Clarity of Mission and Shared Understanding..........................................................237
14.3.4 Relevant Observations During the Current Investigation.........................................238
14.3.5 Organizational Support and Incentives.....................................................................238
14.4 Conclusions ......................................................................................................................239
14.5 References ........................................................................................................................239

Asymmetric Warfare Group

Table of Contents

xiv

Chapter 15. Five ways OBTE can enable the Army Leader Development Strategy....................242
15.1 Background ......................................................................................................................242
15.2 An Emerging Consensus ..................................................................................................244
15.2.1 What Part to Balance?...............................................................................................244
15.2.2 Improving Training, by Design ................................................................................245
15.2.3 Increased Use of dL and Dependence on Self-Development ...................................246
15.2.4 Future Orientation, Unknown Requirements............................................................247
15.2.5 The Quality Instructor Challenge .............................................................................247
15.2.6 Purpose and Design are Key .....................................................................................248
15.2.7 A Natural Advantage ................................................................................................249
15.2.8 Task Specialization or Generalized Competency .....................................................249
15.3 Conclusion........................................................................................................................251
15.4 References ........................................................................................................................252
Epilogue. Integration of Leader Development, Education, Training, and Self-Development .....254
Toward Values-Based Standards for Army Doctrinal Requirements ......................................254
Nested Standards and Quality Assurance.................................................................................256
Needs and Opportunities for Staff & Faculty Development ....................................................259
A Role for Science and Measurement .................................................................................259
Toward Best Practices in Instructor Education....................................................................260
Critical Considerations for Further Scientific Investigation ....................................................263
The Necessity of Long-Term Studies ..................................................................................263
False Dichotomy of Objective-Subjective ...........................................................................264
Clarity About What Is Evaluated.........................................................................................265
Next Steps ............................................................................................................................266
References ................................................................................................................................268
Section IV. Appendices...............................................................................................................270
Appendix A. OBTE Principles & Practices: Instructor Measures................................................271
A.1 Genesis of Formative Measures for Instructors ................................................................271
A.2 Principles of Outcomes-Based Training & Education ......................................................272
A.3 Guide to Using Measures of Instructor Behavior..............................................................276
A.4 Complete Menu of Instructor Measures............................................................................279
Appendix B. OBTE Principles & Practices: Student Measures ...................................................318
B.1 Guide to Using Measures of Student Behavior .................................................................318
B.2 Complete Menu of Student Measures ...............................................................................319
Appendix C: A Commanders View of Outcomes-Based Training and Education .....................340
Summary ..................................................................................................................................340
Definition .............................................................................................................................340
Description...........................................................................................................................340
Elements of OBTE. ..................................................................................................................341
Developing the Outcomes....................................................................................................341
Developing the Training Plan ..............................................................................................341
Conducting Training ............................................................................................................342
How Training is Assessed....................................................................................................344
Conclusion................................................................................................................................344

Asymmetric Warfare Group

xv

Riccio, Diedrich, & Cortes (Eds.)

Appendix D: Warrior Ethos..........................................................................................................345

Analysis of the Concept and Initial Development of Applications..........................................345
Current Understanding of Warrior Ethos.............................................................................345
Purpose.................................................................................................................................348
Approach..............................................................................................................................348
Expansion of the Definition of Warrior Ethos.....................................................................348
The Tenets of Warrior Ethos ...............................................................................................349
Clarifying the Definition of Warrior Ethos..........................................................................351
Warrior Attributes Derived from the Tenets of Warrior Ethos ...........................................353
References ................................................................................................................................355
Supplementary Work Product from Warrior Ethos Project .....................................................355
Appendix E: Indicators of Warrior Ethos.....................................................................................356
Methods....................................................................................................................................356
Participants...........................................................................................................................356
Instruments and Facilities ....................................................................................................356
Procedure .............................................................................................................................356
Results ......................................................................................................................................358
Qualitative Findings.............................................................................................................358
Quantitative Findings...........................................................................................................358
Discussion ................................................................................................................................359

Asymmetric Warfare Group

Evolution of the investigation as reflected in the chapters of this monograph.