Riccio, G., & Diedrich, F. (2010). Implications for Service System Development. In: Riccio, G., Diedrich, F.

, & Cortes, M. (Eds.). An Initiative

in Outcomes-Based Training and Education: Implications for an Integrated Approach to Values-Based Requirements (Chapter 11). Fort Meade,
MD: U.S. Army Asymmetric Warfare Group. [Cover art by Wordle.net represents word frequency in text.]

Implications for Service System Development

198

Chapter 11. Implications for Service System Development

Gary E. Riccio
The Wexford Group International
Fred Diedrich
Aptima, Inc.
11.1 Lessons Learned about Transfer of OBTE
The overall goal of OBTE is to foster the development of Soldiers with respect to personal
attributes and competencies that are critical for success in Full Spectrum Operations (Table 1). It
offers a variety of strategies to enable progress toward such outcomes (Darwin, 2008; CornelldEchert, 2009a,b). More specifically, CATC focuses on the role of instructors in the
development of Soldiers through a field-based marksmanship course. Results from the surveys
indicate that participants are generally positive when asked about their experiences in CATC and
that they hold the instructional approach in high regard. Moreover, their comments tend to focus
on outcomes around which the approach is organized (problem solving, intangibles,
understanding and awareness, and combat performance). This suggests that they understand the
intent of OBTE. In addition, when asked about longer-term impact, respondents indicated that the
AWG course had an influence on their behavior when in theater, that it influenced their training
approaches with their own units, and that they experienced growth with respect to core outcomes.
The evidence from observations of actual instruction following CATC reveals both opportunities
and challenges. In particular, our findings suggest that DS do not consistently apply OBTE to the
extent possible in Initial Entry Training. In the particular application of OBTE at Fort Benning
and Fort Jackson, transfer of training from CATC is not complete (see also, Dean et al. 2009).
CATC provides exemplary marksmanship training while developing outcomes consistent with
OBTE. While marksmanship is a valuable motivator, it may distract participants in CATC from
learning as much as they otherwise might learn about OBTE as instructors. Perhaps more
importantly, as the AWG surveys suggest, a single one-week exposure to OBTE probably is not
sufficient to educate instructors about the varied opportunities for detailed implementation. This
has implications for further integration and development of instructor education about OBTE into
the Institutional Army (Cornell-dEchert, 2009a,b). In particular, there is a need for capabilities
that can provide or guide opportunities for continuing education about OBTE beyond field-based
courses like CATC or occasional workshops and seminars.
From the perspective of OBTE for process improvement, our data provided actionable feedback
about gaps in DS application of OBTE in Initial Entry Training. For instance, stress often was
added too quickly due to throughput considerations, such that increasing complexity was not used
as a training enabler. Privates did not come to understand why stress is added and how to manage
it. Instead, as training increased in complexity, Privates failed to execute fundamentals adequately
in the face of increasing complexity. This worked against development of confidence, initiative,
and combat performance. In addition, the level of one-on-one verbal interaction between DS and
Privates was very low. Privates rarely asked questions, and DS rarely asked Privates questions.
This limited the opportunities for growth in problem solving skills and understanding. Privates
often were told what to do and told solutions, rather than being guided to develop those solutions
themselves. Similarly, DS rarely stressed explanations of why. A consequence of this seemed to
be that Privates likely did not understand the deeper meaning behind their tasks and had limited
chances for real sense-making (e.g., Bransford et al., 2000). All these gaps and shortfalls should
Asymmetric Warfare Group

199

Riccio & Diedrich

be relatively easy to address without fundamental changes to doctrinally prescribed Programs of

Instruction or Training Support Packages (Cornell-dEchert, 2009b).
Collectively, the evidence suggests considerable potential for OBTE in the Army, especially if it
becomes explicitly more directed toward instructor education (Cornell-dEchert, 2009a,b). It is
noteworthy that, even without explicitly addressing instructor education, our findings suggest that
OBTE can motivate individuals to take ownership of their own learning and development and that
it can increase self-efficacy for teaching and developing others. The approach motivates a greater
interest in instructor-student interactions and how these interactions affect progress toward
developmental outcomes while satisfying course-specific learning objectives (Cornell-dEchert,
2009b). This has clear implications for the kind of future research that could have a practical
impact on training in the Army (Bandura, 1995; Deci & Ryan, 1985, 2008; Ryan & Deci, 2008;
Schwartz, Lin, Brophy, & Bransford, 1999; Sidman, Riccio, Semmens, Dean, & Diedrich, 2009).
11.2 Implications for Service System Development
In the context of CMMI for Services (CMMI Product Team, 2009), our findings lead to
recommendations for both instructors and their units. It emphasizes aspects of instruction to
sustain or to improve with respect to the ideals of OBTE (Cornell-dEchert, 2009a,b).
11.2.1 Further Development and Analysis of Stakeholder Requirements for OBTE
The practices associated with requirements in the Service System Development process area of
CMMI are not simply a front-end activity. The needs of end-users and stakeholders can change
over the life cycle of a service system. It also is the case than the understanding of requirements
becomes more refined and elaborated over experience with service system development,
transition and delivery (CMMI Product Team, 2009, p. 440; Swain, 2005). Development and
analysis of requirements should be continuous, for example, in due diligence on service incidents
and institutional constraints, restrictions, and limitations on the delivery of service. Some problem
areas observed in the current investigation warrant further collaboration with end-users and
stakeholders, such as:

Emphasis on throughput, or efficiency over effectiveness, puts understanding and

mastery at risk. While it is certainly the case that logistical demands necessitate
efficiency, this criterion is meaningless if knowledge or skills are not retained or not
readily adaptable to Full Spectrum Operations.
To the greatest extent possible, advancement across events and application of increasing
complexity should be based on leaner needs, implying that the pace of training needs to
be more fully differentiated between learners. From the perspective of OBTE, this
emphasizes mastery as a pursuit that builds confidence and effective combat application.
Where possible, instructors should allow Soldiers to assume responsibility for safety and
their learning. From the perspective of OBTE, such self-reliance builds confidence,
initiative, and accountability. Moreover, to the extent that Soldiers become accountable,
DS may be able to concentrate more on instruction and less on management.

11.2.2 Further Development of OBTE as a Service System

While CATC is focused on marksmanship, it is not about marksmanship per se, and this
distinction can be difficult to grasp given the participants intense interest in marksmanship. The
weeklong course, to which the present sample of DS was exposed, focused almost solely on
marksmanship. There was only a small amount of discussion on application of the methods to a
Asymmetric Warfare Group

Implications for Service System Development

200

different domain (e.g., land navigation). DS had difficulty understanding the instructional
techniques they witnessed (e.g., discussing minutes of angle in ballistics) as exemplars of a more
general instructional strategy (e.g., explaining the why, encouraging deliberate thought, and
awareness of context dependence of skills). This is not surprising given that exposure to only one
exemplar is not likely to promote generalization and abstraction. Hence, we recommend that
instructor education include applications of OBTE across multiple domains of skill and learning.
It also is clear that additional support can be provided to participants in a field-based train-thetrainer course soon after they complete the course. For instance, elements such as take-home
materials that explore application to different domains could be provided, perhaps employing
multimedia-based vignettes that challenge the instructors to apply lessons learned (Bruny,
Riccio, Sidman, Darowski, & Diedrich, 2006). In addition, such materials and exercises could be
employed in workshops, following marksmanship training for example, that promote student
application to domains outside of marksmanship (Sidman et al., 2009). The critical point is to
provide support that gives instructors opportunities to apply what they have learned about OBTE
to a variety of domains. Consistent with OBTE, the approach to further instructor education
would be to introduce additional domains that provide instructors with general strategies and
considerations rather than an apparent script for instruction in the new domain. Some general
strategies are reiterated below:

Where possible, instructors should allow Soldiers to solve problems. The emphasis
should be on asking leading questions instead of telling Soldiers the solution. From the
perspective of OBTE, this strategy gradually builds skills of thinking and problem
solving, and reinforces expectations of accountability rather than dependence on their
superiors.

The easiest change with a potential for significant impact is to create an environment that
fosters communication with Soldiers, rather than communication directed at Soldiers or
the absence of communication. From the perspective of OBTE, to the extent that
Soldiers feel that they can ask questions, and make some mistakes, they will come to
better understand combat application, master skills, and grow in confidence.

Where possible, peer coach should be utilized to overcome the limitations of instructorstudent ratio. Although peer coaches will no doubt provide inappropriate guidance on
occasion, this may be outweighed by the benefit of Soldiers coming to see themselves as
active participants in problem solving and discovery. This is especially true with the high
student-to-instructor ratios that are likely to be the biggest perceived obstacle to full
implementation of OBTE. To the extent that instruction is viewed not only as the transfer
of knowledge, but as an opportunity for collaborative problem solving, then the role of
the peer coach becomes critical.

It is important to note that inculcation of a mindset consistent with OBTE requires that the work
environment should be open to and supportive of change (e.g., Bandura, 1995; Burke & Hutchins,
2007; Rasmussen, 1997). As noted by Dean et al. (2009), DS indicated that they felt somewhat
constrained by the nature of the programs of instruction and associated resources. It is likely that
gaps in transfer of OBTE were rooted, at least in part, in perceived constraints on rifle
marksmanship training and the extent to which the DS believed they could deviate from the
common practices. A key challenge, therefore, is the extent to which programs of instruction can
be made to be more flexible and DS can be empowered to believe that they can take initiative to
more fully implement OBTE. Hence, there is a critical need to educate commanders and
supporting units about the value of OBTE as a service system. Without a command climate that
Asymmetric Warfare Group

201

Riccio & Diedrich

fosters agility in instructional units, an approach like OBTE is not likely to be successful
(Appendix C; Haskins, 2009; Schwitters, 2009).
11.2.3 Further Verification and Validation of OBTE
There is a broad range of best practices for peer review, verification, and validation in Service
System Development (CMMI Product Team, 2009, pp. 454-462). As in the current investigation,
it is important to employ a multifaceted and multidisciplinary approach to these activities to avoid
sub-optimized solutions and unintended consequences. Consideration of a portfolio of
methodologies, and even some exploration with alternative methods, also helps an organization
find the right level of effort and detail for its process improvement (Garcia & Turner, 2006). In
the present context, a specific aim was to identify ways in which the behavioral and social
sciences can provide guidance to verification and validation of an instructional service system.
There are two general characteristics of our approach in this respect: (a) develop grounded theory
for OBTE; and (b) promulgate scientific inquiry into OBTE as a process over time conducted
within a diverse community of practice.
On our view, identification and development of theoretical foundations is critically important in
service system development because of the diversity of methodologies that are available and that
may have to be employed for verification and validation. Theoretical coherence arguably is the
only way to identify convergence among otherwise incommensurate sources of evidence. From
the outset, we intentionally included in the research team, a diversity of subdisciplines and
theoretical commitments, mostly within the discipline of scientific psychology. The grounded
theory that emerged in this investigation came to be dominated by theoretical commitments allied
with the interrelated lineages of social learning theory, situated cognition, motivation and
emotion, ecological psychology, and dynamical systems theory in the social and behavioral
sciences. We are explicit about this bias wherever appropriate. More generally, we believe that
reflection and candor about theoretical biases should be a characteristic of systems engineering
applied to the integration and development of capabilities that have an impact on behavioral and
social phenomena (Flyvbjerg, 2001; Godfrey-Smith, 2003; Schrim & Caterino, 2006; see also,
Epilogue in this monograph).
We believe that the value of a theoretical commitment (e.g., coherence and directedness)
outweighs the potential disadvantage of narrowed vigilance in the development of grounded
theory. At the same time, we recognize that involvement of a broader scientific community can
mitigate the potential problem of narrowness. This is important beyond the value of debate,
skepticism, and alternative sources of evidence. It emphasizes an aspect of science that is
consistent with the intent of peer review, verification, and validation in systems engineering.
Scientific and technical understanding becomes more refined and elaborate over time with the
accumulation of evidence. One should avoid presumptive judgments based on point estimates
and isolated comparisons with questionable generalizability in domains as broad as training and
education or in specific applications that are replete with uncontrollable sources of variance.
In this regard, there is much to be learned from the other evidence-based services systems and
associated scientific disciplines about how to use and how not to use scientific evidence in social
decision-making (American Association for the Advancement of Science, 1993/2009; Bailar,
1997; Best, Trochim, Haggerty, Moor, & Norman, 2008; Foster & Huber, 1998; Glasziou &
Haynes, 2005; Kohn, Corrigan, & Donaldson, 1999; Mislevy & Riconscente, 2006; National
Research Council, 2009; Pellegrino, Chudowsky, & Glaser, 2001; Sackett, Rosenberg, Gray,
Haynes, & Richardson, 1996; Swales, 2000). A socially aware and scientifically based approach
to verification and validation should strive for theoretical coherence, juxtaposition of
Asymmetric Warfare Group

Implications for Service System Development

202

complementary and opposing perspectives, traceability to programs of research that span decades,
empirical evidence from multiple methods that are replicable, and consideration of sources of
variability and uniqueness in empirical findings to ensure that conclusions are credible,
transferable, dependable, and confirmable (cf., Denzin & Lincoln, 2003; Foucault, 1966/2002;
Godfrey-Smith, 2003; Kuhn, 1962/1970; Popper, 1959). It should not be limited by narrow
conceptions of hypothesis testing (cf., Flyvbjerg, 2001; Henkel & Morrison, 1970/2006; Kline,
2004; A. Ryan, 1959; Schrim & Caterino, 2006).
With respect to verification and validation, as well as scientific influence on these endeavors, an
important contribution of this investigation is to stress the importance of productive dialog about
OBTE within a diverse community of stakeholders. The intent of this dialog should not be to
prescribe the use of particular instructional methods or techniques but to provide some scientific
guidance about the most fruitful topics of conversation and innovation by instructors (cf., James,
1899/1907, pp. 7-11). In particular, we believe this investigation can stimulate productive dialog
because of the definition of OBTE in terms of instructor behavior and instructor-student
interactions and, more specifically, because of the measures of instructor and student behavior
that enable OBTE to be verifiable. They provide one topic of conversation for forums in which
there is peer-to-peer sharing of information about best practice in training and education (e.g.,
Costanza, Leibrecht, Cooper, & Sanders, 2009). In the context of continuous verification and
validation, we believe such conversations should include scientists as well as instructors,
instructional designers, course developers, quality assurance personnel, and commanders in
instructional programs. The most radical departure would be to facilitate contributions from
stakeholders in theater (Riccio, dEchert, et al., 2006; Riccio, Lerario, et al., 2006).
There is dialog and debate about OBTE in a diverse community of stakeholders, and it appears to
be growing (AWG, 2009). Figure 6 suggests a challenge in achieving efficient sharing of issues
and lessons learned about OBTE. Stakeholders are widely dispersed. Decentralized collaboration
and the resulting lessons learned thus are not readily apparent in timely fashion to decision
makers. Potential solutions to this problem are emerging in the nascent Army Training Network
(ATN) that builds on the recently revised Army Field Manual, FM 7-0 Training for Full
Spectrum Operations, and transforms FM 7-1 into a Virtual Field Manual (Davis, 2009). One
use of this forum would be for continuous peer review that is central to verification and validation
in CMMI Service System Development (CMMI Product Team, 2009). A scientific approach to
verification and validation can help ATN establish a topic of peer-to-peer discussion that is more
likely to be on point, efficient, systematic, and actionable for OBTE in particular and for good
training in general.

Figure 6: Needs for distributed peer-to-peer collaboration about OBTE (after Devens, 2009)

Asymmetric Warfare Group

203

Riccio & Diedrich

11.3 References
American Association for the Advancement of Science (2009). Benchmarks for scientific literacy.
New York: Oxford University Press. Retrieved April, 2009, from American Association for
the Advancement of Science, Project 2061, http://tinyurl.com/c9mrcr. (Original work
published 1993)
Asymmetric Warfare Group (2009). U.S. Army Asymmetric Warfare Group workshop on
Outcomes-based Training and Education. Applied Physics Laboratory, Johns Hopkins
University, Laurel, MD.
Bailar, J. (1997). The promise and problems of meta-analysis. New England Journal of Medicine,
337(8), 559-561.
Bandura, A. (Ed.) (1995). Self efficacy in changing societies. Cambridge, UK: Cambridge
University Press.
Best, A., Trochim, W., Haggerty, J., Moor, G., & Norman, C. (2008). Systems Thinking for
Knowledge Integration: New Models for Policy-Research Collaboration. In L. McKee, E.
Ferlie & P. Hyde (Eds.), Organizing and Reorganizing: Power and Change in Health Care
Organizations (pp. 154-166). New York, NY: Palgrave Macmillan.
Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000). How people learn: Brain, mind,
experience, & school. Washington, DC: National Academy Press.
Bruny, T., Riccio, G., Sidman, J., Darowski, A., & Diedrich, F. (2006). Enhancing warrior ethos
in initial entry training. Proceedings of the 50th Annual Meeting of the Human Factors and
Ergonomics Society, San Francisco, CA.
Burke, L.A., & Hutchins, H.M. (2007). Training transfer: An integrative literature review. Human
Resources Development Review, 6, 263-296.
CMMI Product Team (2009). CMMI for services, version 1.2. (CMU/SEI-TR-2009-001; ESCTR-2009-001). Pittsburgh, PA: Carnegie Mellon University.
Cornell-dEchert, B. (2009a). An introduction to outcomes-based training and education. Fort
Meade, MD: Asymmetric Warfare Group.
Cornell-dEchert, B. (2009b). Outcomes-based training and education: Implementation guide.
Fort Meade, MD: Asymmetric Warfare Group.
Costanza, M., Leibrecht, B., Cooper, W., & Sanders, W. (2009). Peer-To-Peer Training
Facilitators Guide (ARI Research Product, in press). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences.
Darwin, M. (2008). Outcomes-based training and education: fostering adaptability in full
spectrum operations (Briefing, December 2008). Fort Meade, MD: Asymmetric Warfare
Group.
Davis, J. (2008, January 21). Call for Army Training Network (ATN) training products. Retrieved
March, 2009 from US Army Combined Arms Center, Blog Network, Army Training
Network, http://tinyurl.com/alxknw.
Dean, C., Diedrich, F., Artis, S., Horn, Z., Jefferson, T., & Riccio, G. (2008). Outcomes-based
training and education: Student measures for rifle marksmanship. Report to the Asymmetric
Warfare Group (Contract W9113M-06-D-0005). Vienna, VA: Wexford-CACI, Inc.
Devens, M. (2009, March). Introductory remarks. Presentation at the US Army Asymmetric
Warfare Group workshop on Outcomes-based Training and Education, Applied Physics
Laboratory, Johns Hopkins University, Laurel, MD.
Deci, E.L. & Ryan, R.M. (1985). Intrinsic motivation and self-determination in human behavior.
New York: Plenum.
Deci. E.L. & Ryan, R.M. (2008). Self determination theory: A macrotheory of human motivation,
development, and health. Canadian Psychology, 49, 182-185.

Asymmetric Warfare Group

Implications for Service System Development

204

Flyvbjerg, B. (2001). Making social science matter: Why social inquiry fails and how it can
succeed again. Cambridge, UK: Cambridge University.
Foster, K, & Huber, P. (1998). Judging science: Scientific knowledge and the federal courts.
Cambridge, MA: MIT.
Foucault, M. (2002). Order of things: An archaeology of the human sciences New York:
Routledge Classics. (Original work published 1966).
Garcia, S, & Turner, R. (2006). CMMI(R) survival guide: Just enough process improvement.
Upper Salle River, NJ: Addison-Wesley.
Glasziou, P., & Haynes, B. (2005). The paths from research to improved health outcomes. ACP
Journal Club, 142(2), A8-10.
Godfrey-Smith, P. (2003). Theory and reality. Chicago, IL: University of Chicago Press.
Haskins, C. (2009, March). Development of outcomes based training. Presentation at the US
Army Asymmetric Warfare Group workshop on Outcomes-based Training and Education,
Applied Physics Laboratory, Johns Hopkins University, Laurel MD.
Henkel, R. & Morrison, D. (Eds.) (2006). The significance test controversy: A reader.
Piscataway, NJ: Aldine Transaction. (Original work published 1970)
James, W. (1899/1907). Talks to teachers on psychology: and to students on some of life's ideals.
New York, NY: Henry Holt and Company.
Kline, R.B. (2004). Beyond significance testing: Reforming data analysis methods in behavioral
research. Washington, DC: APA.
Kohn, L., Corrigan, J., & Donaldson, M. (Eds.) (1999). To err is human: Building a safer health
system. Washington, DC: National Academy Press.
Kuhn, Thomas (1970). The structure of scientific revolutions. Chicago: University of Chicago
Press. (Original work published 1962)
Mislevy, R. J., & Riconscente, M. M. (2006). Evidence-centered assessment design: Layers,
concepts, and terminology. In S. Downing & T. Haladyna (Eds.), Handbook of Test
Development (pp. 61-90). Mahwah, NJ: Erlbaum.
National Research Council (2009). Strengthening forensic science in the United States: A path
forward. Retrieved February, 2009 from National Academies Press,
http://www.nap.edu/catalog/12589.html.
Pellegrino, J., Chudowsky, N., & Glaser, R. (Eds.) (2001). Knowing what students know: The
science and design of educational assessment. Washington, DC: National Academy Press.
Popper, Karl (1959). The logic of scientific discovery. New York: Basic Books.
Rasmussen, J. (1997). Risk management in a dynamic society: a modeling problem. Safety
Science, 27, 183-213.
Riccio, G., dEchert, B.C., Lerario, M., Pound, D., Bruny, T., & Diedrich, F. (2006). Enhancing
Joint Task Force Cognitive Leadership Skills. Report to the Army Research Institute for the
Behavioral and Social Sciences, contract number Army contract no. W74V8H-06-P-0186.
Vienna, VA: The Wexford Group International.
Riccio, G., Lerario, M., Cornell dEchert, B., Pound, D., Bruny, T., & Diedrich, F. (2006).
Training a Joint and expeditionary mindset. Report to the Army Research Institute for the
Behavioral and Social Sciences, contract number W74V8H-06-P-0189. Vienna, VA: The
Wexford Group International.
Ryan, T. (1959). Multiple comparisons in psychological research, Psychological Bulletin, 56, 2647.
Sackett, D.L., Rosenberg, W.M., Gray, J.A., Haynes, R.B., & Richardson, W.S. (1996). Evidence
based medicine: What it is and what it isn't. British Journal of Medicine, 312(7023), 71-72.
Schwartz, D.L., Lin, X., Brophy, S., & Bransford, J.D. (1999). Toward the development of
flexibly adaptive instructional designs. In: C.M. Reigilut (Ed.). Instructional Design Theories
and Models: Volume II. Hillsdale, NJ: Erlbaum.

Asymmetric Warfare Group

205

Riccio & Diedrich

Schwitters, J. (2009, March). Command imperative for change. Presentation at the US Army
Asymmetric Warfare Group workshop on Outcomes-based Training and Education, Applied
Physics Laboratory, Johns Hopkins University, Laurel, MD.
Schrim, S. & Caterino, B. (2006). Making political science matter: debating knowledge,
research, and method. New York: New York University.
Sidman, J., Riccio, G., Semmens, R., Geyer, A., Dean, C., & Frederick Diedrich, F. (2009).
Reshaping Army institutional training: Current training. Final Report to the Army Research
Institute for the Behavior and Social Sciences, contract number W74V8H-04-D-0047 DO
0010.
Swain, R. (2005). Changes in instructional system design (ISD): Improving training product
delivery to United States Army Soldiers. USAWC Strategic Research Project. Carlisle
Barracks, PA: U.S. Army War College.
Swales, J. (2000). The troublesome search for evidence: Three cultures in need of integration.
Journal of the Royal Society of Medicine, 93, 402-407.
Tobias, S. & Duffy, T. (eds.) (2009). Constructivist instruction: Success or failure. New York:
Routledge:

Asymmetric Warfare Group

ix

Riccio, Diedrich, & Cortes (Eds.)

TABLE OF CONTENTS
.

page
Prologue: A Programmatic View of the Inquiry into Outcomes-Based Training & Education.......1
Historicity of our Research on OBTE ..........................................................................................1
The Approach and Lessons Learned from the Research..............................................................3
Documentation of the Research ...................................................................................................4
Section I. Development of Stakeholder Requirements for OBTE..............................................6
Chapter 1. Preparation for Full Spectrum Operations ......................................................................7
1.1 Requirements of Full Spectrum Operations ...........................................................................8
1.2 Outcomes-Based Training and Education (OBTE)..............................................................10
1.2.1 Exemplar of OBTE: Combat Applications Training Course........................................11
1.2.2 OBTE as a Multifaceted Instructional System .............................................................12
1.3 An Appraisal of Instruction with Respect to OBTE ............................................................13
1.3.1 A Systems Engineering Framework for Integration and Development of OBTE ........13
1.3.2 Preparation for Validation and Verification .................................................................14
1.4 References ............................................................................................................................17
Chapter 2. Formative Measures for Instructors ..............................................................................20
2.1 Development of Formative Measures ..................................................................................20
2.1.1 The COMPASS Methodology ......................................................................................20
2.1.2 Development of Measures for OBTE ...........................................................................21
2.2 Description of Formative Measures .....................................................................................21
2.2.1 Results of the COMPASS Process................................................................................21
2.2.2 Elaboration on the Description of Measures.................................................................23
2.3 OBTE Performance Measures: Planning for Training.........................................................23
2.3.1 Define Outcomes ..........................................................................................................23
2.3.2 Create a Positive Learning Environment ......................................................................25
2.3.3 Create the Parameters of Learning................................................................................27
2.4 OBTE Performance Indicators: Training Execution............................................................28
2.4.1 Communicate the Parameters of Learning....................................................................28
2.4.2 Training Emphasizes Broad Combat or Mission Success ............................................29
2.4.3 Customize Instruction When Possible Based on Constraints/Conditions ....................31
2.4.4 Facilitates Learning of Concepts ..................................................................................32
2.4.5 Creates a positive learning environment.......................................................................34
2.4.6 Instructors Utilize Measures of Effectiveness & Self-Evaluation ................................36
2.4.7 Uses scenarios to facilitate learning..............................................................................38
2.4.8 Instructors exhibit intangible attributes in own actions ................................................40
2.4.9 Hotwashes and Mini-AAR............................................................................................42
2.5 Uses of the Measures ...........................................................................................................43
2.5.1 Formative Measures for Instructors ..............................................................................44
2.5.2 Quality Assurance and Instructor Education ................................................................44
2.5.3 Continuous Improvement of Assessments....................................................................45
2.5.4 Program Evaluation and Organizational Change..........................................................46
2.6 References ............................................................................................................................46

Asymmetric Warfare Group

Table of Contents

Chapter 3. Principles and Practices of Outcomes Based Training & Education............................50

3.1 Multifaceted Inquiry.............................................................................................................50
3.1.1 Interaction with Progenitors of OBTE..........................................................................51
3.1.2 AWG Documents on OBTE .........................................................................................52
3.1.3 Collaborative Reflection on Participant Observation in CATC ...................................52
3.1.4 Interaction with Stakeholders .......................................................................................53
3.2 Essential Characteristics of OBTE.......................................................................................53
3.2.1 The Meaning of Developmental is a Critical Difference..............................................53
3.2.2 The Definition of Outcomes is a Critical Difference....................................................56
3.2.3 The Emphasis on Values and Causally Potent Intangibles is a Critical Difference .....58
3.2.4 The Meaning of Experience is a Critical Difference ....................................................61
3.2.5 The Emphasis on Instructor-Student Interactions is a Critical Difference ...................62
3.2.6 The Emphasis on Learning to Learn is a Critical Difference .......................................63
3.2.7 The Emphasis on Collaborative Design and Development is a Critical Difference.....65
3.3 Toward a Grounded Theory for OBTE ................................................................................66
3.3.1 Need for an Integrated Interdisciplinary Framework ...................................................66
3.3.2 Formative Measures of Instructor Behavior as Evolving Best Practices of OBTE......67
3.4 Emerging Best Practices in OBTE for a Community-Centered Environment.....................68
3.4.1 Leadership and Enculturation of Soldiers.....................................................................68
3.4.2 Robust and Adaptable Plan...........................................................................................70
3.4.3 Instructors as Role Models ...........................................................................................70
3.4.4 Collaborative Identification of Outcomes and Measures .............................................71
3.5 Emerging Best Practices in OBTE for a Knowledge-Centered Environment .....................71
3.5.1 Integrated Understanding of Basic Soldier Skills in Full Spectrum Operations ..........72
3.5.2 Task Relevance of Planned Instructional Events..........................................................72
3.5.3 Reveal Operational Relevance of Training...................................................................73
3.5.4 Incorporate Stress into Instructional Events .................................................................73
3.5.5 Identify General Lessons Learned and Extrapolate to New Situations ........................74
3.6 Emerging Best Practices in OBTE for an Assessment-Centered Environment ...................74
3.6.1 Collaborative Reflection and Problem Solving ............................................................75
3.6.2 Communication.............................................................................................................75
3.6.3 Nature and Extent of Guidance.....................................................................................76
3.6.4 Establish a Pervasive Mindset of Collaborative Reflection..........................................76
3.7 Emerging Best Practices in OBTE for a Learner-Centered Environment ...........................77
3.7.1 Soldier Motivation and Development of Intangibles....................................................77
3.7.2 Plan for Development of the Individual .......................................................................78
3.7.3 Get Students to Take Ownership ..................................................................................78
3.7.4 Collaborative Reflection as a Means to Develop Self Efficacy....................................79
3.8 References ............................................................................................................................79
Chapter 4. Grounded Theory for Values-Based Training & Education .........................................86
4.1 Exploration of Holistic and Functionalistic Underpinnings for OBTE ...............................86
4.1.1 Fundamental Units of Analysis.....................................................................................87
4.1.2 Nested Time Scales and Adaptability ...........................................................................88
4.1.3 Adaptability and Ambiguity .........................................................................................90
4.1.4 Mechanistic Analogies and Predominant Experimental Paradigms .............................92
4.2 Three Pillars for the Scientific Foundation of OBTE ..........................................................93
4.2.1 Ecological Psychology..................................................................................................93
4.2.2 Self-Efficacy Theory.....................................................................................................97
4.2.3 Positive psychology ......................................................................................................98
4.3 A More Integrated Scientific Infrastructure .......................................................................101
Asymmetric Warfare Group

xi

Riccio, Diedrich, & Cortes (Eds.)

4.3.1 Self Determination Theory .........................................................................................101
4.3.2 Situated Learning Theory ...........................................................................................103
4.3.3 Existential Psychology................................................................................................105
4.4 Building on the Scientific Infrastructure for OBTE...........................................................109
4.4.1 Triadic Frameworks ....................................................................................................109
4.4.2 Further Development ..................................................................................................112
4.5 References ..........................................................................................................................112

Chapter 5. Passion and Reason in Values-Based Learning & Development ...............................118

5.1 The Nested Self ..................................................................................................................118
5.1.1 An Alternative to Individual versus Collective ..........................................................118
5.1.2 Cognition and Reality .................................................................................................119
5.2 Conscious Experience and the Dynamics of Thinking ......................................................122
5.3 Emotion, Information, and Engagement ............................................................................125
5.3.1 Ecological Perspective on Emotion ............................................................................125
5.3.2 Emotion as Engagement .............................................................................................126
5.3.3 Implications for Training and Education ....................................................................129
5.4 Emotion, Decision-Making, and Inter-Temporal Choice...................................................129
5.4.1 Toward a More Integrated Theory..............................................................................129
5.4.2 Emotion and Decision-Making ...................................................................................130
5.4.3 Emotion and Nested Time Scales ...............................................................................131
5.4.4 Neuroeconomics and Inter-Temporal Reasoning .......................................................132
5.5.5 Inter-Temporal Reasoning and Adaptive Dynamical Systems...................................133
5.5 Beyond Science ..................................................................................................................134
5.5.1 Existentialism..............................................................................................................134
5.5.2 The Soldier-Scholar as an Emergent Property of a Collective Pursuit.......................135
5.6 References ..........................................................................................................................137
Section II. Verification and Validation of OBTE as a Service System ..................................142
Chapter 6. Initial Impressions of Participation in CATC .............................................................143
6.1 Methods..............................................................................................................................143
6.1.1 Participants..................................................................................................................143
6.1.2 Procedure ....................................................................................................................143
6.1.3 Analyses......................................................................................................................144
6.2 Results ................................................................................................................................144
6.3 Implications for Service System Development: Peer Review ...........................................146
6.4 References ..........................................................................................................................147
Chapter 7. Local Development of Measures of Effectiveness .....................................................149
7.1 What do Instructors Believe Soldiers Should Learn in Initial Entry Training? .................149
7.2 Measure Development Process ..........................................................................................150
7.3 What do OBTE-Trained DS Believe is Important to Assess in BRM/ARM? ...................151
7.4 Implications........................................................................................................................156
7.5 Conclusions ........................................................................................................................158
7.6 References ..........................................................................................................................159

Asymmetric Warfare Group

Table of Contents

xii

Chapter 8. Observations of Behavior and Communication in Rifle Marksmanship Training .....160

8.1 Methods..............................................................................................................................160
8.1.1 Participants..................................................................................................................160
8.1.2 Procedure ....................................................................................................................160
8.1.3 Analyses......................................................................................................................161
8.2 Results ................................................................................................................................163
8.2.1 Behavior of DS ...........................................................................................................163
8.2.2 Behavior and Performance of Privates .......................................................................165
8.2.3 Patterns of Communication ........................................................................................168
8.2.4 Potential Influence of Instructor Behavior on Performance of Privates .....................170
8.3 Implications for Service System Development..................................................................171
8.3.1 Verification of OBTE .................................................................................................171
8.3.2 Validation of OBTE....................................................................................................172
8.4 References ..........................................................................................................................173
Chapter 9. Impact on Rifle Marksmanship Training....................................................................174
9.1 Behavioral Data Collection During Basic Rifle Marksmanship ........................................174
9.1.1 Method ........................................................................................................................174
9.1.2 Assessment..................................................................................................................175
9.1.3 Results An Overview ...............................................................................................177
9.1.4 Evidence for Influence of OBTE ................................................................................178
9.1.5 Behavior of Drill Sergeants after Exposure to OBTE ................................................180
9.1.6 Behavior of Privates....................................................................................................182
9.1.7 Patterns of Communication ........................................................................................186
9.1.8 Summary .....................................................................................................................186
9.2 Attitudes Toward an OBTE in Basic Training...................................................................187
9.2.1 Method ........................................................................................................................187
9.2.2 Results.........................................................................................................................187
9.4 References ..........................................................................................................................191
Chapter 10. Influence of CATC in an Operational Setting ..........................................................192
10.1 Methods............................................................................................................................192
10.1.1 Participants................................................................................................................192
10.1.2 Procedure ..................................................................................................................192
10.1.3 Analyses....................................................................................................................193
10.2 Results ..............................................................................................................................193
10.2.1 Downstream Impact on Marksmanship ....................................................................193
10.2.2 Downstream Impact on Training in the Units ..........................................................194
10.2.3 Downstream Impact on Self Efficacy .......................................................................195
10.3 Implications for Service System Development: Validation.............................................196
10.4 References ........................................................................................................................197
Chapter 11. Implications for Service System Development.........................................................198
11.1 Lessons Learned about Transfer of OBTE.......................................................................198
11.2 Implications for Service System Development................................................................199
11.2.1 Further Development and Analysis of Stakeholder Requirements for OBTE..........199
11.2.2 Further Development of OBTE as a Service System ...............................................199
11.2.3 Further Verification and Validation of OBTE ..........................................................201
11.3 References ........................................................................................................................203

Asymmetric Warfare Group

xiii

Riccio, Diedrich, & Cortes (Eds.)

Section III. Further Development of OBTE as a Service System ..........................................206

Chapter 12. Development of General Measures for Students ......................................................207
12.1 Intent ................................................................................................................................207
12.2 Performance Measure Development Process...................................................................207
12.2.1 Phase One: Define Performance Indicators (PI).......................................................207
12.2.2 Phase Two: Translate PI into performance measures ...............................................208
12.2.3 Phase Three: Measure refinement.............................................................................208
12.2.4 Phase Four: Retranslation of Measures ....................................................................208
12.3 Product of Measure Development....................................................................................209
12.3.1 Learner Perception of the Instructor and Course ......................................................209
12.3.2 Learner Engagement .................................................................................................211
12.3.3 Student Relationship with Teacher ...........................................................................212
12.3.4 Student Results .........................................................................................................214
12.3.5 Self-Report Measures ...............................................................................................216
12.4 Conclusion........................................................................................................................217
12.5 References ........................................................................................................................217
Chapter 13. Adapting OBTE in a Classroom Environment .........................................................219
13.1 Intent ................................................................................................................................219
13.2 Observing OBTE in the Classroom Environment............................................................219
13.2.1. Participants...............................................................................................................219
13.2.2. Procedure .................................................................................................................220
13.2.3. Measures ..................................................................................................................220
13.3 Utility of OBTE Measures in a Classroom Environment ................................................220
13.3.1 Generality of Measures .............................................................................................220
13.3.2. Implications for Improvement of Measures.............................................................221
13.3.3 Implications for improvement of course design .......................................................222
13.4 Use of 360 Reviews for Collaborative Reflection..........................................................223
13.4.1 The Role of a 360 Review in OBTE .......................................................................223
13.4.2 Narrative of a Participant Observer ..........................................................................225
13.5 Learning, cognitive load and motivation..........................................................................228
13.5.1 The NASA Task Load Index as a subjective measure of student workload.............228
13.5.2 Results.......................................................................................................................229
13.5.3 Implications ..............................................................................................................230
13.6 Conclusions ......................................................................................................................230
13.7 References ........................................................................................................................231
Chapter 14. Organizational Climate and Creation of Durable Change ........................................233
14.1 The Need ..........................................................................................................................233
14.2 Initial Indications of Possible Resistance to Change .......................................................234
14.3 Models and Considerations for Sustainable Change........................................................235
14.3.1 The Change Transition Period ..................................................................................235
14.3.2 Organizational Culture..............................................................................................237
14.3.3 Clarity of Mission and Shared Understanding..........................................................237
14.3.4 Relevant Observations During the Current Investigation.........................................238
14.3.5 Organizational Support and Incentives.....................................................................238
14.4 Conclusions ......................................................................................................................239
14.5 References ........................................................................................................................239

Asymmetric Warfare Group

Table of Contents

xiv

Chapter 15. Five ways OBTE can enable the Army Leader Development Strategy....................242
15.1 Background ......................................................................................................................242
15.2 An Emerging Consensus ..................................................................................................244
15.2.1 What Part to Balance?...............................................................................................244
15.2.2 Improving Training, by Design ................................................................................245
15.2.3 Increased Use of dL and Dependence on Self-Development ...................................246
15.2.4 Future Orientation, Unknown Requirements............................................................247
15.2.5 The Quality Instructor Challenge .............................................................................247
15.2.6 Purpose and Design are Key .....................................................................................248
15.2.7 A Natural Advantage ................................................................................................249
15.2.8 Task Specialization or Generalized Competency .....................................................249
15.3 Conclusion........................................................................................................................251
15.4 References ........................................................................................................................252
Epilogue. Integration of Leader Development, Education, Training, and Self-Development .....254
Toward Values-Based Standards for Army Doctrinal Requirements ......................................254
Nested Standards and Quality Assurance.................................................................................256
Needs and Opportunities for Staff & Faculty Development ....................................................259
A Role for Science and Measurement .................................................................................259
Toward Best Practices in Instructor Education....................................................................260
Critical Considerations for Further Scientific Investigation ....................................................263
The Necessity of Long-Term Studies ..................................................................................263
False Dichotomy of Objective-Subjective ...........................................................................264
Clarity About What Is Evaluated.........................................................................................265
Next Steps ............................................................................................................................266
References ................................................................................................................................268
Section IV. Appendices...............................................................................................................270
Appendix A. OBTE Principles & Practices: Instructor Measures................................................271
A.1 Genesis of Formative Measures for Instructors ................................................................271
A.2 Principles of Outcomes-Based Training & Education ......................................................272
A.3 Guide to Using Measures of Instructor Behavior..............................................................276
A.4 Complete Menu of Instructor Measures............................................................................279
Appendix B. OBTE Principles & Practices: Student Measures ...................................................318
B.1 Guide to Using Measures of Student Behavior .................................................................318
B.2 Complete Menu of Student Measures ...............................................................................319
Appendix C: A Commanders View of Outcomes-Based Training and Education .....................340
Summary ..................................................................................................................................340
Definition .............................................................................................................................340
Description...........................................................................................................................340
Elements of OBTE. ..................................................................................................................341
Developing the Outcomes....................................................................................................341
Developing the Training Plan ..............................................................................................341
Conducting Training ............................................................................................................342
How Training is Assessed....................................................................................................344
Conclusion................................................................................................................................344

Asymmetric Warfare Group

xv

Riccio, Diedrich, & Cortes (Eds.)

Appendix D: Warrior Ethos..........................................................................................................345

Analysis of the Concept and Initial Development of Applications..........................................345
Current Understanding of Warrior Ethos.............................................................................345
Purpose.................................................................................................................................348
Approach..............................................................................................................................348
Expansion of the Definition of Warrior Ethos.....................................................................348
The Tenets of Warrior Ethos ...............................................................................................349
Clarifying the Definition of Warrior Ethos..........................................................................351
Warrior Attributes Derived from the Tenets of Warrior Ethos ...........................................353
References ................................................................................................................................355
Supplementary Work Product from Warrior Ethos Project .....................................................355
Appendix E: Indicators of Warrior Ethos.....................................................................................356
Methods....................................................................................................................................356
Participants...........................................................................................................................356
Instruments and Facilities ....................................................................................................356
Procedure .............................................................................................................................356
Results ......................................................................................................................................358
Qualitative Findings.............................................................................................................358
Quantitative Findings...........................................................................................................358
Discussion ................................................................................................................................359

Asymmetric Warfare Group

Evolution of the investigation as reflected in the chapters of this monograph.