A DECISION-MAKING FRAMEWORK FOR TOTAL OWNERSHIP COST MANAGEMENT OF COMPLEX SYSTEMS: A DELPHI STUDY

by Russel J. King

A Dissertation Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Business Administration

University of Phoenix November 2007

UMI Number: 3302636

Copyright 2008 by King, Russel J. All rights reserved.

UMI Microform 3302636 Copyright 2008 by ProQuest Information and Learning Company. All rights reserved. This microform edition is protected against unauthorized copying under Title 17, United States Code.

ProQuest Information and Learning Company 300 North Zeeb Road P.O. Box 1346 Ann Arbor, MI 48106-1346

A DECISION-MAKING FRAMEWORK FOR TOTAL OWNERSHIP COST MANAGEMENT OF COMPLEX SYSTEMS: A DELPHI STUDY by Russel J. King November 2007

Approved:

Marilyn K. Simon, PbD., Mentor

John DeNigris, Ph.D., Committee Member Tom G r i m D.B.A., Committee Member

Accepted and Signed: Accepted and Signed:

Dean, School of ~dvanced Studies University of Phoenix

ABSTRACT This qualitative study, using a modified Delphi method, was conducted to develop a decision-making framework for the total ownership cost management of complex systems in the aerospace industry. The primary focus of total ownership cost is to look beyond the purchase price when evaluating complex system life cycle alternatives. A thorough literature review and the opinions of a group of qualified experts resulted in a compilation of total ownership cost best practices, cost drivers, key performance factors, applicable assessment methods, practitioner credentials and potential barriers to effective implementation. The expert panel provided responses to the study questions using a 5point Likert-type scale. Data were analyzed and provided to the panel members for review and discussion with the intent to achieve group consensus. As a result of the study, the experts agreed that a total ownership cost analysis should (a) be as simple as possible using historical data; (b) establish cost targets, metrics, and penalties early in the program; (c) monitor the targets throughout the product lifecycle and revise them as applicable historical data becomes available; and (d) directly link total ownership cost elements with other success factors during program development. The resultant study framework provides the business leader with incentives and methods to develop and implement strategies for controlling and reducing total ownership cost over the entire product life cycle when balancing cost, schedule, and performance decisions.

iii DEDICATION For my wife, Cathy and our children, Anthony and Kelly.

iv ACKNOWLEDGMENTS I wish to thank my distinguished committee, Dr. Marilyn Simon, Dr. John DeNigris, and Dr. Tom Griffin for their contribution, analyses and assistance throughout this dissertation. Special thanks to my chairperson and mentor, Dr. Marilyn Simon for her patience, understanding, and support. I would like to express my appreciation to Elbit Systems of America and EFW Inc. for sponsoring me through much of the program.

v TABLE OF CONTENTS LIST OF TABLES...............................................................................................................x LIST OF FIGURES ........................................................................................................... xi CHAPTER 1: INTRODUCTION ........................................................................................1 Background of the Problem .................................................................................................2 Problem Statement ...............................................................................................................8 Purpose of the Study ............................................................................................................9 Significance of the Study .....................................................................................................9 Significance of the Study to Leadership ............................................................................10 Nature of the Study ............................................................................................................10 Research Questions............................................................................................................13 Theoretical Framework......................................................................................................13 Definition of Terms............................................................................................................16 Assumptions.......................................................................................................................17 Scope and Limitations........................................................................................................18 Delimitations......................................................................................................................19 Summary ............................................................................................................................20 CHAPTER 2: LITERATURE REVIEW ...........................................................................22 Title Searches, Articles, Research Documents, and Journals ............................................23 Total Ownership Cost ........................................................................................................24 Defining the Need for Total Ownership Cost ....................................................................26 Benefits of Total Ownership Cost Analysis.......................................................................28 Barriers to Implementation of Total Ownership Cost........................................................29

vi Total Ownership Cost Critical Cost Drivers......................................................................31 The Product Life Cycle......................................................................................................33 Product Life-Cycle Management.......................................................................................35 Specifications and Requirements Development ................................................................36 Customer Relationship Management.................................................................................37 Acquisition and Procurement.............................................................................................38 Supply Chain Management................................................................................................39 Design and Development...................................................................................................41 Systems Engineering..........................................................................................................42 Supportability.....................................................................................................................43 Manufacturing Quality and Reliability Practices...............................................................44 Product Quality Management ......................................................................................45 Product Reliability Management .................................................................................48 Reliability Assessment.................................................................................................49 Product Maintainability Practices ................................................................................49 Operational Availability.....................................................................................................51 Military Aerospace Operational Availability...............................................................51 Commercial Aerospace Operational Availability........................................................52 Operational Availability Applications .........................................................................52 Activity-Based Costing......................................................................................................53 Theory of Constraints ........................................................................................................53 Earned Value Management................................................................................................54 Warranties ..........................................................................................................................55

vii Customer Service ...............................................................................................................56 Product Disposal ................................................................................................................57 The Delphi Method ............................................................................................................57 Summary ............................................................................................................................61 Conclusion .........................................................................................................................63 CHAPTER 3: METHODOLOGY .....................................................................................65 Research Design.................................................................................................................66 Appropriateness of Design.................................................................................................70 Research Questions............................................................................................................72 Selection of a Population of Experts..................................................................................72 Informed Consent...............................................................................................................73 Sampling Frame .................................................................................................................74 Confidentiality ...................................................................................................................75 Geographic Location..........................................................................................................75 Instrumentation ..................................................................................................................76 Data Collection ..................................................................................................................77 Data Analysis .....................................................................................................................79 Validity and Reliability......................................................................................................81 Summary ............................................................................................................................82 CHAPTER 4: PRESENTATION AND ANALYSIS OF DATA......................................84 Data Collection for the Pilot Study....................................................................................87 Pilot Study Results.............................................................................................................88 Pilot Study Group ..............................................................................................................95

viii Main Study Results ............................................................................................................96 Data Collection ..................................................................................................................96 Round 3 Results .................................................................................................................97 Research Question 1 ....................................................................................................98 Research Question 2 ..................................................................................................106 Summary ..........................................................................................................................109 CHAPTER 5: SUMMARY AND RECOMMENDATIONS ..........................................111 Overview of the Study .....................................................................................................111 Conclusions......................................................................................................................114 Research Question 1 ..................................................................................................115 Research Question 2 ..................................................................................................125 Design and Development.................................................................................................126 Acquisition.......................................................................................................................127 Product Reliability ...........................................................................................................127 Repair...............................................................................................................................128 Assumptions, Scope, and Limitations and Delimitations of the Study............................129 Recommendations............................................................................................................133 Operations Management ............................................................................................134 Marketing...................................................................................................................135 Finance.......................................................................................................................135 Academia ...................................................................................................................136 Framework and Application ............................................................................................136 Future Studies ..................................................................................................................141

ix REFERENCES ................................................................................................................143 APPENDIX A: LETTER OF INFORMED CONSENT .................................................161 APPENDIX B: TOTAL OWNERSHIP COST ASSESSMENT KEY PERFORMANCE FACTORS........................................................................................................................164 APPENDIX C: MAIN STUDY ROUND 3 RESULTS ..................................................165 APPENDIX D: MAIN STUDY ROUND 3 FRIEDMAN NONPARAMETRIC STATISTICS TEST RESULTS ......................................................................................174 APPENDIX E: TOTAL OWNERSHIP COST FRAMEWORK CHECKLIST EXAMPLE180

x LIST OF TABLES Table 1 Criteria for the Identification of Complex System Total Ownership Cost Experts74 Table 2 Demographics of Experts in the Pilot Study .........................................................85 Table 3 Demographics of Experts in the Main Study ........................................................86 Table 4 Best Practices and Characterization ..................................................................138 Table 5 Sample Total Ownership Cost Trade off Analysis ..............................................140

xi LIST OF FIGURES Figure 1. Classical product life-cycle model.....................................................................34 Figure 2. Complex system life-cycle model......................................................................35

1 CHAPTER 1: INTRODUCTION The goal of the rational consumer is to make the most practical purchase of available products that will perform their function when needed and can be operated cost effectively. This assumes that the buyers perception of the value and benefits received from the product is driving the purchase. Gordon (2004) reported consumers purchasing factors are not always based on rational analysis, but can be influenced by the perception of a product, by beliefs and attitudes toward the product, and by the manufacturer or the vendor. Consumers purchase decisions should extend beyond the initial purchase price and influences of the brand name, product features, and functions. Very often, both the enterprise and the customer limit their decisions to the purchase price (Kothari and Lackner, 2005). The total cost of ownership of a product is the sum of both direct and indirect cost over the entire life cycle. It is a common metric used to evaluate capital investments in many industries, and even for consumer purchases. The total cost of ownership is an important consideration because purchase price alone does not provide a complete picture of cost (Padnos, 2006). The consumer often finds the product that is the cheaper choice to buy may cost more to own during the lifetime of the product. Kaye, Sobota, Graham, and Gotwald (2000) noted understanding the impact of total cost of ownership is critical. The authors noted, The greatest challenge is the need to make decisions based on future impacts to break the paradigm of continuously mortgaging the future when faced with the reality of the critical exigencies of today (Kaye et al., p. 367).

2 The application of total cost of ownership strategies is not limited to the consumer. In the aerospace industry, the impact of the total ownership cost of a product can be partitioned into four categories: the costs of research and development; the procurement; the operation, maintenance, and support; and system disposal. The concept of total cost of ownership (TCO) is the development of an understanding of the true cost of doing business with a particular supplier for a particular good or service (Ellram, 1994). Total cost of ownership requires a rather complex approach to the purchasing process. The buying firm must determine which costs it considers to be most significant in the acquisition, possession, use and subsequent disposal of a good or service (Ellram, 1995). Aerospace industry manufacturers that use total cost of ownership strategies for purchasing and supply chain management may gather data and make critical procurement decisions beyond the initial purchase price. The decisions include trade-off analysis of significant cost drivers (Ferrin & Plank, 2002). Background of the Problem A goal of many organizations is to provide the greatest profit for stakeholders for the least capital investment. To develop or maintain competitive advantage in the marketplace, many organizations strive to delight the customer by providing innovative cost-effective solutions to meet their needs. One selection criterion for the consumer may be to receive the maximum benefit from the purchased product for the least total ownership cost. When the consumer does not know the total cost of ownership, the procurement cost is the primary, and sometimes the only, selection criterion for making the purchase. The complex system manufacturer that ensures the customer gets the most use of a product for the least total ownership cost may not know how the customer will

3 view the information in a purchasing decision. With no clear understanding regarding the value of exceeding customer expectations of total ownership cost, there may be little or no incentive for the manufacturer of complex systems to expend the resources or the time necessary to manufacture a product that does little more than meet minimal customer requirements. Providing the consumer with innovative and effective total ownership cost solutions to meet their needs requires a continuous assessment of company priorities and subsequent trade-off decisions. In many cases throughout the life cycle of a product, the price-competitive company strives to find the balance between delighting the customer and maximizing profits, thereby ensuring continued growth through referrals and repeat business while also maximizing profits. Adopting total ownership cost strategies as a strategic marketing philosophy may provide the complex system manufacturer with the data necessary to make cost-effective trade-off decisions. As of the time of this research in 2006, there is no known framework for objectively guiding decision makers in the use of total ownership cost strategies for competitive advantage. The manufacturer of complex systems is a consumer of purchased products. Suppliers are often used to provide components of the system that may be as small as individual parts or as large as complete assemblies. A supply chain management procurement valuation process that includes total ownership cost examines cost from a long-term perspective, considering more than the initial purchase price. The company that considers the total profit life cycle of a product plans for research and development, production, marketing, aftermarket support, and disposal (U.S. Department of Defense [DoD], 2003b). A total cost of ownership model may provide an organization with

4 strategies for decision making regarding supplier selection, evaluation, and performance measurement. Some of the primary benefits of adopting a TCO approach are that it provides a focus and sets priorities regarding the areas in which supplier performance would be most beneficial. It also provides a consistent supplier evaluation tool, thereby improving the value of supplier performance comparisons among suppliers. Over time TCO helps clarify and define supplier performance expectations and creates opportunities for cost savings, as well as supporting continuous supplier improvement (Bhutta and Huq, 2000). The U.S. Defense Acquisition System was developed to ensure the effective management of resources necessary to support current and future security needs. The DoD published Directive 5000.1 (DoD, 2003a), which reported, The primary objective of Defense acquisition is to acquire quality products that satisfy user needs with measurable improvements to mission capability and operational support, in a timely manner, and at a fair and reasonable price (para. 4.2). The focus of the acquisition process is to meet the end users need for technologically state-of-the-art systems that are effective, affordable throughout their entire life cycle, and available when needed. The DoD Directive 5000.1 indicates all system acquisition will follow the mandatory policies and procedures for managing all acquisition programs as provided in DoD Instruction 5000.2 (DoD, 2003b). The assessment criteria in DoD Instruction 5000.2 direct procurement decision makers to consider the entire system life cycle and ensure the acquired system meets operational requirements and the key parameters of cost, performance, and schedule.

5 The DoD acquisition process begins when a clearly identified and defined end users need or goal is validated and assessed. After the need is identified and assessed, a request for information is developed and provided to prospective suppliers. The potential suppliers may be selected as sole-source providers or the decision maker may determine any potential supplier may openly provide a response showing an ability to meet the need. The request for information will be used in the assessment of available technologies and innovations that may fill the current or future needs of the end user and meet the key assessment parameters. Following the request for information, the procurement decision maker may send prospective suppliers a request for proposal. The request for proposal will normally contain a statement of work, the required system performance specifications, an outline of deliverable items, and the terms and conditions of an agreement or contract. The prospective supplier will normally provide a response in the form of a bid to provide the required system and meet the key performance and assessment parameters. The decision maker will assess the response from prospective suppliers based on the weighting of key parameters that include meeting the required system specifications, cost, schedule, and risk. The common practice for the procurement decision maker was to award the contract to the lowest bidder until the implementation of DoD Instruction 5000.2 (DoD, 2003b). In a DoD white paper, AMTSybex (2004) noted, The need to achieve an equitable return on asset expenditure forces the choice of a solution that meets the value for money requirement at the lowest through-life cost, as opposed to a traditional leaning towards the cheapest purchase price (p. 5).

6 Following procurement, the DoD program manager assumes responsibility for the system throughout the remainder of the product life cycle. The expense of maintaining a system in operation once it has been fielded may not be obvious. Barringer and Weber (1996) reported the smallest amount of cash that will be spent on a system is during acquisition and further suggested that 65% expended in sustaining, operation, and support may be used as a heuristic of the total life-cycle cost of a system. The U.S. Air Force established a total cost of ownership reduction initiative in 1997 to lower these operating support and sustaining costs. The goal of the initiative is that the Air Force can expect avoided costs and savings to exceed $3.4 billion by 2009 (Williams and Graveline, 2000). Despite the instructions in DoD Instruction 5000.2 (DoD, 2003b), which clearly outline the responsibility of the decision maker to optimize total system performance and minimize total ownership costs (p. 32), Williams and Graveline (2000) provided several factors that inhibit the implementation of the strategy. Their report to the U.S. General Accounting Office indicated program managers have poor visibility and few incentives to reduce the operating and support cost of fielded equipment and, therefore, reliability, supportability, and affordability improvement initiatives are not priorities. Although the military must be concerned with the initial purchase price of a system, in the commercial aerospace sector the hidden cost is a much greater concern. The fixed cost of operating an airline includes buildings, payments, and maintenance; lease agreements; insurance; and loan interest. Airlines can lease aircraft and pay for airport privileges as needed, rather than face the capital expenditure for infrastructure. Hidden costs amount to much more because the costs include labor, fuel, maintenance, administration, and passenger services. Cubbin (2004) noted, The capital investment in

7 airways and airport infrastructure is borne by governments. As a consequence airlines have lower fixed costs, but higher variable costs (The Nature of Costs, para. 2). The cost of operating the airline is borne by the passenger or the freight. Airlines pay to operate the aircraft regardless of the number of seats filled per flight. As a result, the common practice in the aerospace industry is to report cost of operation by the available seat miles (Cubbin). The high cost of initial aircraft procurement is overshadowed by the hidden cost of maintenance and operation. Butterworth-Hayes (2002) reported the annual cost of maintenance repair and overhaul for the American airline industry was estimated at $37.8 billion in 2002. The American Airlines (2005) annual report provided a net corporate loss with the operation cost of the combined fleet of 699 American and 302 American Eagle aircraft of $0.105 per seat mile. The Continental Airlines (2005) annual report provided a net corporate loss for the combined fleet of 356 aircraft and reported the operating cost of an aircraft seat mile was $0.1015. For the same period, Southwest Airlines (2005) reported an operating profit and reported the operating expenses for their fleet of 445 aircraft per seat mile was $0.0794. Many variables may be assessed to determine the cost of operating an aircraft seat mile. The variables that may be used in the assessment of the total ownership cost of a complex system are different for each airline operator. The commercial aerospace industry adopts many DoD strategies and initiatives in the life-cycle management of complex systems. The development of a framework of total ownership cost strategies to control and reduce life-cycle costs of a complex system may assist the commercial aerospace decision maker with procurement and life-cycle management.

8 Despite the apparent advantages of adopting total ownership cost philosophies, few organizations employ a total cost model or framework. Surveys have shown how few of the respondents actually use total ownership cost, although it is certainly not a new concept (Veenstra, 2000). Beaudreau and Naegle (2005) reported the DoD regularly make procurement decisions without regard for the total ownership cost, but understands the strategies are so critical that DoD would pay nearly anything to have itfor a while at least (p. 109). A framework is necessary that will provide the decision maker with a tool for making continuous assessment of priorities and subsequent trade-off decisions to optimize total ownership cost. Problem Statement Leaders in the military and commercial aerospace industry must plan for total lifecycle management of complex systems. Despite the benefits that may be realized in the industry from utilizing total ownership cost strategies, product cost or life-cycle cost considerations are an afterthought for many organizations, including the military (Crow, 2004). Although there is a potential to save billions of dollars, total ownership cost analysis are not applied very widely causing some experts (Bailey & Heidt, 2003; Beaudreau & Naegle, 2005; Hall, 2005; Stundza, 2006) to determine there is a profound need for military and commercial aerospace manufacturing and procurement decision makers to implement total ownership cost strategies to control and reduce life-cycle costs of a complex system. A possible solution to correct the need for a leadership decision-making model is to identify, characterize, and organize available total ownership cost management strategies and key performance parameters that can be used in the aerospace industry. A

9 study that investigates complex system life-cycle management by group position consensus using a modified Delphi method could remedy the situation. The current study included opinions of experts such as program and project managers and leaders in the military and commercial aerospace industry. Participants in the study necessarily had special knowledge or expertise in the area of complex system life-cycle development and total ownership cost strategies. Purpose of the Study The purpose of the qualitative research study was to develop a framework of best practices for controlling and reducing the total ownership cost of complex systems using a modified Delphi design. The panel consisted of 23 decision makers who had special knowledge and expertise in the area of complex system life-cycle development. Data were obtained to synthesize a framework that identifies, characterizes, and organizes methods for managing total ownership cost of complex systems throughout the entire life-cycle process. The framework will be shared with decision makers in procurement and manufacturing organizations in the south central region of the United States. The data will be made available to other developers and manufacturers in both the military and the commercial sector, including the National Defense Industrial Association (NDIA), who could likely benefit from the information. Significance of the Study The research study provided decision makers with a framework that identifies, characterizes, and organizes total ownership cost strategies across the life-cycle phases of a complex system. The framework provided the decision maker with a course of action for the development and implementation of total ownership cost methods. The methods

10 could assist in the procurement and manufacture of complex systems by providing best practices for controlling and reducing total ownership cost. Significance of the Study to Leadership Both military and commercial aerospace leaders may use the framework throughout the life cycle of the complex system. The framework of best practices may provide decision makers with trade-off alternatives during the research and initial concept development. During the research stage, the viability and risk associated with using current technologies or in the development of new innovations may be assessed. The selection of potential suppliers may be based on past performance as measured in key areas of concern and on total cost of ownership, rather than on the initial purchase price. In the operation and support stages of the complex system life cycle, decision makers may use the framework to select among available maintenance concepts, repair philosophies, the necessity, number, and position of spare parts, tools and test or repair equipment, training and publications, storage and facilities, and data gathering. The cost of disposing of a complex system at the end of its useful life may include such considerations as environmental impacts and the potential reclaiming of precious metals. The framework may provide decision makers with total ownership cost strategies and alternatives to consider for the reduction of cost and risk and an improved competitive advantage. Nature of the Study The qualitative modified Delphi study involved the reliable and creative exploration of ideas with the intent of gathering suitable information for decision making. The Delphi method is based on a structured process for collecting and distilling

11 knowledge from a group of experts by means of a series of surveys interspersed with controlled opinion feedback (Gunaydin, 2006). The expert panel for the study consisted of a diverse group of 23 experts in complex system life-cycle development. Dalkey and Helmer (1963) developed the Delphi technique while performing investigative research of group opinion for the U.S. Air Force project RAND. The primary objective of a Delphi study is to make discussion between experts possible without permitting a certain social interactive behavior, as happens during a normal group discussion, which hampers opinion forming. Decision makers often rely on their own intuition or on expert opinion, such as consultants, when full scientific knowledge is lacking. The Delphi method is widely used to generate forecasts in technology, education, and other fields. The Delphi technique may be compared to a multi-step brainstorming session. The facilitator brings together a group of knowledgeable individuals on the subject of interest to seek their opinion about the future. Delphi research begins with the use of a questionnaire requesting a response from the panel of experts. Ludwig (1997) cited Weaver (1971) as stating, Delphi operates on the principle that several heads are better than one in making subjective conjectures about the future . . . and that experts will make conjectures based upon rational judgment rather than merely guessing (Introduction, para. 1). The participation of a qualified panel of experts is essential to a modified Delphi study. Delphi groups tend to outperform both standard interacting groups and statistical groups (Rowe and Wright, 1999). Murphy et al. (1998) recommended, To define common ground and maximize areas of agreement, groups should be homogeneous; to identify and explore areas of uncertainty, a heterogeneous group is appropriate (p. 50).

12 Pollard and Tomlin (1995) noted 20 to 50 individuals should be members of the panel of experts in a modified Delphi study and members should be told how much time and effort are expected of them prior to participating. The Delphi technique is an iterative process in which individuals have the opportunity to change their minds or add new questions and comments in the next round of questions. The facilitator provides feedback to the members of the expert panel that shows the statistical distribution of previous responses. The process provides group members with the ability to see how their opinion relates to that of the group of experts and then re-evaluate. Participants can align more closely to the group opinion by changing their responses to the questionnaire or may choose to hold their ground based on their individual judgment. Members of the panel of experts are free to change their individual responses, so the result is a statistical summary of the consensus of the group. As technology advances available methods for the development and manufacture of complex systems, decision makers must plan for total life-cycle management. To develop a framework that has broad applicability to complex system procurement and manufacture, the framework must incorporate a robust collection of diverse methods that accommodate a wide variety of complex systems. The framework developed in the modified Delphi study may characterize and organize available and applicable methods for guiding total ownership cost decision-making strategies across the entire life cycle of a complex system. The framework will facilitate a comparison of existing complex system procurement, development and manufacturing practices to those utilizing total ownership cost strategies. With this comparison, the decision maker may be better prepared to

13 determine between cost, schedule, performance, and competitive advantage alternatives. The framework will facilitate an effective approach for the application and implementation of complex system management using total ownership cost strategies throughout all life-cycle phases. Research Questions The qualitative study, using a modified Delphi method, was conducted to develop a decision-making framework for the total ownership cost management of complex systems. The study determined a core set of cost drivers and key performance factors and provided barriers and incentives to implementation of total ownership cost philosophies over the entire product life cycle. The framework will provide decision makers with an understanding of the advantages and disadvantages of implementing total ownership cost strategies. The following research questions guided the study: 1. What are the best practices for controlling and reducing the total ownership cost of complex systems? 2. What are the key performance parameters in the development of a future complex system total ownership cost framework? Theoretical Framework Total cost of ownership includes the direct or actual cost associated with the purchase price, the cost of equipment, and the parts needed to keep it operating. In addition to the direct cost, indirect or hidden costs may include the cost associated with the operation, support, and disposal of the product. Total cost of ownership can be used to discover the hidden costs, as well as the obvious costs, of conducting business with different suppliers (Hurkens, 2006). Much research is dedicated to the effective

14 management of the total ownership cost that results from acquisition and the supply chain (Bailey & Heidt, 2003; Beaudreau & Naegle, 2005; Bhutta & Huq, 2000; Crow, 2004; Ellram, 1993, 1994, 1995; Hurkens et al., 2006; Stundza, 2006). However, little research existed regarding total ownership cost best practices, cost drivers, and key performance parameters (Ferrin & Plank, 2002; Gartner, Inc., 2006; Milligan, 1999). The functional components of a complex system must interact with each other to perform the desired operation as a whole. In the aerospace industry systems engineering plays a key role in the development of effective total ownership cost strategies throughout the entire life cycle. Systems engineering is a discipline that is concerned with the effect of all system components as they affect the product life cycle from system design, operation, support and performance, cost, and schedule (International Council on Systems Engineering [INCOSE], 2003). The systems engineering focus is the transformation of customer requirements and applicable technical factors across the entire product life cycle to optimize functionality and interoperability of system components. The systems engineering processes and systems engineers act as the technical glue that holds the various design disciplines and subsystems functions together to provide an integrated system that performs a specific job (Kludze, 2004, p. 40). Systems engineering processes are a guide in the design, development, operation, support, and eventual disposal of complex systems. The processes are used to effectively optimize the relationship among cost, schedule, and performance based on the requirements and expectations of the stakeholders (INCOSE, 2003; Kludze, 2004; NASA, 1995).

15 When comparing the purchase price of systems that will perform a given process, assessment of the direct cost is relatively uncomplicated. Obtaining an accurate estimate of the indirect cost of the system throughout the entire life cycle is much more difficult to quantify. Whether the customer is military or commercial, the indirect costs make up a significant part of the total cost of ownership. Total ownership cost takes into consideration all costs of an individual system, including the research, development, procurement, operation, logistical support and disposal associated with the system. It also includes the total supporting infrastructure that plans, manages and executes that system over its full life (The U.S. Naval Air Systems Command, 2003). The variables to be considered often include the cost of training, setup, reliability, maintenance, and environmental considerations for disposal. Over the life of a system the indirect costs can far outpace the initial purchase price or direct cost. In a presentation for the U.S. Naval Sea Systems Command, Louden (2006) presented data indicating the average cost of research and development is 2%, acquisition is 34% and operating and support is 64% (p. 24) of the total ownership cost of complex systems. The indirect costs of the product associated with operation and support are dependent upon such variables as quality, reliability, and ease of maintenance. The indirect costs make up the largest share of the total cost of ownership over the life of the system. Based upon the results of the research, a core set of cost drivers and key performance factors were determined. Research studies indicated a potential in the aerospace industry to save billions of dollars by adopting total ownership cost philosophies (Cubbin, 2004; Hurkens et al. 2006; Williams & Graveline, 2000). Despite the benefits that may be available, few firms

16 implement the strategies (Beaudreau & Naegle, 2005; Hurkens et al.; Veenstra, 2000). The barriers to implementation are attributed to a lack of understanding and poor communication of the value of the total ownership cost concept and, in many cases, data and information needed to make effective decisions are not available (Ellram, 1993, 1994, 1995; Ferrin & Plank, 2002; Gartner, Inc., 2006; Milligan, 1999). The results of the study will provide the decision maker with a decision-making framework, incentives, and advantages to implementation of total ownership cost strategies. Definition of Terms The glossary of telecommunications terms found in the Federal Standard 1037C (National Communications System, 2000) was used as a source of definitions for the study. The federal standard provided the following definitions of terms: Complex system: Kirshbaum (2002) provided a definition of the complex system: Any system that involves a number of elements, arranged in structure(s) which can exist on many scales. Complex systems theory also includes the study of the interactions of the many parts of the system (Introduction to Complex Systems Theory: Basic Definition, para. 1). Tesfatsion (2004) defined a complex system: A system that is composed of interacting units (components, primitive elements, constituents, ), and exhibits emergent properties, i.e., properties arising from the interactions of the units that are not properties of the individual units themselves (p. 3). Cost driver: Geiger (1999) defined a cost driver as Another measure that is used to proportionally distribute the cost of activities to cost objects (p. 33), whereas Whittaker (2005) claimed a cost object simply is an activity, output, or item whose cost is to be measured (p. 7).

17 System: 1. Any organized assembly of resources and procedures united and regulated by interaction or interdependence to accomplish a set of specific functions. 2. A collection of personnel, equipment, and methods organized to accomplish a set of specific functions (National Communications System, 2000, System). System design: 1. A process of defining the hardware and software architecture, components, modules, interfaces, and data for a system to satisfy specified requirements. 2. The preparation of an assembly of methods, procedures, or techniques united by regulated interaction to form an organized whole (National Communications System, 2000, System design). System life cycle: The course of developmental changes through which a system passes from its conception to the termination of its use and subsequent salvage. For example, a system life cycle might include the phases and activities associated with the analysis, acquisition, design, development, test, integration, operation, maintenance, and modification of the system. (National Communications System, 2000, System life cycle) Assumptions The study was based on the following assumptions. First, the study assumed there was value added for the military and commercial leader in the development of a framework of total ownership cost strategies and applications throughout the entire life cycle of complex systems. Next, it was assumed no universal framework would be applicable in all applications, but rather a set of best practices may aid decision makers in trade-off analysis and in identifying complex system total ownership cost key performance parameters. For the purposes of the study, a complex system has many

18 individual parts that are coupled to form a system. The theorists and practitioners who participate in the study will be considered experts by their peers. Scope and Limitations The scope of the study will be limited to the development of a decision-making framework that may be used in the military and commercial aerospace industry. The framework will include the entire life cycle of complex systems and will provide strategies for the implementation of a total ownership cost approach, based on key performance parameters. Although customer relationships and perceptions, as well as the organizational emphasis on the cost of money and overhead, may be considered cost drivers in the assessment of total ownership cost, they are regarded as outside the scope of the study. The study will focus on future best practices that may be used in the development of a total ownership cost framework for complex systems that are or may be used in the military and commercial aerospace industry based upon the opinions of a panel of experts. Four limitations are outside the control of the researcher. The first limitation is the best practice. For the purpose of the study, a best practice will be considered the most desirable and most useful total ownership cost practice or strategy as defined by the panel of experts. The second limitation is the complex system. For the purposes of the study, a complex system is a system that performs a function or set of functions in interaction with other systems in the overall performance of a process (Kirshbaum, 2002). As a representative example, an aircraft engine controller is a complex system that must receive information and data from a number of sources or systems to perform the task of optimizing engine performance. The third limitation is in the aerospace application. This

19 study was confined to complex systems used in military and commercial aerospace applications. The focus included commercial airlines and military aviation. Total ownership cost strategies may be applied for decision making by the consumer. The purchase of a vehicle may be based on factors that include depreciation, financing, insurance, taxes and fees, fuel, maintenance and repairs (Reed, 2002, para. 2). The fourth limitation is the experts. The experts in the study included individuals who possess the skills, knowledge, and experience in the development and procurement of complex systems used in the commercial or military aerospace industry to be considered influential leaders and decision makers by their peers. The experts included theorists and practitioners and focused on bridging the gap in the development of a framework of strategies and applications. Delimitations The study was confined to the inquiry of information through a modified Delphi method by working with expert practitioners in the aerospace industry. The study focused on the development of a framework to be used by decision makers for effective implementation of total ownership cost strategies over the entire life cycle of complex systems. The benefits and the barriers to implementation, the critical cost drivers, and key performance parameters were considered in the development of the framework. The experts were selected based upon their special knowledge, skills, expertise, and experience in the development of strategies for total ownership cost of complex systems in the aerospace industry. The sample group of experts was nominated, identified, and selected from a larger population. The experts were asked to share their ideas about the effective implementation of total ownership cost strategies and to focus

20 on the strategies that will now, and in the future, support the development of a decisionmaking framework. The study was confined to complex systems used in military and commercial aerospace applications. The focus included commercial airlines and military aviation. Summary Chapter 1 provided the need for total ownership cost strategies throughout the life cycle of complex systems and the benefits of adopting a total ownership cost philosophy were addressed. Complex system total ownership cost is the sum of direct and indirect costs associated with the system throughout the life-cycle stages, including defining the need and initial planning, development and production, servicing and maintenance, support, and disposal of the product. The initial purchase price does not always provide the full picture because the product that is the least expensive to purchase may cost more to own. According to the DoD, the cost of operating, supporting, and maintaining a complex system once it is in the hands of the customer is the greatest life-cycle expense. Total ownership cost strategies are not limited to the consumer purchase decision. In addition to evaluating between potential suppliers and maintaining historical supplier performance records, the complex system manufacturer may use total ownership cost strategies to develop a strategic marketing advantage. Strategic decision making over the complex system life cycle by both military and commercial aerospace decision makers may include analysis of total ownership cost for supply-chain management. Commercial aerospace decision makers may use total ownership cost strategies to reduce the operating cost per seat mile. Cash flow advantages and savings that amount to billions of dollars are cited as the result of using total ownership cost strategies reported by Williams and

21 Graveline (2000) in an Air Force auditing team white paper to the U.S. Senate. Despite the advantages, the literature indicated implementing total ownership cost philosophies is not a widespread initiative. Framework for effectively implementing total ownership cost philosophies over the entire life cycle of military or commercial aerospace complex systems must be developed. It is not anticipated that any one total ownership cost strategy will work universally because the key performance parameters are different for each application. The study provided a framework decision makers can use to identify, characterize, and effectively implement a total cost of ownership philosophy that will aid the decision maker in procurement and life-cycle management decisions. Chapter 2 provides a review of the literature that highlights the key elements used in assessing total ownership cost of military and commercial aerospace complex systems. Chapter 3 details the modified Delphi method, first developed in the 1950s for the DoD by the RAND Corporation as a research method for collecting the opinion of a panel of qualified experts and thereby predict future needs. Chapter 4 provides the data, results, and findings of the study. Chapter 5 provides conclusions, implications, and recommendations of the study that further support the development of a framework of total ownership cost strategies and key performance parameters.

22 CHAPTER 2: LITERATURE REVIEW The purpose of the qualitative study using a modified Delphi method was to develop a decision-making framework for objectively guiding the total cost of ownership of complex systems over the entire product life cycle. In the development of a complex system in the commercial or military aerospace industry, decision makers may choose from a number of available methods for meeting or exceeding the customers requirements for products on time and within budget. The decision to procure or manufacture products and complex systems is often based predominantly on the initial purchase price. When the total cost of ownership is not known, the procurement cost is often the primary, and sometimes the only, selection criterion for making the purchase. The framework developed as a result of the study may provide military and commercial aerospace industry decision makers with total cost of ownership strategies and best practices. The framework may be used for determining a comprehensive costbenefit analysis, control, and supply chain management of complex system life-cycle cost. A review of the available literature provided insight into total cost of ownership strategies as they apply to complex system life-cycle management and included an overview of the studies accomplished to date. Gaps exist in the literature and, as Ferrin and Plank (2002) noted, The scholarly literature on total cost of ownership consists mostly of case study and anecdotal data (p. 20). The literature was reviewed, assessing the advantages, disadvantages, key performance parameters, and barriers to the development and implementation of total ownership cost strategies. Additional current sources were sought and utilized as the research continued.

23 Title Searches, Articles, Research Documents, and Journals A preliminary search of available resources dealing with total cost management of complex systems was conducted. The search was followed with a basic analysis of the subject matter to determine significances and contributions of the key factors that contribute to the development of total cost of ownership throughout the life cycle of complex systems in the commercial and military aerospace industry. The topic search included theoretical concepts and practical applications of leadership decision-making methods for purchasing, managing suppliers, and the supply chain as well as for the development of potential strategic product advantage from utilizing total ownership cost philosophies. The research included peer-reviewed Web-based articles and magazines, texts, dissertations, open forum discussions, symposiums, and audio and video conferencing. Electronic sources available from the University of Phoenix Online included ProQuest, EBSCOhost, InfoTrac, and ProQuest digital dissertations. Keyword searches included word combinations of total ownership cost, life-cycle cost, and cost as an independent variable. Internet searches were performed using commercial search engines, such as Copernic Agent, Google, Yahoo!, and Dogpile, using the same keyword combinations. Searches using Refdesk, Questia, and Google Scholar online library yielded results similar to that of the University of Phoenix Online sources. Research and references were collected through both electronic and conventional means. Different combinations of keywords and phrases were used to stimulate the search to collect as many academic sources as possible. Although the search for primary source literature was not exhaustive due to the infinite amount of resources that can be

24 retrieved from the World Wide Web, the results provided in the study serve as the basis. A review of past and present studies suggested total ownership cost is not a new concept, yet despite the advantages, few complex system developers utilize the strategies. Total Ownership Cost Total ownership cost may be defined as the entire life-cycle cost incurred to research, develop, procure, operate, maintain, and dispose of a system or subsystem. Total cost of ownership, life-cycle cost, and total cost to own are related concepts (Ferrin & Plank, 2002). The analysis of the total cost of product ownership provides the decision maker with an alternative approach to product procurement that is based on initial purchase price. Total ownership cost and life-cycle cost analysis have been in development by the military since the late 1960s. During the 1960s the DoD determined the initial cost of product procurement was the primary consideration for system acquisition. The lack of planning and analysis of the total cost of the system resulted in support costs that far exceed the initial acquisition cost (Kaminski, 1995). With the rapid improvements in technology, the cost of the logistics resources necessary to support the systems grew substantially. The growth of life-cycle cost analysis was driven by the necessity to reduce the total ownership cost of a system by effectively planning for every phase, or the cradle-to-grave acquisition, of a product (Kaminski, 1995, p. 2). The military has long studied the cost of systems throughout the life cycle. In 1975, the Canadian Department of National Defence initiated a number of system studies and commissioned Bell Northern Research to develop a life-cycle cost model. The Canadian Treasury Board directed, Life cycle costs were to be the determining factor in

25 all government procurement (Material Management Training Center, 1993, p. 2). During the same time period, the United States DoD developed a number of references or military handbooks as guides in the implementation of life-cycle cost, including MILHDBK-259 for life-cycle cost details and MIL-HDBK-276-1 and MIL-HDBK-276-2 as form guides and instructions for importing data into specific computer software programs, mandating total life-cycle cost systems management in the DoD Acquisition Guide, DoD 5000 subsection 4.5, Effective Management. Many automobile owners realize the initial cost of acquiring a vehicle may soon be overshadowed by the cost of financing, taxes, fuel, insurance, maintenance and repairs, and fees. MacMillan and Vella (2007) provide research of the potential total ownership cost savings available from hybrid vehicles. A consumer can determine the total ownership cost of a vehicle by going on the internet to various websites. Edmunds Inc. (2006) has developed a research tool that provides, by ZIP code, the estimated 5-year costs of automotive expenses for vehicles. Assuming a person owns a vehicle for 5 years and drives 20,000 miles per year, and using data available on the Edmunds Web site for nine of the most popular models, the average cost to operate and support a vehicle is 45% of the purchase price. As with most products, the older an automobile is the more maintenance it requires. Hysom (1979) reported life-cycle cost is an effective tool when used to evaluate the total ownership cost of real estate, helping real estate investors to analyze the hidden expenses in a building (p. 332). Real estate investors are encouraged to use life-cycle cost modeling to deal with the uncertainties of such unknown variables and hidden expenses as the ever-increasing cost of energy. Hysom noted although life-cycle cost

26 analysis must be used prudently (p. 332), it is an important factor in considering the economic life of a real estate investment. Defining the Need for Total Ownership Cost A total ownership cost analysis changes the perspective for business from looking at the short-term cost advantages to an emphasis on the lowest long-term cost of ownership. Examining the total ownership cost associated with doing business is a process that, once implemented, takes into account not only the design and development of a product but also the procurement of purchased goods and services and all phases of aftermarket support. During the design and development phase of the product life cycle, engineering may perform trade-off analysis to reduce the long-term failure rate of the product and improve the manufacturability, testability, and ease of maintenance; the purchasing department may choose a better grade of equipment rather than a more favorable cost price. In the production phase of the product life cycle, the operations department may choose the long-term view of low-cost production over the short-term needs to meet flow-through commitments; process and quality engineering may choose to use Six Sigma, lean, or continuous improvement process methods to reduce cost and improve product quality. During the operation and customer support life-cycle phase there may be benefit from implementing preventive and corrective maintenance plans that consider total ownership cost rather than short-term management gains. The logistics support pipeline of spare parts, facilities, product packaging, handling, storage, and transportation considerations may also be taken into account to reduce the overall total ownership cost.

27 Ferrin and Plank (2002) performed exploratory study research in the availability of life-cycle cost and total ownership cost analysis models. Many leading companies used the models with a focus on value purchase opportunities but firms are unsure of their ability to effectively identify the critical cost drivers for estimating total cost of ownership (Ferrin & Plank, p. 24). Ferrin and Plank noted, The study suggests a generic model of total cost of ownership is not appropriate. However, the findings suggest a TCO model based on a core set of cost drivers, along with an auxiliary set of cost drivers, is appropriate (p. 18). No single solution will meet the needs for all total ownership cost models. Ferrin and Plank (2002) contended purchasing managers could use a core set of cost drivers and additional tailored drivers for computation in a particular purchase situation. Ferrin and Plank also noted, It is also suggested that a value-based, multi-firm, or supply chain TCO computation model is needed (p. 18). Total ownership cost strategies form the basis for economic analysis to systematically investigate the problem of choice and facilitate trade-off decision making. Hurkens et al. (2006) claimed, TCO can be used to think about cost at the strategic level; as such, a TCO model could be the starting point to redesign and make the supply chain more effective (p. 27). Total ownership cost strategies provide the decision maker with alternative means of satisfying an objective and a systematic method for investigating the costs and benefits of each of the alternatives. Ellram and Siferd (1998) examined total ownership cost as a key concept in strategic management decision making in a case study of 11 organizations. The case study provided a robust view of both internal and external organization cost management

28 to enhance competitive advantage. The case study determined manufacturers rely heavily on suppliers. Purchase items make up an average of 63.5 percent of total costs for manufacturing firms and 25 percent for non manufacturers (Ellram & Siferd, p. 55). Ellram and Siferds (1998) total ownership cost comparison of goods purchased from three different suppliers and the resultant analysis demonstrated the initial price of the product is not a good indicator of the total price and total ownership cost of product ownership. In this case, products available at the cheapest purchase price from a supplier resulted in the highest total ownership cost, whereas an alternate supplier had the highest initial purchase price but the lowest overall ownership cost. As a result of the case study, Ellram and Siferd concluded the corporate purchasing, sourcing, or procurement department should consider the cost of systems over the entire life cycle as well as the initial purchase price. Benefits of Total Ownership Cost Analysis The primary focus of total ownership cost is to look beyond the purchase price. In a research case study of nine firms, Ellram (1994) examined total ownership cost as used in the procurement process, including the initial idea or concept, design, development, suppliers, manufacturing, and the warranty claims associated with the final product once it is in use by the customer. The benefits resulting from the effective implementation of total ownership cost strategies in the procurement process are described as improved supplier performance measurement, improved purchasing decision making, improved internal and external communications, better insight and understanding into purchased goods/services and supplier performance, and support of the firm's continuous improvement efforts (p. 178).

29 In a meta-case study, Ellram (1994) found many different reasons and methods for using total ownership cost analysis. In some cases, the primary focus for total ownership cost analysis was the savings gained from continuous improvement efforts and reduction of the cost of poor quality. In other cases, companies used total ownership cost to support decisions regarding supplier and equipment selection. Decision makers should consider after market warranties, product reliability, customer support and the provision of spare parts necessary to make required repairs as the associated cost may be in the millions over the life of the product (Ellram, 1994). The common thread in all the cases studied by Ellram (1994) is all the companies agreed the benefits of TCO far outweighed the disadvantages and costs associated with TCO implementation (p. 3). Ahgren and Wierda (2007) found that the implementation of total ownership cost strategies provides the decision maker with effective methods for evaluating cost and comparing alternatives that reduce total net cost. Kilcourse (2007) posits that Total cost of ownership is the one right way to make any technology decision. Understanding the true cost to own any technology is especially important for retailers when making decisions regarding in-store technologies (p. 54). OHara (2007) reports that key technology decisions are being made that employ total ownership cost and return on investment analysis. Barriers to Implementation of Total Ownership Cost Despite the benefits that may be available from implementing TCO strategies, few firms use the analysis as a critical decision-making factor. Gartner, Inc. (2006) attributed the failure of organizations to communicate the effectiveness of TCO as a major barrier to implementation of the strategies. For a TCO evaluation to be most effective, the

30 results must be understood and communicated by an IS organization to members of all business units and must be expressed in understandable business terms (Gartner, Inc., 2006, The Benefits of Communicating TCO, para. 1). The availability and accessibility of the data necessary to calculate TCO are provided as a barrier to implementation. Milligan (1999) determined, Accurate total cost measurement is elusive, because most organizations either don't understand the calculations or don't have, or won't share, the data necessary for such calculations (p. 22). Ellram (1993) reported very few firms actually use total ownership cost modeling and few companies have the information or the reporting systems readily available to them to support such a goal (p. 3). The ability to easily calculate TCO is a significant barrier to TCO implementation. Meckback (1998) attributed the lack of implementation of TCO strategies to the fact they are hard to measure and often decision makers do not take into consideration factors such as user productivity, business benefits, and user satisfaction (p. 43). Barringer and Weber (1996) indicated the calculation of life-cycle cost is easy when you have the data, including such variables as the reliability failure rates and any maintenance required, operating and installation conditions, and costs for lost gross margin for outages of systems when appropriate. Ellram (1994) provided a number of barriers to the implementation of the analysis process, including cultural issues and the resistance to change, education, and training. The failure to educate the purchasing agent is cited as a barrier to implementation where the main focus is initial purchase price. There is little thought given to the overall cost of a system as the goal of the procuring agent is to buy systems at the lowest purchase price.

31 The resolution is to provide the purchasing agent with an understanding of the value of TCO, the key performance and cost factors and the tools necessary to make effective decisions. In early research, Hart (1978) encountered resistance to the adoption of a government-wide mandate to purchase products following a thorough life-cycle cost analysis. Hart reported, The problem, however, is a lack of expertise and an understanding of the benefits that can be achieved (p. 16). Hart encouraged the use of life-cycle cost analysis and incentives for private industry to produce more reliable systems and to provide more extensive equipment warranties. Total Ownership Cost Critical Cost Drivers Limited research existed regarding the variables that provide the critical cost drivers or the relationship to the application of total ownership cost strategies. A review of the available literature provided insight into many variables and key performance parameters that affect the total ownership cost of complex systems over their entire life cycle. The limited scholarly literature on total ownership cost and life-cycle cost consisted mostly of case studies, with very little empirical research of the cost drivers used in modeling. A total ownership cost analysis requires judgment on the part of the decision maker. The leader who is developing a total ownership cost model may consider a core set of cost drivers and appropriate cost elements that are dependent on the application and discard the elements that are not of substantial influence (Ferrin & Plank, 2002). A total ownership cost analysis may include both nonrecurring costs and recurring costs. Nonrecurring costs include the initial phases of the product life cycle, from defining the

32 need through development and production. The nonrecurring cost of the complex system to the customer is the initial purchase price. The recurring costs are invisible in the initial purchase transaction. The recurring cost of product ownership includes elements found in the remainder of the product life cycle: the total cost to operate and maintain the system, the cost associated with system unavailability and the safe and environmentally friendly disposal of the product after use. The recurring cost to the manufacturer may include the cost of warranty, repair and maintenance, and any required aftermarket customer support. While researching the methods used by purchasing managers, Ellram (1993) found the majority of the firms studied had a good grasp of how much time, effort, and expense is involved in adding suppliers to their systems and in placing orders (p. 6) and the managers know the value to their firm of on-time delivery, how much it costs to follow up on problems, match receiving with invoices, and even cut checks (p. 6). The same group of procurement managers failed to consider the significant cost associated with the product once it is in use. The significant cost of complex system ownership once it is in use was provided as considerably higher than the initial purchase price. Sun Tzu, a military general from the 6th century B.C., noted, As to government expenditures, those due to broken-down chariots, worn-out horses, armor and helmets, arrows and crossbows, lances, hand and body shields, draft animals and supply wagons will amount to sixty percent of the total (as cited in McNeilly, 1996, p. 179). The cost of maintaining a product, as in the time of Sun Tzu, often amounts to the greatest part of the product life cycle, 60 to 70% of the total ownership cost (Barringer & Weber, 1996; Kaminski, 1995; Louden, 2006).

33 In a U.S. Government Accounting Office report to Congress, Williams and Graveline (2000) clearly defined the high cost associated with the operation and support of complex military aerospace systems. The report provides the account of historical data indicating that operating and support is about 70 percent of a systems total life-cycle cost (Williams & Graveline, Introduction, para. 1). The cost of maintaining aging systems includes all of the spare parts, maintenance personnel, tools, training, facilities and logistics necessary to perform corrective and preventive maintenance to ensure operational availability. As systems age and more maintenance is required to keep them operational the cost of operating and maintaining is ever increasing. The life of some systems is extended beyond the initial design specifications. Replacement systems are required as aging systems wear out yet the increasing cost of support is depleting the funding available to replace those systems. This dilemma is described as a potential death spiral (Williams & Graveline, Introduction, para. 1). The Product Life Cycle Total ownership cost represents the true cost of product ownership, looking beyond the initial purchase price to include the entire product life cycle. A total ownership cost analysis may take into consideration all phases of the complex system life cycle, including identifying the need, research and development, production, initial operation during a warranty period, cost of using the item and the associated expense of products returned by the customer due to defects and failures, and the cost associated with product disposal. The actual acquisition of the product may only amount to approximately 35% of the total ownership cost, whereas operation may account for approximately 65% of the total cost of ownership.

34 The classical product life cycle is used as a method for assessing the phases a product passes through. The phases may be used to identify the many different challenges, opportunities, and problems faced by the product developer, marketer, or seller. Profits from the sale of products rise and fall throughout the life-cycle progression and different marketing, manufacturing, procurement, and resource planning techniques may be required. Many classical product life-cycle models are portrayed in a bell curve, typically divided into four stages: introduction, growth, maturity and decline (Kotler, 2000, p. 304). Many forms, shapes, and stages of the product life cycle are available and Kotler determined researchers have identified from 6 to 17 different product lifecycle patterns (p. 304). For the purposes of the study, the classical product life cycle is provided in Figure 1.
Develop Introduce Growth Maturity Sell the product Decline

Make the product

Figure 1. Classical product life-cycle model. The DoD acquisition management framework (DoD, 2003b) provides five lifecycle phases, including identification of the need, research and development, production, operation and support, and disposal. The Society of Automotive Engineers developed a life-cycle cost model focused on the manufacturing environment. The Society of Automotive Engineers (1995) model includes acquisition cost, operating costs, scheduled maintenance, unscheduled maintenance, conversion, and decommission or disposal. Research provided many product life-cycle models. The complex system life cycle model that is representative of this study is provided in Figure 2.

35
Research & Development Acquisition

Production

Operation

Support Use

Disposal

Figure 2. Complex system life-cycle model. Product Life-Cycle Management Product life-cycle management is a process for effectively managing a complex system or product and related services throughout the entire life cycle. Johnson (2005) reported, PLM [product life-cycle management] enables collaboration across disciplines, aiming to achieve technological interoperability and optimize processes (p. 14). Product life-cycle management is one method an organization may use to understand internal information needs and process flows and thereby determine where the available technologies can provide best value. Fraser (2005) posited product life-cycle management may be used as a strategy to provide a coherent view of a product from womb-to-tomb. This means having an accurate view of each product at each revision level, from concept to design to manufactured product, and on through service, maintenance, and retirement(p. 36). Product life-cycle management (PLM) is often described as a technology. Hakola and Horning (2004) contended that PLM is more appropriately described as a strategy for making companies more innovative and productive. Hakola and Horning (2004) posit that By applying a number of technologies PLM enables manufacturing companies to capture, use, and build upon the intellectual property created by design and manufacturing engineers, and to do so all the way from the concept of a product to the very end of its life (p. 26).

36 In the aerospace industry, product life-cycle management is an effective strategy for ensuring total ownership cost philosophies are integrated in the decision-making process. Mostefai, Bouras, and Batouche (2005) claimed, PLM is the business activity of managing an organizations products all the way across their lifecycle in the most effective way (p. 206). Product life-cycle management strategies may be used to educate relevant stakeholders about the value of a systems and life cycle perspective for decisions and decision-making processes, such as policy making, corporate strategy development, product design, production changes, purchasing and marketing (Saur et al., 2003, p. 2). The advancement of computer technology makes it possible to store in one place all complex system data that are or will be available throughout the product life cycle. Swink (2006) concluded, So far a comprehensive PLM system has not been developed. No single vendor has offered a complete solution for fulfilling all the functions of PLM (p. 37). The continued development of product life-cycle management may provide the complex system decision maker with an effective tool for reducing the total ownership cost of a product over the entire life cycle. Specifications and Requirements Development Complex system product development begins with the identification of a need. For many companies, identifying what they should create in the first place is the hardest question in developing new products and services (Korman, 2002, Avoiding Roadblocks to Innovation, para. 1). A clear and complete product definition is an essential aspect of any development project. Although it may appear obvious that until the product has been clearly defined the development should not begin, Crow (2004) noted that, despite the

37 importance, there are a number of common shortcomings to the process of product definition in many companies (Introduction, para. 1). Without clear specifications and requirements, the decision maker may be hard pressed to meet the customers needs and manage the total ownership cost. Customer Relationship Management Research and discussion are available on customer expectations and the marketing strategy to delight the customer. Delighting the customer and exceeding performance requirements is a marketing strategy used to encourage repeat business. Customers buy benefits, not features. Features are of value only if the customers can perceive them directly and if they really want them (Himmelfarb, 1999). Rust and Oliver (2000) defined delighting the customer as exceeding expectations by adding some utility to the product that is surprisingly pleasant (p. 87). The implication is delighting the customer heightens the expectation for repeat business, raises the customers expectations, and pulls customers away from the competition. Goel (1998) provided research in the challenges product development professions face when making quality, reliability, and durability engineering decisions. A study of customer behavior determined that product quality is based upon perception. Companies that provide high-quality products and services are at a competitive advantage in the marketplace. Those products that are perceived to have better quality than that of the competition may demand a premium price. The development of total ownership cost strategies provides an awareness of the need for effective cost management of products throughout the entire lifecycle. Rust and Oliver (2000) postulated, Critics have suggested that delighting the customer raises the

38 bar of customer expectations, making it more difficult to satisfy the customer in the next purchase cycle and hurting the firm in the long run (p. 86). Rust noted, Delight programs are likely to be profitable if customer satisfaction can be maintained at the higher expectation levels caused by assimilated levels of delight (p. 86). Ensuring optimal total ownership cost may delight the customer and provide a competitive advantage. Kothari and Lackner (2005) posited to achieve long-term growth, the organization should understand customers dont buy products or services. They buy valuethe total package of product performance, access, experience, and cost (Introduction, para. 1). An aerospace manufacturing firm may choose to delight the customer by providing the lowest possible total ownership cost, thereby increasing the value the customer perceives in the product. Consumers often develop a preference and loyalty for a brand or product category based on the perception of value. Providing a quality product and exceptional service will aid in the retention of customers (Rollo, 2006). Satisfying the needs of the customer must first begin with a clear identification of the requirements. Implementing strategies to provide customers with the lowest possible total ownership cost may be an effective method for maintaining the relationship. Acquisition and Procurement Early in the product life cycle, the total ownership costs of the complex system are greatly affected by the components used to manufacture the end product. The procurement function of ensuring the raw materials, components, and supplies necessary for the manufacturing process are available when and where they are needed requires the analysis of price, quality, and supplier delivery performance. Srinivas (2002) studied

39 multi-criteria life-cycle cost models used in the procurement process and determined the models provided managers with valuable information about the marginal costs of supplier outputs (p. 4) that can be used for negotiation with suppliers when making effective purchasing decisions by setting optimal output prices (p. 4). Carsten and Barry (2007) provide research of the procurement and engineering of pumps and control valves that demonstrate a shift from making purchasing decisions based on initial cost to total ownership criteria. By using total ownership cost in purchasing decisions total ownership cost strategies provide a competitive advantage to the firm. Decisions made during the acquisition phase can dramatically influence lifetime ownership cost (Carsten & Barry, 2007, p. 32). Supply Chain Management Most growing firms have important financial concerns and issues, the answers to which lead to success or failure. Managers make decisions on a daily basis that often have serious financial implications. Effective management of the cost, schedule, people, and material resources is vital for continued growth. The long-term business relationship, the supplier selection and evaluation process, and the development of performance criteria are essential cost drivers in the early development stages of the complex system life cycle. In the aerospace industry, for the development of complex systems, strategic cost management and supply chain management over the entire product life cycle are significantly influenced by the supplier selection and evaluating process. Theng Lau, Razzaque, and Ong (2003) noted, Owing to the complexity of the task and the risks inherent in it, the effectiveness of any organizational buying decision largely depends on

40 the information available for the decision-makers use (p. 82). The purchasing decisions and effective management of the supply chain within the organization are the end result of individual decision makers who may not consider total ownership cost criteria in decision making. The individual making purchase decisions is influenced by the information received and by personal bias. Bemmaor (1995) investigated consumer purchasing behavior and concluded individuals use rational thought before making a decision and multiplicity of influencing factors that influence such behavior could not be predicted (p. 176). Wilson and Woodside (1992) studied organizational and buying center purchase decisions and determined information gained from a variety of sources provided the rational individual making the purchasing decision with significant influence. At the same time the values of the individual are represented in the purchase decision. Wilson and Woodside demonstrated personal bias, including demographic and psychological traits, had an effect on organizational purchase decisions. Complex systems used in the aerospace industry often last 2030 years or more. As an example, the U.S. Air Force B-52 aircraft, first commissioned in 1952, continues to play a significant strategic role in the 21st century. In a case study, Smith and Zsidisin (2002) described how a firm concerned with the extended longevity of complex systems in the aerospace industry implemented target costing within the supplier selection process. Smith and Zsidisin noted firm product development often takes three to four years and total investments of $500 million to $600 million before returns are realized (p. 1). Leaders in the firm studied considered the long life cycle and the long lead time in

41 product development and delivery, making supplier selection a crucial activity and assessing product quality, reliability, and durability in the decision-making process. Significant organizational advantages are reported when supplier evaluation and supply chain management are considered early in the product development stage of the life cycle. Smith and Zsidisin (2002) reported, By engaging suppliers early in product design, the organization has recognized significant cost savings and enhanced its competitive position (Executive Summary). They further explained, These benefits include reducing costs, condensing product development times, and utilizing the technical expertise of supplier organizations (p. 5). Supplier-related cost is a major cost driver for many organizations, as is the cost of managing the purchasing organizations resources, skills, and technology. The factors that influence purchase decisions must be analyzed by the manufacturer. The proactive reduction of the total ownership cost of complex systems is strongly influenced by the selection of suppliers and the effective management of the supply chain. Design and Development For many organizations, the cost of production is the only cost encountered in the product life cycle. The cost of research and development may not be a consideration when a well-developed production package is available. The decisions made during the research and development phase of the product life cycle have a significant effect on the total ownership cost of the system or product. Swink (2006) posited, Some 70 to 80 percent of product lifecycle cost is fixed by decisions made in product design and development (p. 39). Many organizations employ systems engineers as programmatic and technical decision makers throughout the entire product life cycle.

42 Systems Engineering Systems engineering is a knowledge creating process that begins when a customer or sponsor defines an objective to be achieved. Scientific knowledge is put to practical uses and the output from the process is multilevel, highly precise, and highly consistent information which characterizes the objective (INCOSE, 2003). The role of the systems engineer is to work closely with the program manager to ensure the expectations of the customer and the stakeholders are achieved by the most effective and efficient use of available resources. Systems engineering functions are performed throughout the life cycle of a product as customer requirements are better developed and potential solutions are better defined. During the design and development phase of a program, a systems engineer represents the program manager in managing the programs technical effort (INCOSE, 2003, para. 2.1). The systems engineer plays a key role in the discovery and definition of customer requirements and in the development of an effective mix of cost, schedule, and performance based on the expectations of the stakeholders (INCOSE). To make effective decisions concerning the cost, schedule, performance, and reduction of total ownership cost of a system during project design or development, a systematic approach is required. INCOSE (2003) provided a list of pragmatic systems engineering principles that, when effectively implemented, may improve decision making: 1. Know the problem, the customer, and the consumer. 2. Use effectiveness criteria based on needs to make system decisions. 3. Establish and manage requirements.

43 4. Identify and assess alternatives so as to converge on a solution. 5. Verify and validate requirements and solution performance. 6. Maintain the integrity of the system. 7. Use an articulated and documented process. 8. Manage against a plan. (INCOSE, para. 2.1.4) The systems engineering process consists of the following seven tasks: state the problem, investigate alternatives, model the system, integrate, launch the system, assess performance and re-evaluate (INCOSE, 2003, para. 2.2). INCOSE suggested a number of steps to take in the effective implementation of the systems engineering process. The steps should include the definition of the systems objectives and user needs, performance of functional and requirements analysis to establish system functionality, and evolution of the design and operational concepts to meet user needs and expectations (INCOSE). Supportability One of the key performance parameters and cost drivers associated with the total ownership cost of a product is system supportability. Supportability analysis is integral to the design process. The DoD Directive 5000.1 (DoD, 2003a) reported supportability is a key performance parameter that will affect the success of a system. The DoD has mandated that supportability, product performance and life cycle cost should be considered when making cost effective decisions. Planning for the implementation of total ownership cost strategies should be set in place early in new program development to ensure cost effective operation and support. Supportability is not a primary system design consideration. It is dependent upon the availability of spare parts and the logistics of packaging, handling, storage, and

44 transportation. Supportability is defined as the time and cost necessary to provision for, and make available, the necessary logistics support during system operation to facilitate system maintenance (Schaeffer, 2003, p. 2). The elements of supportability include the logistics support necessary to ensure system operational availability. The supportability of a system is dependent upon policies and procedures and the evaluation of alternate design concepts. The analysis of supportability is conducted to ensure that the most cost effective means of maintaining the system over the entire product lifecycle is developed and implemented (Military definitions, 2004). The NDIA Systems Engineering Council provides systems engineers with guidelines to consider for improving total ownership cost while considering supportability concepts early in the design phase of the product life cycle. The council includes experts in the field of systems engineering who agree that supportability and total ownership cost strategies should be planned and implemented early in the development stage of the product life cycle. It is felt that the key to achieving a supportable DoD system is to design supportability in at the very start (NDIA, 2006, Supportability Factors, para. 1). Manufacturing Quality and Reliability Practices Manufacturing processes play a significant role in the quality, reliability, and total ownership cost of complex systems. Research was conducted on the role of quality management practices in manufacturing during the production phase of the complex system life cycle. Ensuring quality and reliability are included as design considerations leads to an increase in market share while lowering production costs and improving productivity, thereby reducing the total ownership cost of a product. Parasuraman,

45 Zeithaml, and Berry (1985) contend that product quality provides a company with comparative advantage by contributing to improved market share and return on investment as well as in lowering manufacturing cost and improving productivity (p. 41). Since most customers will no longer accept or tolerate average quality, high product and service quality have become standard expectations from vendors. Todays executives now view the task of improving product and service quality as their top priority (Kotler 2000). The consumer may compare the quality of similar products as part of the purchase decision. Seawright (1994) conducted research on the perception of the quality difference between Japanese and U.S. products in a study of the automotive industry. An important aspect of the research was no single definition of quality was used for comparison between products. Seawright noted, An analysis of the relationships between productbased quality and user- and value-based quality showed that, in the automobile industry, reliability has the strongest correlation with customer perceptions of both satisfaction and value (Abstract). Product Quality Management The importance of quality management practices in manufacturing was the subject of research by Solis-Galvan (1998), who reported a significant correlation between quality management and competitive advantage. In 1951, Japan implemented the Deming National Quality Prize, the Malcolm Baldrige National Quality Award was established in the United States in the mid-1980s, and the European Quality Award was established in 1993. A set of international quality standards, the ISO 9000 series, was

46 largely adopted to provide standards for testing, training, record keeping, and repairing products (Solis-Galvan). The reliability of a product can be greatly improved by ensuring manufacturing and design processes are well defined, developed, and adhered to (Solis-Galvan, 1998). A great deal of scholarly research exists on statistical process controls as commonly used to understand and control the sources of variation in a process. The implementation of Six Sigma manufacturing process controls is one method of ensuring the customer receives consistent quality and predictable product variability. The Six Sigma process control model was successfully implemented by such companies as General Electric, Motorola, AlliedSignal, Honeywell, and Raytheon is reported to have saved billions of dollars. In 1998, Raytheon implemented a Six Sigma program that generated over $2 billion in financial benefits resulting from over 3,000 projects over 6 years (Blair & McKenzie, 2004, p. 194). One of many effective methods for implementing process management is the integrated capability maturity model (CMMI). The process improvement model developed by the Carnegie Melon Software Engineering Institute was initiated by the DoD in 1997 (NDIA, 2006, Divisions). Product quality is a matter of achieving the original design requirements and characteristics while performing the intended function with little variability (CMMI, 2006, CMMI Benefits). In the CMMI model, process improvement benefits fall into the performance categories of cost, schedule, productivity, quality, customer satisfaction and return on investment (CMMI, Performance Categories). The CMMI model may be of benefit in ensuring conformance to processes and to total ownership cost management.

47 Bergquist (1996) investigated the many available quality and process improvement methods to determine whether there is a difference between manufacturers who do and do not employ the methods. Bergquist posited, There is an innate belief that these awards not only recognize outstanding performers but also motivate other companies to excel in the area of quality. There is considerable anecdotal information to support this, but almost no empirical data (Abstract). Bergquists research provided no significant statistical difference between companies that do implement quality management practices and companies that do not. Bergquist noted, The primary reason for the lack of difference between these groups is that many non-applicants are implementing quality management practices without seeking an award. Further, market influences drive a company's performance, not winning a quality award (Abstract). Despite the results of Bergquists (1996) research on the consumers perception of quality, manufacturers must consider quality and reliability management practices in product design and production. Ittner (1993) studied the value of quality cost measurements and quality improvement programs and the cost of quality and noted Companies committed to continuous improvement can achieve ongoing reductions in non-conformance costs while maintaining, or even reducing, the existing level of conformance expenditures (Abstract). Goering (1983) determined consumers continuously revise their expectations of product quality through observation. Goeing contended, Because consumers receive different samples, their expectations vary. As learning occurs, the demand curve shifts (Abstract). Bayus (1985) examined the way in which consumers receive product information and reported word of mouth is an extremely important factor in consumers final purchase decision, sometimes even more

48 influential than other promotional methods. Bayus also determined negative word of mouth can have disastrous consequences for a manufacturer. In current research, the Keller Fay Group has asked 100 different Americans a day about their speaking habits (Raasch, 2006, Abstract). Raasch claimed, Whereas the conventional wisdom is that bad news travels further than good news, our research indicates that the American people find positive word of mouth to be more credible or believable than negative word of mouth and they are much more likely to pass along what they learn during positive conversations than what they hear during conversations that are negative. (Abstract) The positive return on investment and the potential for increase in market share and competitive advantage provide decision makers with incentive to implement quality management strategies. Improving quality management practices may result in the reduction of product total ownership cost and further improve customer perception of value. Product Reliability Management Considerable research was undertaken to discover the value the consumer places on quality and reliability in the purchasing decision. In a highly competitive business environment, the consumers purchase expectation is to receive products with high quality, reliability, and durability. Parasuraman et al. (1985) posited, Quality is an elusive and indistinct construct often mistaken for imprecise adjectives like goodness, or luxury, or shininess, or weight (p. 41) and further noted, Service quality involves a comparison of expectations and performance (p. 43). Goel (1998) contended, In general, quality is related to the product features and their performance, in terms of target

49 value and variability (p. 6). Quality and reliability are primarily functions of design and are a large part of the total ownership cost drivers of a product. Product reliability is defined as the extent to which the test or test item provides consistent results each time it is used. Reliability is the probability an item will perform its intended function for a specified interval under stated conditions. It can be expressed numerically in quantitative terms as a failure rate over a specified period of time, providing performance guidelines and specifications. Durability is a special case of reliability, the ability of the system and its parts to perform its mission without failure, degradation, or demand on the support system (Military definitions, 2004). Reliability Assessment The Reliability Analysis Centre was commissioned by the DoD, which reported a need within the industry for a reliability assessment technique that can be used to estimate the reliability of systems in the field (Denson, 1999, p. 2). The Reliability Analysis Centre scientists conducted studies indicating the common causes of system failure and, through observation and experimentation, developed a reliability prediction handbook, Mil-Hdbk-217, that may be used to determine the failure rates of a number of part types. Reliability part-type failure rates are a function of probability, performance, time, and operating conditions. The failure rate standards were adopted by many commercial and industrial companies as a means of predicting system reliability. The failure rate of a product is a significant total ownership cost driver. Product Maintainability Practices Maintainability is the quantitative measure of the time it takes to repair a system and return it to operation and is a cost driver of total ownership cost. Maintainability is

50 the time it takes for a qualified person to isolate a failure and to fix the failure expressed as the mean time to repair. Maintainability is generally seen as the ability of an item or equipment to be maintained, whereas maintenance refers to the action required to restore the system to operation (Clark & Johnson, 2004). The maintenance actions performed on a system may be preventive or corrective tasks, such as inspecting, servicing, and repairing modification or overall. Maintainability is commonly measured in the mean time to repair, or the time it takes to perform a maintenance tasks divided by the total number of maintenance tasks for a given period. Clark and Johnson (2004) indicated, In other words, MTTR [mean time to repair] shows how long it takes to fix and how difficult it is to repair or service a system (p. 32). The maintainability of complex systems is a measure of the ease or difficulty to perform a preventive or corrective repair task. Tasks may be performed by the system operator or a highly skilled technician in a specially equipped facility. The time and resources, training and procedures necessary to ensure that the system is maintained in an operational state are assessed (Military definitions, 2004). The Army Material Systems Analysis Activity reported maintainability is primarily a design parameter. The maintainability design characteristics of a system are commonly provided as a developmental goal, since maintainability issues make up roughly 38% of the total life-cycle cost (Barringer, 2001). Maintainability design features might include modular design, ease of access, common interchangeable modules, and redundancy (Clark & Johnson, 2004). Complex system maintainability, as a function of the total ownership cost, may best be considered during product design to ensure preventive and corrective repair actions can be performed cost effectively.

51 Operational Availability Operational availability is a measure of the average percentage of time a product, application, or component is available for operation without either scheduled or unscheduled downtime (C.W. Moore, 2003). Operational availability provides a quantitative figure of the time the product is operational and is a key performance parameter of total ownership cost. Operational availability is of particular importance to the aerospace complex system end user because it is based on the mathematical relationship among product reliability, maintainability, and the effectiveness of the logistics support management system. An operational availability analysis is performed with the objective of determining the percentage of time a system is able to perform its function when needed, providing the consumer with a quantitative measure of availability (C. W. Moore, 2003). Military Aerospace Operational Availability The U.S. Department of the Navy determined, Operational Availability (Ao) is a primary measure for readiness for naval systems, subsystems, and equipment (C. W. Moore, 2003, Purpose section, para. 1), defining operational availability as a measure of time or probability that a systems capabilities will be available for operational use when needed (C. W. Moore, 2003, Concepts section, para. 1). The U.S. Army Material Command created the Logistics Support Activity organization in 1993 to track all Army maintenance, transportation, supply activities, and equipment usage data. The U.S. Air Force Acquisition Logistics Activity and the Space and Missile Systems Center are responsible for Air Force systems, subsystems, and equipment (Space and Missile

52 Systems Center, 2004). The military agencies ensure total ownership cost and operational availability are considerations in all military procurement. Commercial Aerospace Operational Availability The U.S. Federal Aviation Administration (FAA) sets an annual operational availability goal designed to meet the projected demand and reduce potential delays in operation in the aviation industry. The FAA (2006) set a goal to sustain adjusted operational availability at 99.5% for the reportable facilities (FY 2006 performance target) for 35 of the largest U.S. airports. The objective was to measure and track equipment availability time, ensuring optimal safety and security of national aviation. Any air traffic facility outages and failures to meet operational commitments were recorded and tracked monthly. The FAA (2006) claimed, On-Time NAS [National Airspace System] arrivals are affected by the airport and en-route capacity, which are directly impacted by the availability of the equipment and facilities supporting that capacity (Why the FAA Chooses This Measure). Operational availability is used as a measure of aviation performance and is a key performance parameter in the calculation of total ownership cost. Operational Availability Applications Operational availability is a key performance parameter in many total ownership cost applications. In the petroleum industry, UOP, formerly Universal Oil Products, provides the petrochemical and refining industries with leading-edge technology solutions (UOP, 2004, Overview). UOP provides the fuel processing industry with reliability, availability, inspection, and equipment support services. In an industry where unplanned downtime can cost millions, proactive reliability and maintenance programs

53 can improve operational availability, as well as decrease annual maintenance costs (UOP, Services). UOP reported the results of a typical client using proactive maintenance programs that resulted in savings of $5 million annually. Activity-Based Costing One of the most significant cost drivers in the total ownership cost of a complex system is the manufacturing process. The complex system developer that employs activity-based costing methods in the manufacturing process may make better informed decisions concerning product price and total ownership cost, since it provides a method for the measurement of the cost and performance of activities, resources, and cost objects Whittaker (2005). Activity-based costing provides the decision maker with insight into the cost of activities incurred by the firm and, as a result, the organization may choose to discontinue performing non-value-adding activities (Whittaker). Activity-based costing is a method used to recognize the causal relationship of cost drivers to activities. In the implementation of an activity-based costing model, resources are assigned to activities and then activities are assigned to cost objects based on their use. The cost of gathering the required data and making the trade-off analysis is not insignificant. Ramsey (2005) posited, Activity based costing does not come without a cost (data analysis and collection) and a cost/benefit analysis should be conducted before a particular firm adopts it (Abstract). Theory of Constraints The theory of constraints is a method that may be used to reduce production cost drivers and total ownership cost. Goldratt and Cox (1984) introduced the theory of constraints philosophy to provide new global principles of manufacturing

54 (Introduction, para. 1). Goldratt and Cox posited the only reason companies do anything is to make money, that anything a company does to speed up the process or generate throughput to make money is appropriate, and finally that a business operation is a big process with many subprocesses. The basis of the theory of constraints is the premise that every system or organization has a goal, there is a finite capacity for meeting that goal, the capacity is limited by constraints termed bottlenecks. The theory contends removing or reducing the effect of the bottleneck will improve throughput and once a constraint is successfully mitigated, the decision maker should move on to the next constraint (Goldratt & Cox, 1984). Reducing the constraints to improve productivity and throughput may prove to be an effective method for reducing the total ownership cost of the product. Earned Value Management Earned value management is a method for managing the ever-changing budget, schedule, and scope requirements of a project. The earned value project management method is used to assess the cost, schedule, and risks associated with projects. Anbari (2003) noted, It allows calculation of cost and schedule variances and performance indices, and forecasts of project cost and schedule at completion (Introduction, para. 2). The earlier a problem is identified, the better the probability of successfully mitigating the problem. Earned value management incorporates the organized components of the projects schedule, budget estimate, and scope of work into a process by which the projects forecasted costs at the end of the project can be more reliably determined (Warhoe, 2004, p. CSC.07.1).

55 To control and reduce total ownership cost the program manager may effectively use earned value measures to track and maintain performance, schedule and budget. Program managers may require their suppliers to use an earned value management system (EVMS) to accurately report the programs integrated cost, schedule, and technical performance. Solomon (2004) provides that earned value data will be reliable and accurate only if the right base measures of technical performance are selected and if progress is objectively assessed. If the program manager is measuring the wrong things or not measuring the right way, then EVM may be more costly to administer and may provide less management value. Warranties During the operation and support phase of the complex system life cycle, total ownership cost drivers include warranties and customer service. Product warranties became a costly marketing initiative and a major component of total ownership cost, used as product discriminators to increase a manufacturers market share (Warranty Week, 2004). The warranty is an important promise made by a manufacturer that the manufacturer will stand behind the reliability of its product. In the United States, it is a federal law that information about the warranty coverage is available to read prior to making a purchase, whether the purchase is made in person, via catalogue, or on the Internet. The provisions of warranty coverage vary, so the consumer may compare both the product and the guarantee of reliability (Warranty Week). Manufacturers provide warranties to limit the extent of liability in the event of product malfunction or failure. In a competitive marketplace, warranties are used to capitalize on the customers perception of product quality, to gain a competitive

56 advantage, and as a product discriminator (Warranty Week, 2004). Byrne (2004) posited, The 25 largest manufacturers in the United States spend a total of about $15 billion per year on warranty claims (Warranty Week, p. 34) and further explained, Between 10 and 15 percent of warranty payments are the result of fraud or invalid claims (p. 34). In 2002, it was estimated $10 billion was spent by the automotive industry on warranty claims. The automotive industry attempted to reduce warranty costs through better quality and by passing along a larger share of the claims to suppliers (Arnum, 2004). Boulding and Kirmani (1993) and Tan, Lee, and Lim (2001) provided evidence that warranties provide a quality signal to the customer. E. M. Moore and Shuptrine (1993) demonstrated one of the many problems with warranties is the level of education needed to read and understand the warranty, 12.7 median years of completed education for adults 25 years or older (p. 23), which is beyond the education level of over half of the American population. Customer Service Aftermarket repair is a large part of total ownership cost and a major consideration to both the manufacturer and the consumer as a cost driver. Since it is positively related with customer retention and customer loyalty, service quality may have a direct effect on organizations profits (Konstantinos, Nikos and Dimitra, 2002). The cost of maintenance and support is a large percentage of the total ownership cost. By ensuring customers are satisfied with the entire life cycle experience of product ownership, the decision maker may provide a competitive advantage. It is common knowledge in the electronics industries that a customer who has successfully experienced a breakfix episode develops a type of loyalty and is more likely to return and make

57 another purchase, knowing what to expect if a product does fail (Arnum, 2004). Customer loyalty and retention are greatly increased when the retailer or manufacturer provides repair and support in the event of product failure (Arnum, 2004). Product Disposal A contributing cost driver of the total ownership cost of complex systems is in the product disposal phase of the life cycle. The process of returning products from the customer to the supplier, or reverse logistics, demonstrates a significant impact to the manufacturer. Costs related to a product before the product is purchased, costs at the time of purchase, and costs after the purchase may all be affected by reverse logistics (Tibben-Lembke, 1998, p. 10). Because of the ever-increasing costs of product disposal and environmental regulation, firms are taking an active interest in what happens to their products at the end of their sales lives and the effects should be taken into account when calculating total cost of ownership (Tibben-Lembke, 1998). The Delphi Method The Delphi research method is a decision-making tool for obtaining consensus from a panel of experts. The Delphi technique is a forecasting method used to obtain opinions, explore ideas, and develop a consensus in the decision-making process (Franklin & Hart, 2006). The method was designed initially in the early 1950s when the U.S. Air Force commissioned the RAND Corporation to conduct a study of the effectiveness of atomic attack. The leaders of the Rand team determined they needed the opinions of a panel of experts, but were not satisfied with the conventional means of gathering group opinions (Cabannis, 2001). The derivation of the term Delphi relates to the Delphic Oracle, who, according to Greek mythology, was able to predict the future.

58 Project Delphi seemed a suitable name for a project that was to apply expert opinion to the selection of an optimal U.S. industrial target system and to the estimation of the number of atomic bombs required to reduce the munitions output by a prescribed amount (Clayton, 1997). The primary objective of the Delphi technique was to arrive at a set of statements the advocacy group or panel of experts can agree upon. The Delphi method was used in the current study to develop a decision-making model to identify, characterize, and organize available total ownership cost strategies and key performance parameters for future use in the aerospace industry. Dalkey and Helmer (1963) contended the Delphi technique is seen as a procedure to obtain the most reliable consensus of opinion of a group of experts . . . by a series of intensive questionnaires interspersed with controlled opinion feedback (p. 458). Dalkey (1969) described the Delphi techniques as a procedure with three features designed to minimize the biasing effects of dominant individuals, irrelevant communications, and group pressure toward conformity: 1. Anonymous responseopinions of members of the group are obtained by formal questionnaires. 2. Iteration and controlled feedbackinteraction is effected by a systemic exercise conducted in several iterations, with carefully controlled feedback between rounds. 3. Statistical group responsethe group opinion is defined as an appropriate aggregate of individual opinions on the final round. (Summary)

59 A number of questions must be asked before the decision of selecting or ruling out the Delphi technique is made: 1. What kind of group communication process is desirable in order to explore the problem at hand? 2. Who are the people with expertise on the problem and where are they located? 3. What are the alternative techniques available and what results can reasonably be expected from their application? (Adler & Ziglio, 1996; as cited in Gunaydin, 2006, p. 35) The current study, using a qualitative Delphi research method, was based on the perception of a panel of experts. When there is no exact knowledge of the outcome, the Delphi method is an effective tool for obtaining group consensus. While the purpose of the Delphi approach is to reach some level of stability in opinions through continuous rounds of discussion about a given topic, the value of the Delphi approach is to provide a theoretical framework for future exploration based on the thoughts of an expert panel in the research area (Franklin and Hart, 2006). Simon (2006) contended, Delphi techniques are used when the problem does not lend itself to precise analytical techniques, but can benefit from subjective judgements on a collective basis (p. 52). The selection of participants and group size is an important aspect of the Delphi method. The sharing of knowledge and interpersonal exchange is a characteristic of group dynamics. Rowe and Wright (1999) posited, The structure of the technique is intended to allow access to the positive attributes of interacting groups (knowledge from a variety of sources, creative synthesis, etc.), while pre-empting their negative aspects

60 (attributable to social, personal and political conflicts, etc.) (p. 354). The Delphi is considered to be more of a search for public wisdom than it is a search for individual knowledge or deliberative judgment. Although the median group size is normally 15 to 20 participants, the careful selection of individual experts who have knowledge of the subject and a participatory temperament is more important than rigid group size requirements (Twining, 1999). A major characteristic of the Delphi method is anonymity of the participants through use of a questionnaire. In a traditional Delphi technique, the panel of experts is asked to complete at least three rounds of surveys. During the initial round, open-ended questions are provided to experts in the area of study. Participants have an opportunity to revise their views based upon interaction. A second round of questions is developed as a result of the response to the first round of questions. In iterative rounds of questions, individual contributors may change their opinion based on controlled feedback of the statistical distribution of the groups response or hold fast to their reply. The answer to the questions is provided in an individual response, free of the potential influence of dominant members. Another objective of the Delphi method is to determine a measure of central tendency among the panel of experts and the degree of the spread around the measure. The panel members will be given the opportunity to reconsider their answers in relation to the groups mean and revise their responses if they so choose. Murphy et al. (1998) posited, There is an attempt to facilitate consensus yet also to describe the level of agreement since there is value in determining areas of close, moderate or little agreement (p. 21). The respondents are asked to rate their answers on a Likert-type

61 scale. The mean Likert-type scale response is provided back to the participants. When a response is different from that of the majority, the participant is asked to provide rationale. The panel members are then asked to reconsider their previous answers, in reference to the groups mean, and are given the opportunity to revise their responses. Rounds of questions in the Delphi procedure are repeated until participants no longer wish to change their responses. The final result is the statistical distribution of the response of the expert panelists. Summary The concepts of total cost, life-cycle cost, product life-cycle cost, and total cost of ownership are not new. Total ownership and life-cycle cost analysis has been in development since the late 1960s. The DoD determined the initial cost of procurement should not be the primary consideration for system procurement because the support costs far exceed the initial acquisition cost (DoD, 2003a). Adopting a total ownership cost philosophy changes the perspective of business from looking at the short-term cost advantages to an emphasis on the lowest long-term cost of ownership over the entire life cycle of a product (Ellram & Siferd, 1998; Ferrin & Plank, 2002; Hurkens et al., 2006). The benefits resulting from the effective implementation of total ownership cost strategies include improvement in supplier performance measurement, purchasing decision making, internal and external communications, supplier performance, reduction in the cost of poor quality, and focus for organizational continuous improvement efforts (Ellram, 1994). For many organizations, price is the only consideration and decision makers are not interested in cost (Hart, 1978). The poor availability and accessibility of the data necessary to perform total ownership cost analysis, which tools to use and which

62 cost factors are important, the lack of information and reporting systems, and the failure to communicate the strategic value are cited as barriers to implementation (Ellram, 1994; Gartner, Inc., 2006; Milligan, 1999). Total ownership cost strategies represent the true cost of product ownership by looking beyond the initial purchase price to include the six stages of the product life cycle. A total ownership cost analysis requires judgment on the part of the decision maker. A core set of cost drivers will include knowledge of the relationship to the phase of the life cycle, the initial purchase price and other nonrecurring costs, the recurring costs such as operation and maintenance, the costs associated with system unavailability, and the environmentally friendly disposal of the product (Ellram, 1993; Hart, 1978; Hurkens et al., 2006; Williams & Graveline, 2000). Activity-based costing (Whittaker, 2005), theory of constraints (Goldratt & Cox, 1984), earned value management (Solomon, 2004; Warhoe, 2004), the provision of warranties (Byrne, 2004), and exemplary customer support (Arnum, 2004) are provided as total ownership cost drivers. Many best practices are available to decision makers for the management and reduction of total ownership cost. Product life-cycle management is a process for effectively managing the entire life cycle of a complex system. When used as a strategy, product life-cycle management provides decision makers with an understanding of the information and process flows of various technologies necessary to optimize best value (Fraser, 2005; Hakola & Horning, 2004; Mostefai et al., 2005). When used in conjunction with customer relationship management, clear specifications, and requirements development, a quality product is provided at a reduced total cost over the entire life cycle (Crow, 2004; Goel, 1998; Korman, 2002).

63 During the design and development phase of the complex system life cycle, many organizations employ systems engineering practices. Effective implementation of systems engineering processes improved the product development process (INCOSE, 2003). The consideration of supportability or ease of maintenance, product reliability, and system operational availability are provided as total ownership cost key performance factors (Clark & Johnson, 2004; C. W. Moore, 2003). Conclusion Manufacturing and procurement decision makers in the aerospace industry must implement total ownership cost strategies to control and reduce the life-cycle cost of complex systems (Bailey & Heidt, 2003; Beaudreau & Naegle, 2005; Hall, 2005, Stundza, 2006). The U.S. Air Force provided one example of the benefits of employing total ownership cost strategies by establishing a total cost of ownership reduction initiative with the goal of saving or avoiding more than $3.4 billion over 12 years (Williams & Graveline, 2000). Despite the apparent advantages of adopting total ownership cost philosophies, few organizations implement the measures (Beaudreau & Naegle, 2005; Hurkens et al., 2006; Veenstra, 2000). A review of the available literature provided a better understanding of total ownership cost practices and philosophies; the advantages, disadvantages, significance and contribution of key performance factors; and incentives and barriers to implementation over the entire product life cycle. The research indicated the need for a framework that will provide decision makers with a tool for making continuous assessment of priorities and subsequent trade-off decisions of the many available cost management methods to optimize total ownership cost. Chapter 3 explores the

64 development of a decision-making framework for complex system total ownership cost. The framework was developed through a qualitative modified Delphi study with a qualified panel of experts.

65 CHAPTER 3: METHODOLOGY The purpose of the qualitative study using a modified Delphi method was to develop a decision-making framework for objectively guiding the total cost of ownership of complex systems over the entire product life cycle. The topic is representative of an area of decision making that may best be resolved by establishing the consensus of a diverse group of qualified experts. Participants in the modified Delphi technique established a consensus of relevant cost drivers and best practices necessary to develop total ownership cost strategies. For the purposes of the study, a diverse group of expert total ownership cost practitioners from the military and commercial aerospace industry was selected. The modified Delphi technique, according to Custer, Scarcella, and Stewart (1999), is similar to the full Delphi technique in that future events are predicted as the result of a consensus of opinion of a panel of experts. The major modification to the basic Delphi method is in the initiation of the process using a set of carefully selected criteria. The pre-screened or selected criteria may be drawn from various sources, including panelists or study groups. The primary benefit of this modification is that the response rate of the initial round is typically improved, and a solid grounding in previously developed work is provided. In the study, the Internet was used as the medium to direct questions to a panel of expert total ownership cost practitioners from the military and commercial aerospace industry. The participants were asked to provide their opinions concerning complex systems total ownership cost management, best practices, key performance parameters, and implementation strategies and barriers. As a result of the study, decision makers may

66 use the framework for the future development and implementation of total ownership cost strategies, complex system supply chain control and management, and comprehensive cost-benefit analysis. Research Design A review and analysis of research design methods was conducted to aid in the selection of the appropriate research approach for the study. Research is a scientific method that can be viewed as a process, beginning with identifying a research problem or issue of study. The next steps would be reviewing the literature, specifying a purpose for the study, collecting and analyzing data, and forming an interpretation of the information. Finally, a report is generated that may be disseminated to audiences for evaluation (Creswell, 2002). Research design can be considered as the blueprint for the collection, measurement, and analysis of the data. The plan and structure of investigation are developed in a manner as to obtain answers to research questions (Cooper & Schindler, 2003). In selecting the design to use, Cooper and Schindler indicated the researcher should develop a plan that outlines procedures for all research activities. The design should always be based on a research question to guide in the selection of the type and source of information. In the development of a research approach, one of the initial steps is the establishment of the appropriate methodology. Two major approaches to research are identified as quantitative and qualitative. Selecting the most suitable approach is largely dependent on the data type. Neuman (2003) contended, Qualitative research relies largely on the interpretive and critical approaches to social science [whereas] quantitative

67 researchers stress objectivity and more mechanical techniques (pp. 140141). Further explanations of the differences between quality and quantity are that quality is the essential character or nature of something, while quantity is the amount. Qualitative refers to the meaning, the definition or analogy or model or metaphor characterizing something, while quantitative assumes the meaning and refers to a measure of it (Cooper and Schindler, 2003). Exploratory research is customarily conducted when a problem is not clearly defined, or the scope is unclear, with the objective to discover a future state. Qualitative research is a more objective research method in which theories and hypothesis are tested using predetermined designs. The numerical data collected in quantitative research are more deductive and are analyzed using precise statistical methods. In choosing a qualitative method, theories, hypothesis, or understanding of the problem is assessed through observations or interviews (Simon, 2006). Correlation, descriptive, and case study research methods were considered for the study. The development of total ownership cost key performance parameters, best practices, and barriers to implementation are deductive or predictive in nature and reflective of a future state. Case studies emphasize exploration and in-depth examination of the phenomenon within a particular group. In a case study, one, or a few, key cases are selected to illustrate an issue and then analytically studied in detail. The researcher looks for averages or patterns across many units or cases (Neuman, 2003). Each of the research methods is reflective of a viewpoint that is in present time and was not considered an appropriate method for this study.

68 The research study assessed the advantages, disadvantages, key performance parameters, and barriers to the development and implementation of total ownership cost strategies. The study of a current situation for the purpose of contribution to a decision about it, changing it, or establishing a policy about it (Simon, 2006, p. 50) is best developed using a research method that considers the future perspective. The Delphi technique is considered to be the best known qualitative, structured and indirect interaction futures method (Woudenberg, 1991). The traditional Delphi brings together a panel of experts in at least three rounds of surveys (Brunk, 2003; Cabannis, 2001; Dalkey & Helmer, 1963). The modified Delphi method is similar to the traditional Delphi with the aim to reach consensus on a future state. The major difference between the traditional and modified Delphi method is in the procedure (Custer et al., 1999; Franklin & Hart, 2006; MacDonald). Custer et al. noted, The major modification consists of beginning the process with a set of carefully selected items (p. 2). In the modified Delphi technique, using the Internet as a medium, a pretest was given to a pilot group. The initial questions were adjusted as needed, based on the results of the pilot study, and provided to the qualified panel of experts. Participants in the pilot study held a position in military or commercial aerospace complex system total ownership cost strategy development or implementation and were considered subject matter experts by their peers. Each participating expert was asked to make independent judgments of applicable best practices and relevant cost drivers in the development of a framework of complex system total ownership cost strategies. Each of the experts was also asked to provide additional key performance parameters that may be included in the modified Delphi survey. Panel members were asked to suggest the names

69 of people in the military and commercial aerospace industry who, in their opinion, qualified as complex system total ownership cost experts. There were 23 participants in the panel of experts. The members of the panel rated their responses on a 5-point Likert-type scale. The Likert-type scale survey design was named for Rensis Likert, who, in 1932, attempted to improve the levels of measurement in social research by using standardized response categories in survey questionnaires. In February 2006, a forum was conducted of market researchers to understand their preference between 5-point and 6-point Likert scales. It was found that most modern researchers agree that the neutral rating in a 5-point scale is needed when conducting survey research in order to capture the unbiased sentiments of survey respondents (Gwinner, 2006). The responses were collated and the descriptive statistics and interquartile range were calculated for each question. The interquartile range is determined by subtracting the first quartile from the third quartile. The method considers the spread in the middle 50% of the data; it is not influenced by extreme values. The method is common in Delphi studies, which seek consensus among expert opinion. The resultant response to each question from the 5-point Likert-type scale was provided to the participants at the beginning of the third round of questions. The experts were asked to reconsider previous answers in reference to the mean group response and to provide a rationale for answers to questions that were outside the mean. A cumulative list of the answers as provided by the members of the panel was generated. The list formed the basis of the framework of best practices used in the development of aerospace complex system total ownership cost management.

70 Appropriateness of Design The qualitative research study using a modified Delphi method gathered expert opinion, insight, experience, and judgment in predicting the key performance parameters necessary to develop a decision-making framework for total ownership cost strategies. The modified Delphi study based upon the opinion of a panel of expert practitioners may provide the aerospace industry with a means to facilitate the implementation of costsaving strategies. The Delphi method as initially implemented by Dalkey and Helmer (1963) engaged the use of a panel of experts to forecast a future event. The modified Delphi methodology was adapted for the study based upon similar research. Linstone and Turoff (1975) stated, It is not, however, the explicit nature of the application which determines the appropriateness of utilizing Delphi; rather, it is the particular circumstances surrounding the necessarily associated group communication process (p. 4). The modified Delphi method used in the study met many of the criteria provided by Linstone and Turoff, who posited one or more of the following seven properties must be present before adoption of the Delphi technique: 1. The problem does not lend itself to precise analytical techniques but can benefit from subjective judgments on a collective basis; 2. The individuals needed to contribute to the examination of a broad or complex problem have no history of adequate communication and may represent diverse backgrounds with respect to experience or expertise; 3. More individuals are needed than can effectively interact in a face-toface exchange; 4. Time and cost make frequent group meetings infeasible;

71 5. A supplemental group communication process can increase the efficiency of face-to face meetings; 6. Disagreements among individuals are so severe or politically unpalatable that the communication process must be refereed and/or anonymity assured; and 7. The heterogeneity of the participants must be preserved to assure validity of the results, i.e., avoidance of domination by quantity or by strength of personality bandwagon effect. (p. 4) Linstone and Turoff (1975) declared, The Delphi method has had its greatest application and acceptance as a means of compiling a list of future technical events or developments and collecting subjective judgments regarding them (p. 100). The modified Delphi method was chosen for the study as a well-established research technique for predicting a future event or outcome (Cabannis, 2001; Custer et al., 1999; Simon, 2006). The results of a modified Delphi study using the Internet as the medium were proven a valid and reliable basis for the development of a framework or model (Brunk, 2003; Cross, 2002; MacDonald, 2003). The Internet format was effective in decision making, learning, and promoting interaction among participants (MacDonald, 2003). A disadvantage in the use of the Internet was cited by MacDonald, who reported participants in the study wished for more interaction, finding that this media were not responsive enough (p. 133). The study enabled members of an expert panel to combine their expertise into a comprehensive set of conditions for the development of complex system total ownership cost strategies. The

72 provision of a website domain as an anonymous conferencing option improved the participant interactivity. Research Questions The qualitative research study using a modified Delphi method gathered expert opinion, insight, experience, and judgment of the participating panelists in predicting the key performance parameters necessary to develop a decision-making framework for total ownership cost strategies. In the military and commercial aerospace industry there is a need to implement total ownership cost strategies to reduce life-cycle cost of complex systems (Bailey & Heidt, 2003; Beaudreau & Naegle, 2005; Hall, 2005; Stundza, 2006). Although the benefits that may be realized in the industry seem apparent, Hurkens et al. (2006) posited, Despite its conceptual attractiveness, and potential to save billions of dollars, total ownership cost analysis are not applied very widely (p. 29). The following research questions guided the study: 1. What are the best practices for controlling and reducing the total ownership cost of complex systems? 2. What are the key performance parameters in the development of a future complex system total ownership cost framework? Selection of a Population of Experts The success of the Delphi method relies upon informed opinion to support the creation of a decision-making framework of complex system total ownership cost strategies. Random selection of individual panelists was not an option. The panelists in the modified Delphi study included program and project managers, life-cycle management practitioners, and leaders in the military and commercial aerospace industry.

73 Participants in the study had special knowledge or expertise in the area of complex system life-cycle development and total ownership cost strategies. The criteria for selection of panel members should focus on the practitioner consider a subject matter expert (Jones and Hunter, 1995). The NDIA is composed of individuals from academia, government, the military services, small businesses, prime contractors, and the international community (NDIA, 2006, Mission Statement, Members, para. 1). Members of the SE division of the NDIA who participated in ongoing DoD initiatives to improve system supportability, sustainability, and affordability were targeted as panel candidates. Potential panelists were identified experts in their field if they met at least one criterion in each of the following three domains presented in Table 1. Informed Consent A small group of 8 knowledgeable practitioners from the aerospace industry in the southwestern United States who met the selection criteria were invited to participate in a pilot study. Following the pilot study, a demographic survey was distributed to select a panel of 23 volunteers considered by their peers to be experts in the field of total ownership cost analysis. The panel of experts was asked to participate in the modified Delphi method. Individuals nominated were recognized as experts in the field of complex system total ownership cost. Cooper and Schindler (2003) noted, Securing informed consent from respondents is a matter of fully disclosing procedures of the proposed survey or other research design before requesting permission to proceed with the study (p. 123). Participants were contacted to validate the selection criteria. Each panelist was asked to sign a letter of informed consent (Appendix A).

74 Table 1 Criteria for the Identification of Complex System Total Ownership Cost Experts Expert Criteria

Stakeholders Those who are directly affected Those who will be directly affected Experts Those who have an applicable specialty Those who have relevant experience Facilitators Those who are authors Those who are mentors Source: Linstone and Turoff (1975); Jones and Hunter (1995). Sampling Frame A pilot study was initially conducted with a group of 8 total ownership cost experts from the aerospace industry in the fall of 2006. The pilot study should draw subjects from the target population and simulate the procedures and protocols that have been designated for data collection (Cooper & Schindler, 2003, p. 86). Participants were stakeholders, experts in their field, and facilitators who readily imparted their knowledge of the total ownership cost subject. Members of the pilot study group met the prerequisites as outlined in the Population section and provided in Table 1. The goal of the pilot study group was to reduce ambiguity and confusion that may be raised by the seed questions and to ensure the technical features associated with the Internet medium are effective. A pilot study may provide verification that the study questions and measuring system are appropriate and easily understood (Simon, 2006).

75 The pilot study group was asked to identify the attributes that should be present in a complex system total ownership cost expert. Confidentiality The survey was based on the modified Delphi method using the Internet as a medium and ensuring anonymity of the participants. Dalkey (1969) noted the traditional way of pooling individual opinions is by face-to-face discussion. After face-to-face discussion, more often than not, the group response is less accurate than a simple median of individual estimates without discussion (p. 14). The traditional Delphi method ensures anonymity of the participants in an effort to reduce the effect of dominant individuals. Linstone and Turoff (1979) reported, Anonymity is particularly valuable in a highly structured environment where individuals may feel constrained in expressing their own views (p. 74). Anonymity of panel members was maintained by random numbering of each participant. Protecting the confidentiality of the participants is essential. The right to individual privacy is a matter of law in the United States and is taken very seriously (Cooper & Schindler, 2003). The participants in the study were informed individual confidentiality was ensured upon receipt of a letter of agreement (Appendix A). The letter informed participants the results of the study may be published. The letter also assured participants, in an effort to respect the right to privacy, individual names would not be used but would be held in confidence. Geographic Location A pilot study was conducted with a group of 8 total ownership cost experts from the aerospace industry in the South western area of the United States. At the beginning of

76 the Delphi process, a demographic survey was e-mailed to the systems engineering division members of the NDIA. The purpose of the survey was to gather information about prospective panelists relative to their experience with and knowledge about complex system total ownership cost strategies. The demographic survey was used to ensure the diversity of the panel by considering such traits as demographics, gender, and field of expertise prior to selection of the panel of Delphi participants. The panel of experts in the main study consisted of 23 members from the military and commercial aerospace industry. The resulting decision-making framework for the management of complex system total ownership cost will be shared with decision makers in aerospace procurement and manufacturing organizations in the south central region of the United States. The data will be made available to other developers and manufacturers in both the military and the commercial sector, including the NDIA, who could likely benefit from the information. Instrumentation The modified Delphi approach is a multiple-step interactive process aimed at achieving the consensus of a panel of experts (Custer et al., 1999) and was used as the study instrument. Each of the 8 pilot study experts was asked to provide detailed knowledge of total ownership cost key performance parameters, best practices, and barriers to implementation. Each panelist was asked to provide their opinion of the importance of total ownership cost key performance parameters, best practices, assessment methods, barriers to implementation and the required qualifications of a practitioner.

77 The pilot study panel was given the following list of open-ended seed questions intended to provide the first steps in developing the decision-making framework for complex system total ownership cost. 1. What benefits, if any, are there in implementing total ownership cost strategies in complex system procurement, development, or manufacturing? 2. What key performance parameters are necessary to assess total ownership cost? 3. What methods are available, or could be available, and applicable for guiding complex system total ownership cost decision-making strategies? 4. What are the barriers to the implementation and application of total ownership cost methods in procuring, developing, and manufacturing complex systems? 5. What framework can be developed to characterize and organize total ownership cost methods across the entire life cycle of a complex system? 6. What attributes or special knowledge is necessary to qualify a person as an expert in the development of complex system total ownership cost strategies? The responses from the study group panel of experts were used to develop the broad research questions used in the modified Delphi study. The pilot study group was asked to identify the attributes of practitioners considered experts in the field of complex system total ownership cost applications and implementation. The group was also asked to contribute to a list of potential study participants. Data Collection Following the pilot study, the modified Delphi instrument consisted of three rounds of questions in which the panel of experts was asked to respond individually to

78 the survey. The panelists selected to participate in the modified Delphi method collaborated in a process of online group consensus. An asynchronous online conference at www.totalownershipcost.net/Forum engaged the participating panel of experts. Participants on the panel were unknown to each other and a particular response was not attributed to an individual. Anonymity of the participants was maintained to avoid the problems experienced in face-to-face meetings (Cabannis, 2001; Dalkey, 1969; Dalkey & Helmer, 1963; Linstone & Turoff, 1979). During the first round of the study, the panel members were asked to provide their opinions and were encouraged to expand upon the seed questions. The response was compiled and a 5-point Likert-type scale survey was developed. In the second round of questions, each expert panelist received an e-mail with a cover letter and directions to provide a response using the 5-point Likert-type scale. Descriptive statistics were developed as a result of the responses, including the mean, standard deviation, and range (Cabannis, 2001; Custer et al., 1999; MacDonald, 2003). Simon (2006) noted by using descriptive statistics to calculate the results of the responses, the range may readily be used as a measure of the reliability of the forecast (p. 52). Each panelist had the opportunity to construct additional trends, events, and analysis for contribution to the survey. The researcher served as the liaison and provided an exchange of opinions with the goal of achieving group consensus. During the third round, each expert panelist received an e-mail with a cover letter and the groups mean response to each individual question. The data were also made available on the website site domain at www.totalownershipcost/Forum and discussion of the results was encouraged. All panelists were asked to revise their answers if they

79 desired to do so. The panelists with answers outside the group mean were asked to provide a rationale for their answers to clarify the experts difference of opinion from the majority of the group. The difference of opinion may generate a range of alternative solutions to issues and problems (Simon, 2006, p. 52). Descriptive statistics were developed as a result of the responses to the 5-point Likert-type questionnaire. The third round of questions was used in an attempt to achieve group consensus. Data Analysis Data were gathered from the modified Delphi during each survey round. During each round, the panel of complex system total ownership cost experts responded individually to a survey. The panel was asked to make independent judgments about the topic of discussion. The response to survey questions was in the form of a 5-point Likerttype scale. The responses to questions in each successive round were calculated to determine the descriptive statistics. The resultant descriptive statistics were prepared and provided to the panel of experts as feedback at the end of the second round. The panelists were asked to consider the questions that lack a group consensus. Each panelist could revise his or her answer or provide a rationale for the response to a question that was outside the group mean. Murphy et al. (1998) reported there is no guarantee any one judgment is correct but the best that can be done is to identify a method that will likely provide a better judgment than any other. No agreement could be found in the research regarding the one best method of evaluation of responses provided by a panel of experts, although it was noted that using the mean and standard deviation were not as strong as the median and the interquartile range. Prior to administering the third round of questions, where a

80 response from the second round fell below the group median, the question was discarded and considered unimportant to the experts. The interquartile range was used in addition to the mean and standard deviation. The interquartile range is the difference between the third quartile and the first quartile in a set of data. Use of interquartile range effectively removes the highest and lowest quarters of the collected data and eliminates the effect of outliers (Statistics Canada, 2006). Linstone and Turoff (1975) posited, In most Delphis, consensus is assumed to have been achieved when a certain percentage of the votes fall within a prescribed rangefor example, when the interquartile range is no larger than two units on a ten-unit scale (p. 273). Using the interquartile range provided a rationale for strong similarities among the participants. For the study, consensus was achieved when an interquartile range score of less than 1.5 was obtained. Where the response fell outside of the interquartile range, participants were given the opportunity to provide the basis and any additional information. A Friedman two-way analysis of variance (ANOVA) was used to evaluate the hypothesis that each of the expert panel members should provide the same response on the Likert-type scale. The Friedman test is an alternative to the repeated-measures ANOVA when the assumption of normality or equality of variance is not met. The ranks of data are used rather than raw values to calculate the statistic. Pett (1997) posited the Friedman test is very versatile and can be used both with randomized block designs and with multiple observations of a single sample, as in the case with a Delphi study (p. 143). Sheskin (2007) noted, The Friedman two-way analysis of variance by ranks is employed with ordinal (rank-order) data in a hypothesis testing situation involving a

81 design with two or more dependent samples (p. 1075). Both the Friedman test and the repeated-measures ANOVA are used to compare observations repeated on the same participants, as is common in modified Delphi studies. The Friedman test is based on the following assumptions: the rows are mutually independent, the data are ranked, and the result within one row or block does not affect the results within other blocks. The test compares the mean ranks between participants, assuming all experts would answer the questions the same way. As a result of the Friedman two-way ANOVA by ranks test using MINITAB software, where the test statistic (S) has a p value (p), adjusted for ties, less than 0.05, the null hypothesis is rejected. When the p value is less than 0.05, it is concluded that there is significant difference among the opinions of the expert panelists in their response to the Likert-type scale. Validity and Reliability The terms reliability and validity have multiple meanings. Reliability denotes dependability or consistency, that the same thing is repeatable under identical or very similar conditions, whereas validity infers truthfulness (Neuman, 2003). When a reliable test instrument is free of measurement error, repeated treatments will produce a uniform and stable response. A valid test instrument can be depended upon to provide results from the sample or population that are meaningful and justifiable (Creswell, 2002). Neuman (2003) declared, All social researchers want their measures to be reliable and valid. Both ideas are important in establishing the truthfulness, credibility, or believability of findings (p. 178). Validity of the study was improved by ensuring that participants in the modified Delphi panel were considered experts. Reliability is enhanced

82 through the use of the Friedman ANOVA method to evaluate consistently of the response between the panel members. Panel members were provided with the statistical result of the group response to the 5-point Likert type scale to improve upon the interpretation of the data (Franklin and Hart, 2006). Summary A modified Delphi method using the Internet as a medium was selected for the qualitative study. The modified Delphi technique consisted of a panel of qualified experts submitting responses to a questionnaire. A letter of informed consent was distributed to each perspective panelist with assurance that confidentiality would be maintained. Panelist response was anonymous to reduce the effect of dominant individuals (Dalkey, 1969, p. 16). As a result of the study the panel of experts reached consensus concerning many of the factors influencing the development of a decision-making framework for total ownership cost management of complex systems in the aerospace industry. A pilot study was conducted with a group of 8 qualified experts to validate the modified Delphi survey. Each panelist was asked to provide individual responses to questions and additional key performance parameters that may be included in the modified Delphi survey. The pilot study group was asked to nominate potential expert panelists in an effort to reduce any potential researcher bias (Custer et al., 1999). The responses were consolidated and adjusted as needed and formed the basis of the survey of 23 total ownership cost practitioner experts. During the first round of questions in the modified Delphi survey, the panel of experts was sent a list of open-ended questions. The panel of experts was given the opportunity to construct additional considerations that may contribute to the survey. A

83 second round of questions was developed from the responses of the first survey in the form of a 5-point Likert-type scale. The responses were collated and provided as feedback to the panel of experts. Each total ownership cost expert was asked to reconsider previous answers where the resultant descriptive statistics demonstrated no consensus (MacDonald, 2003). The participants were asked to provide a rationale for responses outside of the group mean. A third round of questions was developed based on the responses from round two. Questions that fell below the group median in round two were discarded. The remaining questions were provided in a 5-point Likert-type scale in an effort to achieve group consensus. Chapters 1 and 2 provided the research problem, purpose, and questions. Chapter 3 developed the theoretical foundation and methodology of the study. Chapter 4 provides the data results of the opinion of a panel of qualified total ownership cost experts using the modified Delphi technique.

84 CHAPTER 4: PRESENTATION AND ANALYSIS OF DATA The purpose of the modified Delphi study was to develop a decision-making framework for objectively managing the total cost of ownership of complex systems over the entire product life cycle. The need for the study, established in chapter 1, was to provide a leadership decision-making model of best practices for evaluating among alternatives to reduce and control product cost and risk and improve competitive advantage. The literature review in chapter 2 provided the details of the Delphi research method and an analysis of topics related to the development and implementation of total ownership cost strategies that may influence the life-cycle cost of complex systems. Chapter 3 detailed the modified Delphi research design and the appropriateness of the methodology for the study. The research questions were developed and provided to determine the best practices and key performance parameters for controlling and reducing total ownership cost for complex systems. Chapter 4 explains the data collection methods used and provides the results of both the pilot and the modified Delphi study. The modified Delphi technique used in the study made it possible for a diverse group of qualified experts to establish a consensus of relevant cost drivers and best practices necessary to develop total ownership cost strategies. The experts were selected from both the military and the commercial aerospace industry. In addition, the panel of experts included academics that have published their articles in peer-reviewed journals or met selective criteria and presented their research at educational conferences and symposiums. The panelists who met the criteria were asked to invite colleagues they considered experts in the field of total ownership cost. The credentials of all referred participants were validated by the researcher. Each panel member signed a letter of

85 informed consent prior to participating in the pilot study. The demographic characteristics of the panelists in the pilot study are included in Table 2, and Table 3 provides the demographic characteristics of individuals who participated in the main study. Some individuals who took part in the pilot study participated in the main study and are again included in Table 3. Table 2 Demographics of Experts in the Pilot Study Pilot study Published in peer reviewed journal Design and development practitioner Finance practitioner Procurement practitioner Gender Male Female 2 1 3 2 1 Military Commercial 1 1 2 2 1

The pool of individuals who were considered experts was small and the task of finding individuals who were willing to spend the necessary time to complete the study was daunting. The initial pilot study survey was distributed to 136 people, and the main study questionnaire was distributed to 232 people. Although both males and females participated in both studies, male participation in the main study was much higher than female participation. The response to both the pilot and the main study was less than anticipated, but many individuals who declined to participate due to pressing commitments articulated their interest in the study and requested to be informed of the

86 results. Individuals who did participate in the study expressed significant interest in the importance of the findings to both military and commercial total life-cycle cost strategies. Participants noted the relatively few articles on the future development and implementation of total ownership cost strategies, complex system supply chain control and management, and comprehensive cost-benefit analysis. The time from the beginning of the pilot study to receipt of the final response from the Delphi panel was a little longer than 8 months. Table 3 Demographics of Experts in the Main Study Main study Published in peer reviewed journal Design and development practitioner Finance practitioner Procurement practitioner Gender Males Females 5 1 16 1 Military Commercial 2 2 1 1 1 7 6 3

The total ownership cost experts were invited to participate as panelists to address the following research questions: 1. What are the best practices for controlling and reducing the total ownership cost of complex systems? 2. What are the key performance parameters in the development of a future complex system total ownership cost framework?

87 Chapter 4 provides the results of the study based upon the research and methodology presented in chapters 1, 2, and 3, as well as the results of the pilot study. Chapter 4 also provides an explanation of the data collection methods used in both the pilot and the main studies. The results of the pilot study and an explanation of how the seed questions were used is then presented, with a discussion of how the data gathered were used to develop and improve the main study. The answers provided by the expert panelists to each of the research questions are presented. Data Collection for the Pilot Study The pilot study was completed with a panel of 8 total ownership cost practitioners and published academics considered experts in the field. The pilot was conducted between January 8, 2007, and June 16, 2007, with the purpose of testing the usability of the survey and the modification of the seed questions to better address the research questions. The pilot study aided in the development and improvement of the content of the survey questions and the format (Creswell, 2002). During the first round of questions a letter of informed consent was sent to prospective panel members outlining the role each member would play in the study. The letter included an outline of the credential that would qualify an expert participant. The criteria for the identification of complex system total ownership cost experts are provided in Table 1. The initial contact also provided an outline of the problem as identified by the researcher. The first-round letter also included an explanation of the purpose of the study. The prospective study group panelists were asked to provide their opinion of the problem and purpose as provided by the researcher.

88 Of the 136 letters sent to individuals identified as potential expert panel members, 28 people responded. The consensus of the respondents was agreement that there is a real problem and the study may help in the development and implementation of total cost of ownership strategies. All but 8 of the respondents declared despite the benefits that may be gained from the study they were unable to take the time to provide their input due to conflicting commitments. Participants in the pilot study were asked to provide written responses to six seed questions. Initial data were collected through the use of the World Wide Web using email. The written responses to the seed questions were compiled and posted on an Internet discussion forum at www.totalownershipcost.net/Forums/ and further discussion and interaction were encouraged. The study group was asked whether they were in agreement on the compiled response and to make any additions or changes. The discussion and study group response to each of the six seed questions is provided in the following section. Pilot Study Results Seed Question 1 asked, What benefits, if any, are there in implementing total ownership cost strategies in complex system procurement, development, or manufacturing? Implementing a total ownership cost strategy throughout a program life cycle enables effective decision making based upon true program costs. If a total ownership cost strategy is employed, it provides an understanding of future costs that may not be apparent prior to commencing an activity or making an initial purchase. Total ownership cost provides a method for arriving at the lowest total cost among several alternatives and the ability to more effectively manage financial performance. When

89 implemented, total ownership cost strategies can help decision makers avoid the embarrassment and damage caused by unannounced and unanticipated major costs as the project progresses through the life cycle and also provide for the future funding of key elements. When total ownership cost strategies are employed, all aspects of ownership are taken into account when defining the initial project contract requirements. During the contract development stage of the product procurement process, total ownership cost strategies aid in the selection, qualification, and evaluation of prospective suppliers. Employing total ownership cost strategies creates a common language for crossfunctional engagement and tends to balance the selection criteria among multiple bidders during procurement. When used during the project initiation stage, total ownership cost can serve as a baseline against which future cost reduction approaches can be applied and results measured. If the measures are accurate and the right things are being measured, then an organization not only can garner cost reduction within the purchase parameter, but also related and aligned processes may see significant cost or other benefit improvements. Effective total ownership cost strategies typically translate into improved system performance and may create value for the customer. One of the benefits of implementing a total ownership cost strategy during product development is the achievement of required performance for the minimum cost. The greatest impact in reducing cost will occur if the strategies are put into place early in the life cycle, often referred to as the window of opportunity. Because the intention was to minimize the total ownership cost,

90 it should be possible to insert in the early design various features intended to achieve optimal operation, maintenance, and support over the entire product life cycle. There are benefits in implementing total ownership cost strategies in the manufacturing process. The benefits may help to identify hidden costs and risks related to the need for special tooling and testing, component and process standardization, parts availability and diminishing manufacturing sources, the reliance on sole source suppliers, scheduling constraints, and product quality and reliability. Effectively implementing total ownership cost strategies during the manufacturing phase of the product provides the operator a more realistic view of operational costs to facilitate budgeting and forecasting. Following the manufacture of the product, the operation and support phase of the life cycle may account for 60 to 80% of the total life-cycle cost. The effective implementation of total ownership cost strategies may provide quality, reliability, and warranty information that leads to competitive advantage. Seed Question 2 asked, What key performance parameters are necessary to assess total ownership cost? The key performance parameters for total ownership cost vary from program to program and the type of environment the system will be operating in. Total ownership cost models vary greatly depending on the type of operating setting, location, required performance, and expected usage. Adequate measurement systems, accounting systems, and robust processes must be in place to understand performance differences among alternatives. Clearly defined technical performance parameters and priorities and their associated thresholds are key factors. The factors may also include operational availability, product reliability, availability, maintainability, and support structure. Management systems that account for the cost of quality, risk, schedule, and lost

91 opportunity cost are beneficial. The pilot study group identified an extensive list of key factors found in Appendix B that, in their opinion, should be included in the assessment of total ownership cost. Seed Question 3 asked, What methods are available, or could be available, and applicable for guiding complex system total ownership cost decision-making strategies? The methods for estimating total ownership cost change over the life of the program. Early in the life cycle, parametric estimating tools are most appropriate. Analogy to similar programs can be used throughout the life cycle. Detailed estimates and vendor quotes are best suited for later phases in the life cycle. There are specific DoD instructions and handbooks available including cost-as-an-independent-variable analyses to examine cost implications of programmatic decisions. High-resolution cost models are necessary to support trade-off decisions and yield valuable results. Commercially available software assists with the computation of total ownership cost for various solution strategies. The computer-based software systems make it possible to communicate and organize information in a meaningful way. Distribution requirement and resource planning software, material requirements planning software, electronic data interchange, activity-based costing, and enterprise resource planning software are among the available software packages. A historical review of past performance is essential to give any estimate credibility. Forecasts of production and support costs may be based on an analysis of history. A weighted system evaluation method such as Kepner-Tregoe may be used for decision analysis. Brainstorming may be used to identify issues, and process mapping will aid in understanding interactions and issues.

92 Seed Question 4 asked, What are the barriers to the implementation and application of total ownership cost methods in procuring, developing, and manufacturing complex systems? There is a significant amount of effort and cost involved in the development of a total ownership cost analysis. Implementing total ownership cost strategies requires the allocation of resources in the early stages of a project to obtain future gain as a result of accurate forecasting. The effective implementation of total ownership cost strategies requires ongoing commitment and support throughout the product life cycle. The product developer is charged with looking beyond initial procurement costs in decision making and must be resolved to meet long-term goals. The planning, management, and allocation of resources require long-term responsibility of stakeholders and decision makers. Technology forecasting and future development make the implementation of strategies difficult. Systems are initially designed and developed to meet specific requirements, while providing for future upgrades or modifications. The usable life of a system may be extended beyond the initial total ownership cost analysis assumptions. Decision makers who are responsible for total cost of ownership may not be able to make the long-range decisions that affect it. Department of Defense federal budgeting rules make funding of activities over multiple years difficult to obtain. The people who need to make the investments are often not with the program in the future to realize the benefits, so they are not interested in savings beyond their assignment to the position. Adding additional program length adds additional total ownership cost and makes the analysis lack credibility. No general formula exists for performing a total ownership cost analysis. An effective total ownership cost analysis is dependent upon the identification of relevant

93 cost factors and key performance parameters that may not be readily available. Interagency collaboration and total life-cycle system engineering across diverse entities and suppliers is required to identify the factors. Decision makers may not have systems in place within the firm and may not have either the ability or the desire to share relevant information across firms. The lack of empirical data on previous or like systems makes total ownership cost estimates suspect or lack credibility. A total ownership cost analysis may not assess the risks involved in selecting one approach over another, because the results may be inconsistent and not suitable for high-resolution cost analysis. Seed Question 5 asked, What framework can be developed to characterize and organize total ownership cost methods across the entire life cycle of a complex system? The development and implementation of total ownership cost strategies should be considered a requirement in program management plans for new system design and development. Decision makers should determine what is necessary to perform the computations in terms of a valid work breakdown structure and available technology, systems, agreements, and expertise. The initial steps in developing total ownership cost and trade-off analysis determine the availability of information technology and software systems, followed by gathering adequate cost factor data. Cost targets must be assigned to each element and a plan should be developed to fill any gaps in the existing data. A process should be put in place for retrieving and collecting any required missing data. With the analysis of data, there should be regular reviews of both the factors being tracked and the analysis to ensure as many cost factors as possible are being accounted for in the total cost of ownership analysis and accounted for appropriately. An adequate measurement system should be developed to verify estimates, ensure credible

94 and accurate trade-off analyses, and direct postproduction engineering efforts. The measurement system should include comparative historical analysis and consider costs by what occurs prior to the transaction, during the transaction, and after the transaction. Achieving projected total ownership cost performance should be a program review element and should be linked with the success criteria. A verification mechanism should be established to ensure the total ownership cost analysis is available for review as a required system performance element. Controls should be set in place to ensure effective strategies remain in place after implementation and lessons learned are gathered for future analysis. Seed Question 6 asked, What attributes or special knowledge are necessary to qualify a person as an expert in the development of complex system total cost of ownership strategies? A subject matter expert should be capable of examining the system from a total life-cycle perspective and evaluating total system performance. The expert should be capable of performing analysis to make trade-off decisions concerning the best possible total cost of ownership. The effective implementation of total ownership cost strategies may be the result of experience in doing it successfully or having studied many applications. The expert should have a broad understanding of development, production, and system support activities, the ability to see how things work together, and the ability to be objective when looking at each life-cycle phase. Along with experience, the subject-matter expert should have a broad knowledge of business, management, processes, and financial models. The ideal expert must have credibility with decision makers and be a detail-oriented systems or long-term thinker. The expert should have a solid background in both accounting and supply chain

95 management. The total ownership cost expert should have either formal training or experience in program management, systems engineering, logistics and product support, cost-estimating methods, financial models, complex systems design, manufacturing, integration, testing, and operation. A mix of cost analysts, systems engineers, and software developers working with subject matter experts such as logisticians, industrial engineers, design engineers and systems engineers, testers, and users should be utilized in a multifunctional approach, such as in integrated process teams. Pilot Study Group The results of the first round pilot study response to the seed questions were analyzed for apparent themes, similarity, or overlap. NVivo 7 software, created by QSR International was used to catalog phrases and sentences into emerging categories. From the analysis, an online 5-point Likert-type scale was developed and presented to the expert panel members for their agreement or dissent in Round 2. Panelists were asked to provide additional statements and to provide clarifying comments as desired. The online survey was posted on the Internet at http://totalownershipcost.net/LikertTest. The study group panel was asked to respond to the 5-point Likert-type survey by rating the importance of each of the statements. The mean, median, standard deviation, and range for each statement were calculated and provided as feedback to the pilot study group with a request to provide clarification for any response outside the group mean. Where the group response to a statement fell below the group median, the statement was removed from the 5-point Likert-type survey. The resultant survey was administered to the panel in Round 3. The response from the study group was used to ensure validity and reliability of the Likert-type scale instrument and provides the basis for the main study.

96 Main Study Results Using a modified Delphi method, 232 invitation letters were sent to prospective panelists to join in Round 1 of the main study to provide open-ended response to a set of six seed questions relating to the studys two central research questions. Of the individuals invited to participate, 23 total ownership cost experts agreed to form the panel and each was assigned an identifier, which ranged from P1 to P23. Response was captured via the secure and confidential Web site using the same online survey as the pilot group. Results of the survey were downloaded into a Microsoft Excel document and were immediately backed up. Panel members represented both military and commercial total ownership cost practitioners or academics who published peer-reviewed articles. Each of the main study group panel members was asked to sign a letter of consent and was provided with the Web site address www.totalownershipcost.net/, which provided an overview of the study and links to the open discussion forum and the Likert-type scale survey. Data Collection The expert opinions of the participants in the modified Delphi total ownership cost study formed the basis for advice on the future and current state of total ownership cost management. The findings indicated characteristics and key performance parameters that may be used by the profession to improve the development and implementation of total ownership cost strategies. In addition, the experts provided the attributes of a total ownership cost strategist and the methods that may be used when assessing total ownership cost, as well as the benefits that may be realized and the barriers that may be

97 faced when implementing total ownership cost strategies in complex system procurement, development, or manufacturing. In Round 1, the panel of experts was directed to the open discussion forum found on the Internet at http://totalownershipcost.net/Forums/ and members were asked to provide their feedback, comments, and additions to their response to the initial six seed questions as provided by the study group. No dissimilar comments or additions were determined using the NVivo 7 software program and the study groups open-ended response was not altered. When Round 1 concluded, open-ended responses to the six seed questions were provided by 29 experts. For Round 2, panel members were asked to go to the Internet Web site http://totalownershipcost.net/LikertTest/ to provide their rated responses to the 5-point Likert-type scale. Panelists were asked to provide clarifying comments as desired. Twenty-three panelists completed Round 2 of the survey. Upon completion of Round 2, the mean, standard deviation, and range for each of the responses were calculated. Where the rated response from the 5-point Likert type scale fell below the group median, the statement was discarded and the results were used in the development of survey Round 3. Round 3 Results Round 3 of the survey was created from the responses to Round 2. Using the results of the descriptive statistics, a statement that had a response below the group median was eliminated from the study in Round 3. Twenty-one experts completed responses to Round 3. Panel members were asked to provide their opinions by responding to the 5-point Likert-type scale with a ranking of the importance of each statement, with 1 as least important and 5 as most important. The mean, standard deviation, and range for

98 each statement were calculated and any expert comments were summarized. In addition, the interquartile range (IQR) was calculated for each statement, which is provided as the difference between the 25% and the 75% point of the response. Upon completion of Round 3, panelists were given the opportunity to comment on any ranking in which the IQR was 1.5 or greater and to provide an explanation of their response by posting their remarks on the http://totalownershipcost.net/Forums/Web site domain. The descriptive statistic results of Round 3 using MINITAB software are found in Appendix C. In addition to the descriptive statistics analysis, a Friedman test for nonparametric statistics was run using MINITAB software with the hypothesis that the mean ranks for repeated measures obtained from a group of experts should not differ. The test compares the mean ranks between participants assuming all experts would answer the questions the same way. A p value greater than the chosen alpha level of 0.05 indicated one or more treatments had significant effects. In the study the Friedman test was developed such that a sample of the population of total ownership cost experts, who are considered mutually independent and denoted C3 (panelist), was applied versus C2 (question) and was blocked by C1 (response). The results of Round 3 Friedman analysis is found in Appendix D. Research Question 1 What are the best practices for controlling and reducing the total ownership cost of complex systems? Seed Question 1. A review of Seed Question 1, What attributes or special knowledge is necessary to qualify a person as an expert in the development of complex system total cost of ownership strategies?, provides an indication that consensus was

99 reached among the panel members. The MINITAB test statistic (S) in the Friedman test, shown in Appendix D for Seed Question 1, had a p value of 0.19 when adjusted for ties. Because the p value was greater than 0.05 the null hypothesis cannot be rejected. In response to the 11 statements found within Seed Question 1, there is no significant difference among the expert panelists. An assessment of the interquartile range shows that none of the eleven statements have a value greater than 1, confirming that the study group reached a consensus. The result of the descriptive statistics, found in Appendix C, show that participants in the study (95%) either agreed or strongly agreed the most significant attribute or special knowledge required of a total ownership cost practitioner is the ability to work comfortably across multiple organizations. P3 posited, An expert should possess good personal skills and working knowledge in dealing with both the customer and supplier and must be empowered to act on their decisions. The expert must be both able to and comfortable with working across multiple organizations. The study group (62%) agreed or strongly agreed the ability to communicate effectively is an essential skill required of the total ownership cost practitioner, whereas the remainder of the participants remained neutral. The review of the literature did not find any reference to the practitioners need for good communication skills. P7 indicated the expert should have a broad understanding of development, production and system support activities, the ability to see how things work together and the ability to be objective when looking at each life cycle phase. The majority of the study group (67%)

100 agreed or strongly agreed formal training is required for effective implementation of total ownership cost strategies, whereas 30% remained neutral and 3% disagreed. The study group, when asked open-ended questions, contended the total ownership cost expert should have either formal training or experience in program management, systems engineering, logistics and product support, cost estimating methods, financial models, complex systems design, manufacturing, integration, testing, and operation. When asked whether formal training or practical experience was more essential, the experts agreed or strongly agreed (78%) that practical experience is more essential than formal training. The one area in which the expert panel members agreed or strongly agreed (67%) that formal training is required is systematic implementation of total ownership cost strategies. Seed Question 2. An analysis of Seed Question 2, What framework can be developed to characterize and organize total ownership cost methods across the entire life cycle of a complex system?, was conducted and consensus was not reached among the expert panel members. The MINITAB Friedman test statistic (S), found in Appendix D for Seed Question 2, had a p value of 0.00 and was less than 0.05 adjusted for ties. The data supported the rejection of the null hypothesis because a significant difference existed among the panel members. The interquartile range for the group response to the statement that the establishment of a comprehensive work breakdown structure is important was 1.5 denoting that there was no group consensus reached on this point. The result of the descriptive statistics, found in Appendix C, shows that all expert participants (100%) either agreed or strongly agreed the establishment of a plan from the very beginning of a development program is the most essential factor. P17 declared,

101 Cost targets must be assigned to each element and a plan should be developed to fill any gaps in the existing data. When asked to rate the need to assign cost targets early in the project, 62% of the participants agreed or strongly agreed and 38% remained neutral toward the statement. Participants in the study (76%) either agreed or strongly agreed it is necessary to formalize a process to pull necessary data from various systems to review and analyze. Once the factors were made available, 66% of the participants agreed or strongly agreed, whereas the remainder were neutral that the review should include both the factors being tracked and the data coming out of the systems as essential elements in the development of total ownership cost strategies. The results of the study indicated 76% of participants agreed or strongly agreed and 24% remained neutral on the need to establish a comprehensive system life-cycle work breakdown structure (WBS). The interquartile range of 1.5 indicated no consensus among the expert panel concerning the statement. The standard deviation is 0.79, yet the mean response from the participants (4.14) ranks the development of a WBS as the second most essential element in the development of total ownership cost strategies. Seed Question 3. A review of Seed Question 3, What are the barriers to the implementation and application of total ownership cost methods in producing, developing, and manufacturing complex systems?, was conducted and consensus was not reached among the study group members. The Friedman test, found in Appendix D for Seed Question 3, was conducted and the MINITAB test statistic (S) had a p value of 0.03 adjusted for ties, which is less than 0.05. The data in this case supported the rejection of the null hypothesis because there was a significant difference among the expert panelists.

102 An analysis of the response to the 9 statement within Seed Question 3 found three areas where consensus was not reached as the interquartile range was 1.5 or higher. The panel members did not place equal value on the barrier to implementation of total ownership cost strategies due to procurement budgets that are based on purchase price and do not consider total ownership cost; acquisition and sustainment budgets that are separately identified, acquired and managed; and relevant data that is not shared across firms or business units. An assessment of the descriptive statistics, found in Appendix C, shows that when asked to rate the importance of the barriers to total ownership cost implementation, 76% of the experts either agreed or strongly agreed, whereas 24% of the participants remained neutral that the biggest barrier to implementation is procurement budgets are based on purchase price alone and do not consider total ownership cost. The interquartile range of 1.5 indicated poor consensus among the expert panel concerning the statement, yet the mean response of 4.29 indicated the statement ranks as the fourth most significance barrier to implementation. The study group identified inconsistency between individuals responsible for total ownership cost and individuals able to make strategic decisions as a barrier to implementation. The majority of study group members (87%) agreed or strongly agreed, whereas the remainder were neutral. The lack of long-term system planning is provided as a barrier by the experts, with 95% agreeing or strongly agreeing and 5% remaining neutral. In addition, 90% agreed or strongly agreed, whereas 10% remained neutral that a significant barrier to implementation is that total ownership cost is not a program review element linked with the success of the development.

103 The majority of study participants (90%) agreed or strongly agreed a major barrier to implementation of total ownership cost strategies is that procurement is focused on the lowest purchase price. Federal budgeting rules that restrict the funding of activities over multiple years was considered a barrier to implementation by 62% of the experts who agreed or strongly agreed; 38% remained neutral. Another barrier identified by the study is that acquisition and sustainment budgets are separately identified, acquired, and managed. The study experts (78%) agreed or strongly agreed, whereas the remainder was neutral. The interquartile range of 1.5 indicated poor consensus among the expert panel concerning the statement. When participants were asked to rank the difficulty that relevant data are not shared across firms or business units, the interquartile range of 2.0 indicated little consensus in the response of the study group. The results illustrated only 43% agreed or strongly agreed, 14% remained neutral, and 43% disagreed with the statement. When the study group was asked to rate the significance that the lack of data relevant to making valid assessments plays as a barrier to implementation, the interquartile range of 1.5 demonstrated consensus was not reached. The statement is considered an important factor as the majority of the participants (86%) agreed or strongly agreed, 10% remained neutral, and 4% disagreed. Seed Question 4. A review of Seed Question 4, What methods are available, or could be available, and applicable for guiding complex system total ownership cost decision-making strategies?, provided little consensus among panel members. The Friedman test was conducted and the MINITAB test statistic (S), found in Appendix D for Seed Question 4, had a p value of 0.00 adjusted for ties, which is less than 0.05. The

104 null hypothesis was rejected because significant difference existed among panelists. There was little agreement among the experts concerning the best method for making total ownership cost assessments. The interquartile range does not show a significant area of disagreement yet the range of 1 is provided on four of the five statements within Seed Question 4. No single tool was determined or assessed during the course of the research that could be used universally across a variety of products and programs for performing total ownership cost analysis and implementing the strategies. The majority of the panel members (86%) declared they use tools they developed on their own. An assessment of the descriptive statistics, found in Appendix C, provides that the historical review of past performance is the preferred method for performing total ownership cost assessments; 81% of the experts agreed or strongly agreed, 14% were neutral, and 5% disagreed with the statement. Activity-based cost management for better visibility of the cost drivers was also considered useful in the assessment of total ownership cost. The majority of the expert panel members (71%) agreed or strongly agreed, 19% were neutral, and 10% disagreed. The other available methods included the analysis of cost as an independent variable, earned value management systems, and weighted system evaluation methods. Each of the methods was rated by the study group. The majority of the panel members (67%) in the main study were neutral as to the effectiveness of these total ownership cost assessment methods. Seed Question 5. An analysis of Seed Question 5, What benefits, if any, are there in implementing total ownership cost strategies in complex system procurement, development, or manufacturing?, was conducted and shows that study group consensus was not reached. A Friedman test was applied to determine if there is significant

105 difference between the panelists. The MINITAB test statistic (S) in the Friedman test, shown in Appendix C for Seed Question 5, has a p value of 0.00 when adjusted for ties. The p value is less than the specified level of significance 0.05 and as a result the null hypothesis is rejected because there is significant difference among the expert panelists. The modified Delphi study group failed to come to a consensus on one of the five statements. An assessment of the interquartile range of 1.5 shows little consensus concerning the ability of total ownership cost to provide an effective measure of return on investment. The 5-point Likert type scale response to the five statements within this seed question was evaluated using descriptive statistics. The result of the descriptive statistics, found in Appendix C, show that all 21 experts who participated in the main study agreed or strongly agreed implementing total ownership cost strategies in complex system procurement, development, or manufacturing provided decision makers with an effective tool for making cost and performance trade-off decisions. The majority of panelists (81%) either agreed or strongly agreed implementing total ownership cost strategies provided a method to make effective decisions based upon total system cost. Eighty-six percent either agreed or strongly agreed implementing total ownership cost strategies provided decision makers with a measure of cost necessary to achieve required performance over the entire system life cycle, whereas 14% remained neutral. The majority of experts (86%) either agreed or strongly agreed total ownership cost provides a measure of the return on investment necessary to assess cost reduction initiatives. P7 posited, When used during the project initiation stage, total cost of ownership can serve

106 as a baseline against which future cost reduction approaches can be applied and results measured. When asked to rate the implementation of total ownership cost strategies as an effective method for providing quantitative data used to predict future cost, 81% of participants either agreed or strongly agreed, whereas the remainder of the participants remained neutral. P9 contended, If the measures are accurate and one is measuring the right things, then an organization can garner not only cost reduction within the purchase parameter, but related and aligned processes may also see significant cost and/or other benefit improvements. Research Question 2 What are the key performance parameters in the development of a future complex system total ownership cost framework? Seed Question 6. In Seed Question 6, panel members were asked to provide their expert opinion of the key performance parameters that are necessary to assess total ownership cost for all complex system programs. There was little consensus reached among the participants ranking the importance of the nineteen key performance elements within Seed Question 6. The MINITAB Friedman test statistic (S), found in Appendix D for Seed Question 6, provided a p value of 0.00 when adjusted for ties and the null hypothesis was rejected because a significant difference existed among the opinions of the expert panel members. Members of the study group did not reach consensus on 8 of 19 statements as the interquartile range on the response to these statements was 1.5 or higher.

107 An assessment of the descriptive statistics found in Appendix C for Seed Question 6 provides that when asked to rate the key performance parameters required to develop total ownership cost strategies, the experts agreed with a mean of 3.52 that the cost of the initial design is the most important factor. Although the factor is provided as the most important from the experts perspective, there was poor consensus as indicated by the interquartile range of 1.5. The study indicated 43% of the participants agreed or strongly agreed, whereas an equal number of participants (43%) remained neutral and 14% disagreed. The cost associated with product development is rated as the second most significant performance factor, where 43% of the participants agreed or strongly agreed, 48% remained neutral, and 9% disagreed. The cost of redesigning a system is considered slightly less significant; 38% agreed or strongly agreed, 52% remained neutral, and another 10% disagreed. The cost of quality is considered in the design cost but the interquartile range of 1.5 indicated poor consensus among participants. In this instance 57% of the experts agreed or strongly agreed, whereas 38% were neutral and 5% disagree that the cost of quality is a key performance factor. The participants ranked knowing the initial acquisition cost as the most important performance parameter in the area of product acquisition but there was no clear consensus because the interquartile range is 2.0. The study indicated 71% of the panel members agreed or strongly agreed acquisition cost is a key total ownership cost performance factor whereas 19% were neutral and 10% disagreed. There was also little consensus concerning the ranking of the manufacturing cost as a key parameter where the interquartile range was 1.5. When asked to rank the manufacturing cost as a key

108 performance parameter, 48% agreed or strongly agreed it is important, whereas 52% of the participants remained neutral. The cost of subcomponents and assemblies was not seen as a significant contributing factor; 38% of the participants agreed or strongly agreed but 52% were neutral and 10% disagreed acquisition cost is a key performance factor. The majority of participants (53%) agreed the cost of raw materials and sourcing costs should be considered important factors, whereas 36% were neutral and 11% disagree that these are key total ownership cost performance parameters. Participants ranked knowledge of the failure rate or system reliability as a significant factor; 86% of the panelists agreed or strongly agreed, 10% remained neutral, and 4% disagreed. The majority of the panel members (62%) agreed or strongly agreed the environment in which the system is operating and the operational availability are important performance parameters. The interquartile range for both elements is 2.0, which indicated little consensus in the response among the experts. When asked to clarify the response, P2 declared, When you are home and your vehicle is not available for some reason you can take a taxi. When you are in combat and the vehicle is not available that could be the difference between life and death. A neutral response was received from 29% of the participants whereas 9% disagreed that environment and operational availability are key to the development of total ownership cost strategies. The study groups placed knowing the cost of maintenance and operation as a leading performance parameter in the development of total ownership cost strategies. The clear majority, 95%, agreed or strongly agreed and 5% were neutral when ranking knowledge of the operating cost as a key performance parameter. The cost of repair was

109 ranked as a significant performance parameter; 76% of the expert panel agreed or strongly agreed and 19% were neutral, but an additional 5% disagreed. The experts failed to come to a clear consensus that the cost of repairs is a key parameter because the interquartile range was 1.5. When asked for clarification, P7 posited, The cost of repairs may be considered part of the maintenance cost and I did not consider it separately. The majority of the expert panel group (95%) either agreed or strongly agreed knowing the operating cost of the system is a key performance parameter, with the remainder providing a neutral response. The cost of preventive and corrective maintenance activities were both recognized as key to the development of total ownership cost strategies. Participants (76%) agreed or strongly agreed preventive and corrective maintenance cost should be considered, whereas 12% remained neutral and 12% disagreed. The interquartile range result of 1.5 when asked to rank the importance of corrective maintenance demonstrated poor consensus of the group. When asked for clarification, P7 posited, Corrective and preventive maintenance can be rolled up into operating cost in the early stages of the life cycle. As more data is gathered over the life cycle the more granular the analysis can be. Summary Based on the analyses of data, strong consensus was evidenced among total ownership cost experts regarding the value and viable future for developing a framework for the development, implementation, and management of total ownership cost strategies. Respondent P17 stated, In my opinion, total ownership cost should, itself, be a key performance parameter, that is specified by the user and of such importance that failure to achieve the parameter may be cause for cancellation of the program. The panelists

110 strongly agreed there is no universal total ownership cost strategy that will work in all cases. P15 noted, TOC [total ownership cost] models vary greatly dependent upon the type of operating setting, location, required performance and expected usage. There are few identified models for estimating total ownership cost change over the life of a program. Before implementing any total ownership cost strategies, the panel affirmed, Historical review of past performance is essential to give any estimate credibility (P11). The experts agreed the largest barrier faced by decision makers when implementing total ownership cost strategies is that it is not a recognized program review element. P7 noted, The effective implementation of TOC strategies requires ongoing commitment and support throughout the product life cycle. The most essential skills for the decision maker who develops and employs total ownership cost strategies are the ability to work comfortably across multiple organizations and to effectively communicate with others. Chapter 4 provides the results of the study based upon the research and methodology presented in chapters 1, 2, and 3 and an explanation of the data collection methods used in both the pilot and the main studies. The answers provided by the expert panelists to each of the research questions were presented. Chapter 5 concludes the study and provides recommendations for a framework of best practices for controlling and reducing the total ownership cost of complex systems. Chapter 5 provides a brief overview of the study; an assessment of the implications, interpretation, and significance of the response to the guiding research questions; and recommendations for future research.

111 CHAPTER 5: SUMMARY AND RECOMMENDATIONS Chapter 5 is divided into three sections. The first section provides a brief overview of the study, which sought to develop a framework that identifies, characterizes, and organizes methods for managing total ownership cost of complex systems throughout the entire life-cycle process. The second section contains the conclusions to the study, providing a review and application of the main research questions. The third section of chapter 5 summarizes the findings and includes recommendations for further study. The summary also includes suggestions and explanations of total ownership cost assessments that provide decision makers with a plan to implement total ownership cost strategies. Overview of the Study The intent of the modified Delphi study was to reach consensus among a panel of experts regarding the future of total ownership cost as it applies to the life cycle of complex systems. The need for the study was based on three factors. First, leaders in the military and commercial aerospace industry must plan for total life-cycle management of complex systems (Fraser, 2005; Hakola & Horning, 2004; Mostefai et al., 2005). Second, despite the benefits and potential savings of billions of dollars that may be realized over the entire life cycle of a product, total ownership cost strategies are not widely accepted or applied (Crow, 2004; Hurkens et al., 2006). Third, a profound need exists for military and commercial aerospace manufacturing and procurement decision makers to implement total ownership cost strategies to control and reduce life-cycle costs of a complex system (Bailey & Heidt, 2003; Beaudreau & Naegle, 2005; Hall, 2005; Stundza, 2006).

112 The concepts of total cost, life-cycle cost, product life-cycle cost, and total cost of ownership can be traced back to the time of Sun Tzu around 400 B.C. (McNeilly, 1996). Prior research dedicated to total ownership cost practices and philosophies, advantages, disadvantages, significance and contribution of key performance factors, incentives and barriers to implementation over the entire product life cycle determined the initial cost of procurement should not be the primary consideration for system acquisition because the support costs far exceed the initial purchase cost (DoD, 2003a; Ellram & Siferd, 1998; Ferrin & Plank, 2002; Hurkens et al., 2006). In addition, prior studies focused on the benefits that resulted from the effective implementation of total ownership cost strategies, including improvement in supplier performance measurement, purchasing decision making, internal and external communications, reduction in the cost of poor quality, and focus for organizational continuous improvement efforts (Ellram, 1994; Gartner, 2006; Milligan, 1999). Prior research (DoD, 2003a; Ellram & Siferd, 1998; Ferrin & Plank, 2002; Hurkens et al., 2006) determined adopting total ownership cost strategies changes the perspective of the business and the decision maker from looking at short-term cost advantages to the significance of long-term cost of ownership. When used as a strategy, product life-cycle management provides the decision maker with an understanding of the information and process flows of various technologies necessary to optimize best value (Fraser, 2005; Hakola & Horning, 2004; Mostefai et al., 2005). When used in conjunction with customer relationship management, clear specifications, and requirements development, a quality product is provided at a reduced total cost over the entire life cycle (Crow, 2004; Goel, 1998; Korman, 2002).

113 The qualitative study focused on the development of a framework that may be used by the practitioner in the assessment and successful implementation of total ownership cost strategies. The views and opinions of a diverse group of total ownership cost experts were sought using a modified Delphi method. The surveys, interviews, and observations determined which best practices and key performance parameters could be incorporated in a framework for future total ownership cost strategy assessment and implementation. A 5-point Likert-type scale survey was administered and the results assessed using descriptive statistics and the interquartile range. A Friedman two-way ANOVA was used to evaluate the hypothesis that each of the expert panel members should provide the same response on the Likert-type scale. The Friedman test was conducted using MINITAB software with the hypothesis that the mean ranks for repeated measures obtained from a group of experts should not differ. The modified Delphi panel started with 29 experts and concluded with 21 experts, either academics who published total ownership cost topics in peer-reviewed articles or practitioners who developed and implemented total ownership cost strategies as part of their occupation. Data were obtained to provide answers to two questions that guided the direction of the study: What are the best practices for controlling and reducing the total ownership cost of complex systems? What are the key performance parameters in the development of a future complex system total ownership cost framework? Answers to the guiding questions were used to create a framework to provide organizational leaders support and beneficial information in the process of assessing and implementing total ownership cost strategies.

114 The next section presents and interprets conclusions drawn from the literature review, the modified Delphi methodology, and the analysis of data. The answers provided by the expert panelists to each of the research questions are presented. The interpretation and analysis of the data provided the significance and meaning of the study to total ownership cost practitioners, decision makers, and organizational leaders in the process of developing and implementing total ownership cost strategies. Conclusions The findings from each of the seed questions and their application to the research presented in chapter 1 follows. The interpretations and conclusions drawn from the research of the available literature, the modified Delphi methodology, and analysis of the data indicated total ownership cost and life-cycle cost analysis of complex systems is a legitimate, dynamic, and challenging field of endeavor (Fraser, 2005; Hakola & Horning, 2004; Mostefai et al., 2005). Some benefits that result from the effective implementation of total ownership cost strategies in the procurement process are described as improvements in monitoring and controlling suppliers, decision making, communication, performance evaluation, and continuous improvement efforts (Ellram, 1994; Ferrin & Plank, 2002). Leaders in the military and commercial aerospace industry who plan for the total ownership cost management of complex systems may realize the benefits of reductions in product cost that may net the industry billions of dollars, while improving their own competitive advantage (Bailey & Heidt, 2003; Beaudreau & Naegle, 2005; Crow, 2004; Hall, 2005; Hurkens et al., 2006; Stundza, 2006; Veenstra, 2000). The aim of the modified Delphi study was to obtain the consensus of a representative group of the population of total ownership cost experts in response to two

115 research questions. Six seed questions were asked of the expert panel to provide an in depth explanation, clarification and understanding of the information. The conclusions reached as a result of the response of the study panelists provides the basis for the development of a decision making framework for total ownership cost management of complex systems. Using the Internet as a medium for the modified Delphi study, the expert panelists provided an open ended response to the seed questions. The response was used to develop a list of elements or attributes that the study group rated from least important to most important using a 5-point Likert-type scale survey. Following the response to the survey, using the results of a descriptive statistical analysis, any element that fell below the group median response was discarded from the next round of questions and considered unimportant. In the third round study group members again responded to a 5point Likert-type scale. The response was collated and the descriptive statistical analysis was provided as feedback to the panel members. Where a response fell outside of the mean, participants were given the opportunity to reconsider their answer and provide a rationale for a response. Research Question 1 What are the best practices for controlling and reducing the total ownership cost of complex systems? Seed Question 1. Seed Question 1 asked, What attributes or special knowledge is necessary to qualify a person as an expert in the development of complex system total cost of ownership strategies? Twenty-two elements were provided in the second round 5point Likert-type scale survey based upon the response obtained from the open-ended

116 questions. As a result of the assessment of the descriptive statistics found in Appendix C, the 11 elements considered by the panel to be the most important were provided in the survey Round 3. Eight of the 11 elements that were below the group median and subsequently discarded concerned the requirement for formal training. When asked whether formal training or practical experience was more essential, the experts determined that practical experience is more essential. The one exception is that panel members agree formal training is required in the systematic implementation of total ownership cost strategies. The study group determined that the most important attribute is the ability to comfortably work across multiple facets of an organization or between organizations. Based upon the hypothesis that the response from each study group member should be the same if all are equally capable, a Friedman test, found in Appendix D for Seed Question 1, was performed and the interquartile range was calculated for each response. Both the p-value result of the Friedman test and the interquartile range provided evidence that the panel was able to reach a consensus. The panel of experts provided that effective communication skills, practical operation and support cost estimating experience, practical total ownership cost strategy implementation, and practical experience in systems engineering across the life cycle of complex systems are the next most important attributes or skills required in the development and implementation of total ownership cost strategies. No literature providing insight into the required skill set that a total ownership cost practitioner should have was discovered. Seed Question 2. Seed Question 2 asked, What framework can be developed to characterize and organize total ownership cost methods across the entire life cycle of a

117 complex system? In this assessment seven elements were provided to the study group in the second round 5-point Likert-type scale survey. The five elements considered by the panel to be the most important were provided in the survey Round 3. The group did not consider an enterprise resource planning system and an earned value management system necessary in the development of a framework and these elements were subsequently discarded before Round 3. The descriptive statistics of the study group response is found in Appendix C and clearly indicates, with a mean of 4.86 and a standard deviation of 0.36, that the panel members strongly agree the most important element in the development of a total ownership cost framework is the need to plan to make total ownership cost a priority from the very beginning in a development program. The view of the panel supports the literature as Swink (2006) noted, Some 70 to 80 percent of product lifecycle cost is fixed by decisions made in product design and development (p. 39). Planning for the execution of total ownership cost strategies should be conducted in the early stages of complex system product design and development to be most cost effective. When total ownership cost strategies are employed, all available aspects of ownership should be taken into account and integrated into the program plan. The review of the literature provides that in the aerospace industry, product life-cycle management is an effective strategy for ensuring total ownership cost philosophies are integrated in the decision-making process (Crow, 2004; Korman, 2002; Mostefai et al., 2005; Saur et al., 2003). The development and implementation of total cost of ownership should be considered early in the initial planning, design and development stages of a new system. A Friedman test, found in Appendix D for Seed Question 2, was performed and

118 the interquartile range was calculated for each study group response in Round 3. Both the p-value result of the Friedman test and the interquartile range provided evidence that the panel was unable to reach a consensus on all of the factors. The panel did not come to a consensus that there is a need for a comprehensive system life cycle work breakdown structure yet this factor is considered the second most important. A work breakdown structure is used extensively in military program planning and execution but is less evident in commercial system development and therefore considered less important. The expert panelists did reach a consensus provided in order of importance that data should be reviewed, tracked, and updated; a process should be implemented to gather, review, and assess data as it becomes available; and achievable cost targets need to be assigned. To optimize the effectiveness of total ownership cost assessments, plans made during the early stages of complex system development should include realistic and achievable cost targets and incorporate methods for updating assessments as actual data become available. Seed Question 3. Seed Question 3 asked, What are the barriers to the implementation and application of total ownership cost methods in producing, developing, and manufacturing complex systems? An analysis of the descriptive statistics found in Appendix C for Seed Question 3 provided evidence of an agreement with the available literature that there are identifiable barriers to the implementation of total ownership cost decision-making strategies in the development of complex systems. Twenty elements were provided in the second round 5-point Likert-type scale survey. Eleven points considered inconsequential were discarded and the remaining nine

119 elements considered by the panel to be the most significant barriers to effective total ownership cost implementation were provided in survey Round 3. The review of the literature revealed that many barriers to implementation are attributed to a lack of understanding and poor communication of the value of the total ownership cost concept (Ellram, 1993, 1994, 1995; Ferrin & Plank, 2002; Gartner, 2006; Milligan, 1999). The results of the third round 5-point Likert-type survey are found in Appendix C and show two elements with a mean of 4.38. The standard deviation of 0.67 is used to confirm that the study group reached a consensus that the most significant barrier to the implication of total ownership cost strategies is that total ownership cost is not a significant review element linked to program success factors. A review of the literature determined the total cost of ownership of a product is the sum of both direct and indirect cost over the entire life cycle and the initial purchase price alone does not provide a complete picture of total product cost (Ellram, 1993; Hart, 1978; Hurkens et al., 2006; Kaye et al., 2000; Padnos, 2006; Williams & Graveline, 2000). Consumers purchase decision should extend beyond the initial purchase price and influences of the brand name, product features, and functions. The study group supported the view as the consensus is that the second most significant barrier to implementation of total ownership cost strategies is that procurement is focused on the initial purchase price. In a review of the literature the poor availability and accessibility of the data necessary to perform total ownership cost analysis, which tools to use and which cost factors are important, the lack of information and reporting systems, and the failure to communicate the strategic value are cited as barriers to implementation (Ellram, 1994; Gartner, 2006; Milligan, 1999). The study group determined that the unavailability of

120 data necessary to make strategic decisions and assessments ranked as the third most significant barrier to implementation. A Friedman test, found in Appendix D for Seed Question 3, was performed and the interquartile range was calculated for each study group response in Round 3. Both the p-value result of the Friedman test and the interquartile range provided evidence that the panel was unable to reach a consensus on all of the factors. There was no consensus reached among the group when asked to rank the importance of procurement budgets that are based on purchase price without consideration of the total ownership cost. Despite the lack of consensus this element was ranked as the fourth most significant barrier to implementation. The lack of long-term systems planning was ranked as the fifth most significant barrier to implementation of total ownership cost strategies. There is a significant amount of effort and cost involved in the development and implementation of total ownership cost strategies. The effective implementation of this decision making philosophy requires the allocation of resources early in a program to obtain future gains. To overcome the barriers to implementation, the complex system leader must be convinced of the value of total ownership cost prior to making plans and decisions based on long-term forecasts and goals. Seed Question 4. Seed Question 4 asked, What methods are available, or could be available, and applicable for guiding complex system total ownership cost decisionmaking strategies? In this assessment, nine elements were provided to the study group panel in the second round 5-point Likert-type scale survey. Four elements were considered inconsequential and the five remaining points considered by the panel to be

121 the most important were used in survey Round 3. A Friedman test, found in Appendix D for Seed Question 4, was performed and the interquartile range was calculated for each study group response in Round 3. The p-value result of the Friedman test provided evidence that the panel was unable to reach a consensus. There was little agreement among the experts concerning the best method for making total ownership cost assessments. The historical review of past performance is rated by the study group as the preferred method for guiding the development of total ownership cost strategies. The advancement of computer technology makes it possible to store in one place all complex system data that are or will be available throughout the product life cycle. The advent of product life-cycle management may provide complex system decision makers with an effective tool for reducing the total ownership cost of a product over the entire life cycle (Swink, 2006). The availability of applicable data, collected throughout the system life cycle, is considered by the panel to be essential to ensure credibility. Updates of the assessment should be made as actual or historical data becomes available. A review of the available literature indicated the ability to easily calculate total cost of ownership is provided as a significant barrier to implementation. Barringer and Weber (1996) posited the calculation of life-cycle cost is easy when the data are available, including such variables as the reliability failure rates and any maintenance required, operating and installation conditions, and costs for lost gross margin for outages of systems when appropriate. As a result of the survey, the study group rated activity based cost management and cost as an independent variable analyses as the second and third preferred methods for performing total ownership cost assessments. No one

122 software program or assessment tool was found that could be used universally. No general formula was found during the course of the research or the study for performing a total ownership cost analysis in all cases. As a complex system progresses throughout the life cycle, the methods used in the assessment change. P7 declared, Models vary greatly dependent upon the type of operating setting, location, required performance and expected usage. During the design and development phase assumptions and predictions of key performance parameters may be used. Commercially available software and programs developed by experienced practitioners that assist the decision maker in the development of cost and programmatic strategies are limited by the accuracy of the available data. To provide an accurate total ownership cost assessment of the available data on all programs, the tool used must provide flexibility and adaptability to change over time. Seed Question 5. Seed Question 5 asked, What benefits, if any, are there in implementing total ownership cost strategies in complex system procurement, development, or manufacturing? In the assessment of this question, 10 elements were provided to the study group panel for the second round 5-point Likert-type scale survey. Five of those elements were considered inconsequential as they were below the group median and the five remaining points considered by the panel to be the most important were used in Round 3 of the survey. In a review of the literature, a number of benefits originated following the decision to implement total ownership cost strategies. Planning for the implementation of total ownership cost strategies provides an objective and systematic method for investigating the costs and benefits of applicable alternatives and provides a significant

123 tool for the redesign of a more effective supply chain (Hurkens et al., 2006). The study group concluded the most beneficial aspect of implementing total ownership cost strategies is that it provides a method for making effective decisions based upon the total system cost. In the early development of a program the decision maker can use total ownership cost to assess the cost of implementing product quality and reliability improvements. It is at this stage in the product life cycle that an assessment can be made between competing suppliers. Following the manufacture of the product, the operation and support phase of the life cycle may account for 60 to 80% of the total life-cycle cost. A total ownership cost analysis will show the savings generated in the reduction of future customer support, repair, maintenance, and warranty cost. The cost of implementing these improvements can be weighed in a trade off analysis against the resultant increase in the initial selling price. Total ownership cost provides a method for arriving at the lowest total cost among several alternatives and the ability to more effectively manage financial performance. The study group rated using total ownership cost strategies as a method to achieve required complex system performance over the entire life cycle as the next most important benefit to implementation. One of the benefits of implementing a total ownership cost strategy during product development is the achievement of required performance for the minimum cost. When total ownership cost strategies are employed future product cost that may not be apparent prior to commencing an activity are considered. The organization implementing total ownership cost for supplier selection and initial purchase will garner cost reduction within the purchase parameter over the entire product life cycle. Hidden costs and risks related to the need for special tooling and

124 testing, component and process standardization, parts availability and diminishing manufacturing sources, the reliance on sole source suppliers, scheduling constraints, and product quality and reliability are identified. The study group rated the availability of quantitative data that may be used to predict cost as the third important benefit to implementation. When implemented, total ownership cost strategies help decision makers avoid unannounced and unanticipated major costs as the project progresses through the life cycle. An active assessment of total ownership cost provides for the anticipated need for future funding of key product elements. During the manufacturing phase the operator is provided with a more realistic view of operational costs to facilitate budgeting and forecasting. Warranty Week (2004) indicated in a competitive marketplace, warranties are used to capitalize on the customers perception of product quality, to gain a competitive advantage, and as a product discriminator. The modified Delphi study failed to come to a consensus concerning the value of total ownership cost to enhance competitive advantage. The p-value obtained from the Friedman test found in Appendix D and the interquartile range of 1.5 found in the descriptive statistics in Appendix C both conclude that there is no consensus between study group members. There is little agreement that effective total ownership cost strategies provide a measure of the return on investment necessary to assess cost reduction initiatives. The panelists agree that the greatest impact in reducing cost will occur if the strategies are put into place early in the life cycle. The intention is to achieve optimal operation, maintenance, and support over the entire product life cycle for the lowest cost. P11 posited, The effective implementation of total ownership cost may provide quality, reliability and warranty information that leads to

125 competitive advantage. Panel members expressed concern that the data being measured, the accuracy of the measures and the effective measurement of the right things makes implementing total ownership cost strategies difficult. Research Question 2 What are the key performance parameters in the development of a future complex system total ownership cost framework? Seed Question 6. What key performance parameters are necessary to assess total ownership cost? The assessment criteria found in DoD Instruction 5000.2 directs procurement decision makers to consider the entire system life cycle and ensure the acquired system meets operational requirements and the key parameters of cost, performance, and schedule. Appendix B provides a list of 38 elements that were provided by the study group as key performance parameters that may be used in the assessment of total ownership cost. The list was reduced to 18 significant factors that were above the group median and included in the 5-point Likert type scale for Round 3. The result of the study provides that little consensus was reached among the participants ranking the importance of the key performance elements. The MINITAB Friedman test statistic (S), found in Appendix C, provides a p value of .001 when adjusted for ties and the null hypothesis was rejected because a significant difference existed among the opinions of the expert panel members. The interquartile range, also used to measure the group consensus, confirms that there is little agreement among the panelists. P11 posited, Parameters will vary from program to program and the type of environment the system will be operating in. The ranked group response obtained from the Round 3 survey for Seed Question 6 is found in Appendix C. The group response is in

126 agreement with the literature that total ownership cost requirements change throughout the product life cycle. The group determined that operating cost, maintenance cost, product reliability, cost of repair, corrective and preventive maintenance cost and finally acquisition and sourcing costs are the most important factors. The response from the study group fell into four main areas: design and development, acquisition, product reliability, and repair costs. Design and Development When asked to rate the key performance parameters required to develop total ownership cost strategies, the experts agreed with a mean of 3.52 that the cost of the initial design is the most important factor. In the review of the literature Swink (2006) posited, Some 70 to 80 percent of product lifecycle cost is fixed by decisions made in product design and development (p. 39). Although the factor is provided as the most important in this category from the experts perspective, there was poor consensus concerning the rank of the level of importance as indicated by the interquartile range of 1.5. Planning for total ownership cost implementation early in the product design phase is an essential total ownership cost parameter. The cost associated with product development is rated as the second most significant performance factor while the cost of redesigning a system is considered slightly less significant. The cost of quality is provided as an important element for consideration in the total complex system design cost but the interquartile range of 1.5 indicated poor consensus of the rank of the level of importance among participants.

127 Acquisition The views of the panel are in agreement with the findings in the literature where total ownership cost and life-cycle management may be used as a strategy to effectively optimize the relationship of cost, schedule, and performance based on the requirements and expectations of the stakeholders (Fraser, 2005; Hakola & Horning, 2004; INCOSE, 2003; Kludze, 2003; NASA, 1995, Saur et al., 2003). Much of the research was dedicated to the effective management of the total ownership cost that results from acquisition and the supply chain (Bailey & Heidt, 2003; Beaudreau & Naegle, 2005; Bhutta & Huq, 2000; Crow, 2004; Ellram, 1993, 1994, 1995; Hurkens et al., 2006; Stundza, 2006). The results of the study agreed with the literature that there is a significant need to include acquisition cost as a significant performance parameter in the development of total ownership cost implementation strategies. The participants, with a mean of 3.95, ranked knowing the initial acquisition cost is the most important performance parameter in the area of product acquisition but there was no clear consensus as the group response interquartile range is 2.0. There was also little consensus concerning the ranking of the importance of knowing manufacturing cost as a key parameter where the interquartile range was 1.5. The cost of subcomponents and assemblies, the cost of raw materials and sourcing costs should be considered important factors, but again there was poor consensus concerning the ranking or the importance of Product Reliability A review of the literature indicated the consideration of supportability or ease of maintenance, quality and product reliability, and the provisions of warranties and system operational availability are provided as total ownership cost key performance factors

128 (Clark & Johnson, 2004; C. W. Moore, 2003; Goel, 1998; Parasuraman et al., 1985; Kotler, 2000; Seawright, 1994). One of the most significant values customers expect to receive from vendors is high product and service quality. The product warranty is an important promise made by a manufacturer that the manufacturer will stand behind the reliability (Warranty Week, 2004). The research is verified by the results of the study, which indicated product development decision makers face these challenges when making quality, reliability, and durability engineering decisions and total ownership cost assessments. There was consensus achieved from the study group where panelists ranked knowledge of the failure rate or system reliability as the most important factor in this category. The importance of the environment in which the system is operating and the operational availability provided little consensus among the panelists. The interquartile range for both elements is 2.0, which indicated little consensus in the response among the experts. There are few applications outside of the military where a complex system may well have to operate in remote Arctic or desert areas. For the nonmilitary panelist there is less importance afforded the need to have a system perform its function when required. When asked to clarify the response, P2 declared, When you are home and your vehicle is not available for some reason you can take a taxi. When you are in combat and the vehicle is not available that could be the difference between life and death. Repair In a review of the literature, Barringer and Weber (1996) reported the least system ownership expense is the acquisition and most sustaining expenses are not obvious. They further posited when assessing total ownership cost, The often cited rule of thumb is

129 65% of the total life cycle cost (Barringer & Weber, p. 17) of a system is expended in sustaining, operation, and support. The view was supported by Louden (2006), who presented data indicating the average cost of research and development is 2%, acquisition is 34% and operating and support is 64% (p. 24) of the total ownership cost of complex systems. Further research provided similar findings. During the 1960s, the DoD determined the initial cost of product procurement was the primary consideration for system acquisition and a lack of planning and analysis for the total cost of the system resulted in support costs that amounted to 60-70% of the initial cost of procurement, far exceeding the initial acquisition cost (Kaminski, 1995, p. 2). The study groups came to a consensus and placed knowing the cost of operation, with a group mean of 4.67, and the cost of maintenance, with a group mean of 4.29, as the leading performance parameters in the development of total ownership cost strategies in this category. The cost of preventive and corrective maintenance activities were both recognized as key to the development of total ownership cost strategies. The experts failed to come to a clear consensus that the cost of repairs is a key parameter as the interquartile range is 1.5. When asked for clarification, P7 posited, The cost of repairs may be considered part of the maintenance cost and I did not consider it separately. P7 posited, Corrective and preventive maintenance can be rolled up into operating cost in the early stages of the life cycle. As more data is gathered over the life cycle the more granular the analysis can be. Assumptions, Scope, and Limitations and Delimitations of the Study The qualitative research study was conducted using a modified Delphi method to gather expert opinion, insight, experience, and judgment. The panel of experts was

130 selected based upon the criteria provided in Table 1. The opinions of the experts were used in predicting the advantages, disadvantages, key performance parameters, and barriers to the development and implementation of total ownership cost strategies. The factors were deemed necessary to develop a decision-making framework to provide the aerospace industry with a means to define, develop, and facilitate the implementation of cost-saving methods. The study was based on the following assumptions. First, it was assumed there is value added for military and commercial leaders to implement total ownership cost strategies throughout the entire life cycle of complex systems. Next, no universal framework is applicable for all total ownership cost applications, but rather the development of a framework of best practices will aid decision makers in trade-off analysis and in identifying complex system total ownership cost key performance parameters. Finally, the theorists and practitioners that participated in the study were considered experts by their peers. Each assumption was validated in the study. The scope of the study was limited to the development of a decision-making framework that may be used in the military and commercial aerospace industry. The framework included the entire life cycle of complex systems. Four limitations were outside the control of the researcher. The first limitation is in defining the best practice as the most desirable and most useful total ownership cost practice or strategy as provided by the panel of experts. The second limitation is the provision that a complex system is a system that performs a function or set of functions in interaction with other systems in the overall performance of a process (Kirshbaum, 2002). The third limitation is the study was confined to complex systems used in military and commercial aerospace applications.

131 The fourth limitation was the experts in the study included individuals, both academics and practitioners, who possessed skills, knowledge, and experience in the development and procurement of complex systems used in the commercial or military aerospace industry and were considered by their peers to be influential leaders and decision makers. The review of research and literature identified the need for a decision-making framework to objectively guide the total cost of ownership of complex systems over the entire product life cycle. The topic is representative of an area of decision making best resolved by establishing the consensus of a diverse group of qualified experts. Participants in the modified Delphi technique came to a consensus concerning many relevant cost drivers and best practices necessary to develop total ownership cost strategies. The study enabled the modified Delphi panel to combine their expertise into a comprehensive set of conditions for the development of complex system total ownership cost strategies. In the study the Internet was used as the medium to direct questions and to ensure anonymity of the expert panel members. The Internet format was effective in decision making, learning, and promoting interaction among participants, which was supported in the literature (Brunk, 2003; Cabannis, 2001; Cook, 2005; Cross, 2002; Custer et al., 1999; MacDonald, 2003; Simon, 2006). In the development of the research an interactive Internet domain was created at http://totalownershipcost.net/Forums, where panel members were encouraged to participate in open discussion. The addition of the anonymous conferencing option was intended to improve on the participant interactivity and alleviate the disadvantage identified by MacDonald (2003), who reported participants in an Internet-based modified

132 Delphi study wished for more interaction, finding that this media were not responsive enough (p. 133). The delimitations of the study were established in the three-round modified Delphi study with 23 expert participants, each with special knowledge, skills, expertise, and experience in the development of total ownership cost initiatives. At the beginning of the Delphi process, a demographic survey and a letter of informed consent were e-mailed to perspective panel members. The survey was used to ensure the diversity of the panel and verify qualification criteria in the field of expertise were met. The modified Delphi study was preceded by a pilot study with 8 expert participants to establish validity and usability of the survey questions (Creswell, 2002; Franklin & Hart, 2006; Simon, 2006). The sample group of pilot study experts was identified and selected from a larger population. The pilot study group was asked to nominate potential expert panelists in an effort to reduce any potential researcher bias (Custer et al., 1999). During the first round of questions in the modified Delphi survey, the panel of experts was sent a list of open-ended questions. The panel was provided the opportunity to construct additional considerations that may contribute to the survey. The responses were collated by the researcher and provided as feedback to the panel in the second round in the form of a 5-point Likert-type scale. Following the second round, where the responses from the panel were below the group median, those statements were discarded as exhibiting little value or consensus, and feedback was again provided in the form of a 5-point Likert-type type scale for Round 3. Each panelist was asked to reconsider previous answers where the resultant descriptive statistics and interquartile range

133 indicated no consensus. The participants were asked to provide a rationale for their response outside of the group mean. The biases in the study were in the field of occupation of the participants, the applicability to complex systems, and the geographical location. The results of the study may not be consistent among purely military, commercial, or academic participants from other areas of the country for applications on other than complex systems. The panel of experts for the modified Delphi study was asked to apply their responses specifically to complex systems. The members of the panel represented diverse backgrounds and placed different values on such key performance parameters as operational availability and operating environment. The next section provides the recommendations and implications of the study for total ownership cost decision makers in the future development and implementation of total ownership cost strategies, complex system supply chain control and management, and comprehensive cost-benefit analysis. The recommendations also provide suggestions for consideration in the expansion, duplication, or modification of future studies. Specific recommendations are provided for operations managers, marketing, finance and academia. Recommendations From the results obtained in the modified Delphi study and following a review of the literature, it is apparent the development and implementation of total ownership cost philosophies is a viable strategic business initiative. The expert panel confirmed no single solution will meet the needs for all total ownership cost models (Ferrin & Plank, 2002). Overall, the panelists believed in every case the initial purchase price, the failure rate or

134 reliability, and the operating and maintenance cost must be factored into the assessment of total ownership cost. The cost associated with system availability and the operating environment is important to decision makers, yet extremely difficult to define. The experts indicated if a system operating in a commercial application is not available for use when needed, it may be a trivial annoyance, yet the same system, when unavailable for use in a military application, could result in the loss of life. As the product progresses through the life cycle, relevant cost data should be gathered and the total ownership cost assessment should be updated. A checklist should be developed as a guideline for the implementation of strategies throughout the phases of the life cycle because each product requires individual total ownership cost assessment. The checklist may provide the decision maker with a number of factors to consider from a strategic, tactical and operational standpoint during the entire lifecycle of the system. The checklist is provided as an example only based upon the combined opinions given by the panel of experts to the importance of key assessment and implementation parameters. It should be noted that each program has unique requirements that must be taken into account during individual phases of the product lifecycle and not all elements may be required. An example of a total ownership cost implementation checklist is provided in Appendix E. Operations Management It is recommended total ownership cost be considered an equally important metric and critical decision factor for operations managers as the design of goods and services, management of quality, development of process, facility layout and location strategies,

135 human resources, supply chain management, inventory, scheduling, and maintenance (Heizer & Render, 2006). Operations managers should be encouraged to set realistic and achievable total ownership cost targets for all stakeholders at the beginning of a program. The targets should be an ongoing part of the program review metrics, tracked and updated as the product progresses through its life cycle. The metrics to be tracked should be program specific, for example, dollars per flying hour, dollar per mile, and miles per gallon. It is recommended that contract, statement of work, and requirements documentation should be written to reflect the target metrics. Extended warranties and no-cost repair penalties should be imposed in the event the supplier fails to meet the target metrics. The suppliers that show continued failure to meet total ownership cost metrics should be required to provide no-cost product redesigns until the targets are met. Marketing Marketing strategies should be developed to reflect the competitive advantage of reliable products with the lowest total ownership cost. Extended warranties provide a perception of quality and total ownership cost assessments provide the basis for effective implementation. The development of total ownership cost as a competitive advantage requires linking reliability and product quality perception to overall life-cycle performance advantages. Finance A model should be developed for making total ownership cost assessments that can be refined over the product life cycle as more data are made available. By adopting total ownership cost philosophies there is a potential to save billions of dollars (Cubbin,

136 2004; Hurkens et al. 2006; Williams & Graveline, 2000). Ferrin and Plank (2002) indicated purchasing managers can use a core set of cost drivers and additional tailored drivers for computation in a particular purchase situation and a computation model is needed for making the assessments. There are many barriers to the successful implementation of total ownership cost strategies, including a natural resistance to change, the failure to communicate needed data across business units, and scepticism that total ownership cost projections provide accurate forecasting information. Total ownership cost and financial calculations and assessments should be kept as simple as possible and based on historical information to avoid scepticism concerning the validity. Data collected during the entire product life cycle should be used to refine estimates and assumptions and to make trade-off decisions based on the data, not on initial purchase price alone. Academia Little data are available concerning educational programs for total ownership cost. The majority of the modified Delphi panel agreed formal training is required for the systematic implementation of total ownership cost strategies. Training facilitators must know what knowledge and skills are required to support awareness, assessment, and implementation of total ownership cost. Facilitators should assess the need and develop a curriculum to address the requirement for cognitive development. Framework and Application The literature review and the data collected from the panel of experts in the modified Delphi study provided emerging themes that practitioners apply when making total ownership cost assessments. The emerging themes and best practices form the

137 recommended framework to support leaders and decision makers in the development and implementation of complex system total ownership cost strategies. The recommended framework of best practices when considering total ownership cost strategies, benefits, barriers to implementation, key performance parameters, available methods, and required practitioner skill sets are listed in Table 4. Where possible, in the performance of a total ownership cost trade-off analysis, the assessment should be made using historical data. In lieu of available historical data assumptions and predictions may be used. Any assumptions should be applied equally on all data and updated when historical information becomes available. In a typical analysis, as more relevant system cost are provided or predicted, the assessment should be adjusted to reflect that additional data. A basic sample total ownership cost trade off analysis is shown in Table 5. The assessment includes either assumptions or actual information that the study group has identified as necessary in the performance of all complex system total ownership cost analysis. In the supplier trade-off analysis in Table 5, the number of complex systems or vehicle fleet size is 10, each system operates 360 hours annually, and the anticipated total program life cycle is 20 years. This basic trade-off analysis may be used as the basis for procurement decision making or to assess the return on investment that improved reliability or reduction in maintenance costs may provide.

138 Table 4 Best Practices and Characterization Best practice Make trade-off decisions related to performance, quality, and cost based on Characterization Ensure procurement is focused on long-term total cost of ownership criteria rather than on short-term acquisition cost. Business leaders must use total ownership cost assessment to

total ownership cost criteria make trade-offs and return on investment decisions. The decision maker is provided a method to predict the cost of achieving required performance and warranty commitments while providing a competitive advantage discriminator. The assessment method must remain consistent throughout the product life cycle. Establish cost targets, Total ownership cost targets must be developed early in the

metrics, and penalties early product life cycle. The targets must be specific to the in the program. program, measureable, achievable, repeatable, and based on a specified schedule. Penalties must be agreed to for failure to meet the targets and may include extended warranties, nocost repair, and product redesigns. Monitor metrics throughout Metrics must be defined and kept throughout the product life the product life cycle and revise the targets. cycle to validate successful achievement of target criteria. As product matures, more historical data are used to refine the total ownership cost assessment. The cost associated with the loss of operational availability and the effects of the

139 operating environment must be determined in the early stages of development and tracked throughout the life cycle. Initial assessment is based on product reliability, Keep total ownership cost assessments simple and uncomplicated. Make historical data available across

procurement, maintenance, business units or sectors of the firm to implement effective and operating cost. trade-off decisions. Share data obtained in development of the assessment to assist with analysis on similar programs. Avoid using estimates and predictions to limit business leader skepticism of the validity of the projections. Incorporate formal training Share knowledge gained from performing total ownership for the systematic implementation of total ownership cost strategies. cost assessments to improve the visibility, applicability, and credibility of future strategies. Good communication skills and the ability to work comfortably across multiple organizations are essential practitioner attributes. Plan to make total ownership cost a product priority. Total ownership cost strategies must be implemented effectively to provide cost-effective decision making criteria. The organizational culture must accept total ownership cost as a product priority. Make historical data available across business units or sectors of the firm to implement effective trade-off decisions. Share the data and successes obtained in the development of the assessment to assist with analysis and awareness on similar programs.

140 Table 5 Sample Total Ownership Cost Trade off Analysis Predicted or historical data provided Initial purchase price Mean time between failures Basic per unit cost Predicted failure rate reliability assessment Cost of each corrective maintenance repair Preventative maintenance cost per year per unit Sourcing cost Assume 30% of initial purchase price Assume 1% of initial purchase price Identification of the need and $100,000 initial acquisition effort Operating cost per hour Operational environment Resale value End-of-life disposal cost Trade-off analysis Anticipated corrective maintenance returns per year Corrective action cost over program life returns per year * program life * repair cost $1,350,000 $1,200,000 fleet size * operating hours / predicted failure rate 4.5 3 Unknown at this time Unknown at this time Unknown at this time Unknown at this time 0 0 0 0 0 0 0 0 $100,000 $500 $300 $15,000 $20,000 Supplier A $50,000 800 hours Supplier B $60,000 1,200 hours

141 Table 5 (Continued) Predicted or historical data provided Preventative maintenance cost over program life Initial acquisition cost fleet size * program life * preventive cost fleet size * initial purchase price Total ownership cost sum of maintenance and acquisition cost Total ownership cost per unit Total ownership cost per operating hour $295,000 $819 $286,000 $794 $1,950,000 $1,860,000 $500,000 $600,000 Supplier A $100,000 Supplier B $60,000

Future Studies There is a need for continued study of total cost of ownership and life-cycle cost analysis because of the rapid pace of change in technology and the highly competitive business environment. Product developers strive to increase or maintain market share, competitive advantage, and product discriminators. Consumer perception of product quality and reliability will continue to effect the purchase decision. The expansion of total cost of ownership awareness will necessitate cultural change to shift from making decisions based on initial purchase price to total cost of ownership. The need for comprehensive cost analysis over the entire product life cycle will require further study. Future studies may consider the need to develop total ownership cost as an academic course of study. The review of the available literature did not uncover evidence of a curriculum to develop the concepts of total ownership cost strategies, supply chain

142 control and management, and comprehensive cost-benefit analysis. A program for the professional development of total ownership cost strategies for program and project management, operations strategies, finance and procurement, or for the systems engineering field may be of value. Facilitators may provide total cost of ownership as an important consideration of the decision maker, the business leader, the product developer, and the consumer. There is a need to develop the best practices that lead to the improved success and the ongoing development of total ownership cost strategies. Future studies should not be limited to complex systems and may benefit from limiting the study to either commercial or military application. The growing interest in extended product warranties, product quality and reliability, and total cost of ownership as competitive advantages will encourage the continued publication of literature regarding existing and future trends and successful implementation methods.

143 REFERENCES American Airlines. (2005). American Airlines aircraft maintenance procedures: American Airlines 2005 annual report. Retrieved July 16, 2006, from http://www.aa.com/content/amrcorp/corporateInformation/facts/fleet.jhtml AMTSybex. (2004). The defense industry in the United Kingdom: Maintaining the edge. Retrieved July 15, 2006, from http://www.amt-sybex.com Anbari, F. T. (2003). Earned value project management method and extensions. Project Management Journal, 34(4), 12-23. Retrieved August 13, 2005, from ProQuest database. Arnum, E. (2004). Selling customer care. Warranty Week, 9, 1-5. Retrieved October 20, 2004, from http://www.warrantyweek.com/archive/ ww20040921.html Ahgren, P., & Wierda, F. (2007, September). Reducing total cost of ownership. Surface Mount Technology, 2(9), 24-26. Retrieved October 11, 2007, from ProQuest database. Bailey, J. T., & Heidt, S. R. (2003, November). Why is total cost of ownership (TCO) important? Darwin Magazine, 1-3. Retrieved June 18, 2006, from http://www.darwinmag.com/ read/110103/question74.html Barringer, P. (2001, July). Maintainability. Retrieved September 24, 2004, from http://www.barringer1.com/jul01prb.htm Barringer, P., & Weber, D. P. (1996). Life cycle cost tutorial. Retrieved July 9, 2004, from http://www.barringer1.com/pdf/lcctutorial.pdf

144 Bayus, B. L. (1985). Word of mouth: The indirect effects of marketing efforts. Journal of Advertising Research, 25(3), 31-40. Retrieved October 16, 2004, from ProQuest database. Beaudreau, M. W., & Naegle, B. R. (2005). Total ownership cost considerations in key performance parameters and beyond. Defense AR Journal, 12, 109-123. Retrieved June 14, 2006, from www.infotrac.com Bemmaor, A. C. (1995). Predicting behaviour from intention-to-buy measures: The parametric case. Journal of Marketing Research, 32, 176-191. Retrieved October 24, 2005 from ProQuest database. Bergquist, T. M. (1996). An assessment of the operational and financial impact on companies of quality awards in the United States. Doctoral dissertation, University of Oregon, Eugene. (This is a title; I think there should be capitols in it?) Retrieved December 2, 2004, from http://80wwwlib.umi.com.ezproxy.apollolibrary.com/dissertations/fullcit/9714483 Bhutta, K. S., & Huq, F. (2000, April 1-4). A comparative study of the total cost of ownership and analytical hierarchy process approaches to the supplier selection problem. Proceedings of the 11th Annual Conference of the Production and Operations Management Society, POM-2000, San Antonio, TX. Retrieved June 25, 2006, from http://www.poms.org/Meeting2000/SB6E2.doc Blair, B., & McKenzie, J. (2004). Raytheon-new challenges, new solutions, and documented results. Defense AR Journal, 11(2), 194-222. Retrieved November 28, 2004, from ProQuest database.

145 Boulding, W., & Kirmani, A. (1993). A consumer-side experimental examination of signaling theory: Do consumers perceive warranties as signals of quality? Journal of Consumer Research, 20, 111-123. Retrieved December 2, 2004, from ProQuest database. Brunk, B. (2003). A framework for understanding the privacy space. Doctoral dissertation, University of North Carolina, Chapel Hill. Retrieved July 10, 2005, from ProQuest database. Butterworth-Hayes, P. (2002, November). Maintenance costs tumble. Retrieved July 15, 2006, from http://www.janes.com/business/news/jwa/jwa021112_1_n.shtml Byrne, P. M. (2004). Making warranty management manageable. Logistics Management 43(8), 31-32. Retrieved October 18, 2004, from ProQuest database. Cabannis, K. (2001). Counselling and computer technology in the new millennium: An Internet Delphi study. Retrieved August 20, 2006, from ProQuest database. Capability Maturity Model Integrated. (2006). What is CMMI: CMMI benefits. Retrieved July 18, 2006, from http://www.sei.cmu.edu/cmmi/general/general.html#info Carsten, T. & Erickson, B. (2007, February). Gain a competitive edge by calculating total ownership costs. Power Engineering 111(2), 32-38. Retrieved October 12, 2007, from ProQuest database. Clark, J. J., & Johnson, T. D. (2004, October) A primer on acquisition logistics. Retrieved November 28, 2004, from http://www.almc.lee.army.mil/alog/issues/MayJun02/MS757.htm

146 Clayton, M. (1997). Delphi: A technique to harness expert opinion for critical decisionmaking tasks in education. Educational Psychology, 17, 373-386. Retrieved August 20, 2006, from ProQuest database. Continental Airlines. (2005). Excellence the world over: Continental Airlines 2005 annual report. Retrieved July 16, 2006, from http://www.continental.com/ company/investor/ docs/continental_ar_2005.pdf Cook, V. S. (2004). Exploration of leadership competencies needed by future Illinois community college presidents: A Delphi study. Retrieved July 10, 2005, from ProQuest database. Cooper, D. R., & Schindler, P. S. (2003). Business research methods (International ed.). New York: McGraw-Hill. Creswell, J. W. (2002). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. Upper Saddle River, NJ: Merrill Prentice Hall. Cross, D. D. (2002). Determining standards for sources of free information on the Internet for inclusion in academic library holdings by 2010. Doctoral dissertation, East Tennessee State University. Retrieved July 13, 2005, from http://etdsubmit.etsu.edu/etd/theses/available/etd-1105102-120319/unrestricted/ CrossD111802b.pdf Crow, K. (2004, November). Achieving target cost: Design-to-cost objectives. Retrieved May 27, 2006, from the Association of International Product Marketing and Management Product Management Network Web site:

147 http://www.aipmm.com/html/newsletter/archives/ cat_new_product_development.php Cubbin, K. (2004). Airline economics: Is there a formula for success? Retrieved June 26, 2006, from http://kcubbin.tripod.com/id36.html Custer, R. L., Scarcella, J. A., & Stewart, B. R. (1999). The modified Delphi technique: A rotational modification. Journal of Vocational and Technical Education, 15(2), 111. Retrieved August 27, 2006, from http://scholar.lib.vt.edu/ejournals/JVTE/ v15n2/custer.html Dalkey, N. C. (1969, June). The Delphi method: An experimental study of group opinion. Retrieved June 11, 2005, from http://www.rand.org/pubs/research_memoranda/ RM5957/ Dalkey, N. C., & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts. Management Science 9, 458-467. Retrieved June 11, 2005, from http://www.rand.org/pubs/research_memoranda/RM727 Denson, W. (1999). A tutorial: PRISM. Journal of the Reliability Analysis Center, 1, 6. Retrieved September 24, 2004, from http://rac.alionscience.com/ pdf/3RD_Q1999.pdf Dikolli, S. S., & Sedatole, K. L. (2004). Deltas new song: A case on cost estimation in the airline industry. Issues in Accounting Education, 19, 345-359. Retrieved June 27, 2006, from http://find.galegroup.com Edmunds Inc. (2006, October). True Cost to Own. Retrieved October 16, 2006, from http://www.edmunds.com/reviews/tco/2006/index.html# Ellram, L. M., (1993). Total cost of ownership: Elements and implementation.

148 International Journal of Purchasing and Materials Management. 29(4), 3-11. Retrieved March 1, 2003, from ProQuest database. Ellram, L. (1994). A taxonomy of total cost of ownership models. Journal of Business Logistics, 15, 171-191. Retrieved July 9, 2004, from ProQuest database. Ellram, L. M. (1995, May). Total cost of ownership: An analysis approach for purchasing. International Journal of Physical Distribution & Logistics Management, 25(8), 4-23. Retrieved July 6, 2004, from ProQuest database. Ellram, L. M., & Siferd, S.P. (1998). Total cost of ownership: A key concept in strategic cost management decisions. Journal of Business Logistics, 19, 55-84. Retrieved July 6, 2004, from ProQuest database. Faherty, V. (1979). Continuing social work education: Results of a Delphi survey. Journal of Education for Social Work, 15, 12-19. Federal Aviation Administration. (2006). Adjusted operational availability. Retrieved October 30, 2004, from http://www.faa.gov/about/plans_reports/performance/ performancetargets/details/Op_Availability_(35_OEP).doc Ferrin, B. G., & Plank, R. E. (2002). Total cost of ownership models: An exploratory study. Journal of Supply Chain Management, 38(3), 18-30. Retrieved October 16, 2004, from ProQuest database. Franklin, K. K., & Hart, J. K. (2006). Influence of Web-based distance education on the academic department chair role. Educational Technology & Society, 9, 213-228. Retrieved August 15, 2006, from www.ifets.info/journals/9_1/18.pdf

149 Fraser, J. (2005). Production and the product lifecycle: In perspective. Manufacturing Business Technology, 23(3), 36. Retrieved March 21, 2006, from ProQuest database. Gartner, Inc. (2003, April). Why total cost of ownership (TCO) matters. Retrieved June 18, 2006, from https://k12tco.gartner.com/home/homepagepromo/files/ TCO_Overview.pdf Gartner, Inc. (2006). Removing the barriers for communicating total cost of ownership to the enterprise. Retrieved June 18, 2006, from http://www.gartner.com/4_decision_tools/measurement/measure_it_articles/july0 1/mit_removing_barriers1.html Geiger, D. R. (1999). Practical issues in cost driver selection for managerial costing systems. The Government Accountants Journal, 48(3), 32-39. Retrieved November 8, 2006, from ProQuest database. Goel, P. S. (1998). Integration of quality, reliability, and durability engineering concepts in product development. Doctoral dissertation, Wayne State University. Retrieved December 2, 2004, from http://80wwwlib.umi.com.ezproxy.apollolibrary.com/dissertations/fullcit/9827205 Goering, P. A. (1983). Consumer learning and brand loyalty when product quality is unknown. Doctoral dissertation, University of Illinois at Urbana-Champaign. Retrieved December 2, 2004, from http://80wwwlib.umi.com.ezproxy.apollolibrary.com/dissertations/fullcit/8324560 Goldratt, E. M., & Cox, J. (1984). The goal: A process of ongoing improvement (2nd ed.). Great Barrington, MA: North River Press.

150 Gordon, A. (2004). Closing the gap: Linking marketing decisions with consumer decision making. Retrieved June 27, 2006, from http://us.acnielsen.com/pubs/ 2004_q4_ci_closing.shtml Gunaydin, M. H. (2006). Impact of information technologies on project management functions. Doctoral dissertation, Illinois Institute of Technology. Retrieved June 26, 2006, from ProQuest database. Gwinner, C. (2006). Infosurv white paper: 5-point vs. 6-point Likert scales. Retrieved September 4, 2006, from http://www.infosurv.com/images/ Likert_Scale_Debate.pdf Hakola, T., & Horning, M. (2004, December). Managing the product lifecycle: A strategy to increase productivity. Tooling and Production, 12, 26-29. Retrieved November 25, 2005, from http://www.manufacturingcenter.com/tooling/archives/ 1204/1204managing_product.asp Hall, D. (2005). Whats the big deal about ROI and TCO. Learning & Leading with Technology, 33, 48. Retrieved June 18, 2006, from Academic OneFile database. Hart, G.W. (1978). DoD should share its experience in life cycle costing. Defense Management Journal, 14, 16. Retrieved October 30, 2004, from ProQuest database. Heizer, J. H., & Render, B. (2006). Operations management (8th ed.). Upper Saddle River, NJ: Prentice Hall. Himmelfarb, P. A. (1999). The importance of good product definition. Retrieved September 26, 2006, from http://www.bizbasics.com/html/ product_definition.html

151 Hurkens, K., Van der Valk, W., & Wynstra, F. (2006). Total cost of ownership in the services sector: A case study. Journal of Supply Chain Management, 42, 27-28. Retrieved June 18, 2006, from ProQuest database. Hysom, J. L. (1979). Life cycle costing in energy conservation analyses. Journal of Property Management, 44(6), 332-333. Retrieved October 30, 2004, from ProQuest database. International Council on Systems Engineering. (2003). Systems engineering body of knowledge. Retrieved June 18, 2006, from http://g2sebok.incose.org/ Ittner, C. D. (1993, February). The economics and measurement of quality costs: An empirical investigation. Retrieved May 21, 2003, from ProQuest database. Johnson, N. S. (2005). PLM is here to stay, so get ready. CaDalyst 22(3), 14-15. Retrieved May 21, 2006, from ProQuest database. Jones, J., Hunter, D. (1995, August) Qualitative research: Consensus methods for medical and health services research. BMJ Journal 311, 376-380. Retrieved June 26, 2006, from http://www.bmj.com/cgi/content/full/311/7001/376 Kaminski, P. G. (1995). Investing in tomorrow's technology today. Defense Issues, 10 (46), 1-22. Retrieved September 26, 2006, from http://www.defenselink.mil/ speeches/1995/t19950328-kaminski.html Kaye, M., Sobota, M., Graham, D. R., & Gotwald, A. L. (2000, Fall) Cost as an independent variable: Principles and implementation. Acquisition Review Quarterly, 353-371. Retrieved June 27, 2006, from http://www.dau.mil/ pubs/arq/2000arq/kaye.pdf

152 Kilcourse, B. (2007, April). Show me the money. Chain Store Age, 83 (4), 54-54. Retrieved October 12, 2007, from ProQuest database. Kirshbaum, D. (2002). Introduction to complex systems. Retrieved July 26, 2006, from http://www.calresco.org/intro.htm Kludze, A. K. P. (2004). Engineering of complex systems: The impact of systems engineering at NASA. Retrieved October 7, 2006, from ProQuest database. Konstantinos, A., Nikos, D., & Dimitra, M. (2002). Can perceptions of service quality predict behavioural intentions? An exploratory study in the hotel sector in Greece. Managing Service Quality, 12(4), 224232. Retrieved October 30, 2004, from ProQuest database. Korman, J. (2002, August). Turning requirements into product definition. Retrieved August 13, 2005, from http://www.cooper.com/newsletters/2002_08/turning_requirements_into_product _definition.htm Kothari, A., & Lackner, J. (2005). A value based approach to management. Virtual Strategist, 7, 1-11. Retrieved June 27, 2006, from http://www.virtualstrategist.net/Issue7/7-3-1.HTM Kotler, P. (2000). Marketing management (Millennium ed.). Upper Saddle River, NJ: Prentice Hall. Linstone, H., & Turoff, M. (1975). The Delphi method: Techniques and applications. Retrieved August 31, 2006, from http://www.is.njit.edu/pubs/delphibook/delphibook.pdf

153 Louden, J. J. (2006). Naval Sea Systems Command: Total ownership cost. Retrieved June 10, 2006, from www.acquisition.navy.mil/navyaos/content/download/1395/6741/file/toc.pdf Ludwig, B. (1997). Predicting the future: Have you considered using the Delphi methodology? Journal of Extension, 35(5), 1-2. Retrieved June 10, 2006, from http://www.joe.org/joe/1997october/tt2.html MacDonald, J. E. (2003). An exploration of the use of an online Delphi method within an advocacy group. Retrieved July 10, 2005, from ProQuest database. MacMillan, D., Vella, M. (2007, January). Hybrids cost effective over long haul. Business Week Online 9, 11-11. Retrieved October 12, 2007, from ProQuest database. Material Management Training Centre. (1993). Life cycle costing course. Vanier, Canada: Canadian Armed Forces. McNeilly, M. (1996). Sun Tzu and the art of business: Six strategic principles for managers. New York: Oxford University Press. Meckback, G. (1998). Study reveals organizations blas about TCO. Computing Canada, 24(43), 15-16. Retrieved October 16, 2004, from ProQuest database. Military definitions. (2004). Retrieved October 10, 2004, from http://www.fas.org/news/ reference/lexicon/des.htm Milligan, B. (1999). Total cost of ownership proves illusive. Purchasing 127(3), 22-23. Retrieved October 16, 2004, from ProQuest database.

154 Moore, C. W. (2003, September 2). OPNAV Instructions 3000.12: Operational availability of equipments and weapons systems. Retrieved June 3, 2004, from http://neds.nebt.daps.mil/Directives/3000_12a.pdf Moore, E. M., & Shuptrine, F. K. (1993). Warranties: Continued readability problems after the 1975 Magnuson-Moss Warranty Act. Journal of Consumer Affairs, 27, 23-37. Retrieved December 2, 2004, from ProQuest database. Mostefai, S., Bouras, A., & Batouche, M. (2005). Effective collaboration in product development via a common sharable onthology. International Journal of Computational Intelligence, 2(4), 206-212. Retrieved May 21, 2006, from ProQuest database. Murphy, M. K., Lamping, D. L., McKee, C. M., Sanderson, C. F., Askham, J., & Mateau, T. (1998). Consensus development methods and their use in clinical guideline development. Health Technology Assessment, 2(3), 1-100. Retrieved August 24, 2005, from http://www.hta.nhsweb.nhs.uk/fullmono/mon203.pdf NASA. (1995, June). National Aeronautics and Space Administration (NASA) Systems Engineering Handbook SP-610S. Retrieved October 7, 2006, from http://ldcm.gsfc.nasa.gov/library/Systems_Engineering_Handbook.pdf National Communications System, Technology & Standards Division. (2000). Telecommunications: Glossary of telecommunication terms (Federal Standard 1037C). Retrieved July 26, 2006, from http://www.its.bldrdoc.gov/fs-1037/fs1037c.htm National Defense Industrial Association. (2006). Systems engineering division. Retrieved August 31, 2006, from http://www.ndia.org/Template.cfm?Section=Divisions

155 Neuman, L. W. (2003). Social research methods: Qualitative and quantitative approaches (5th ed.). Boston: Allyn & Bacon. OHara, J. (2007, June). Maturing workforce impacts technology purchasing. Government Procurement, 15(3), 46-47. Retrieved October 12, 2007, from ProQuest database. Padnos, G. (2006, May). Cost of ownership considerations. Surface Mount Technology Magazine, 1-5. Retrieved June 27, 2006, from http://smt.pennnet.com/Articles/ Article_Display.cfm? Parasuraman, A. Zeithaml, V. A., & Berry, L. L. (1985, Fall). A conceptual model of service quality and its implications for future research. Journal of Marketing, 49(4), 41-51. Retrieved December 2, 2004, from ProQuest database. Pett, M. A. (1997). Nonparametric statistics for health care research. Thousand Oaks, CA: Sage. Pollard, R., & Tomlin, M. (1995). The use of expert teachers to improve education. Education, 116, 3-9. Retrieved June 18, 2006, from http://calbears.findarticles.com/p/articles/ mi_qa3673/is_199510/ai_n8710283 Raasch, C. (2006). Americans talk a lot, and much of its positive. Gannett News Service McLean, 1. Retrieved September 17, 2006, from ProQuest database. Ramsey, M. (2005). The challenges and benefits of activity based costing: An overview. Carpe Diem, 1, 1-6. Retrieved from http://dlibrary.acu.edu.au/research/ CarpeDiem/pages/journal1/paper2.htm Reed, P. (2002). True cost to own: Revealing the hidden cost of car ownership. Retrieved June 27, 2006, from http://www.edmunds.com/advice/specialreports/articles/

156 59897/article.html Rollo, C. (2006). Some assembly required: Building an integrated CRM solution to delight and retain customers. Retrieved October 24, 2005, from http://www.crm2day.com/library/EpFkZFVEupeOneMtvr.php Rowe, G., & Wright, G. (1999). The Delphi technique as the forecasting tool: Issues and analysis. International Journal of Forecasting, 15, 353-375. Retrieved July 13, 2005, from www.elsevier.com/locate/jiforecast Rust, R. T., & Oliver, R. L. (2000). Should we delight the customer? Academy of Market Science, Journal, 28, 86-94. Retrieved October 30, 2004, from ProQuest database. Saur, K., Donato, G., Flores, C., Frankl, P., Jensen, A. A., Kituyi, E., et al. (2003). Draft final report of the LCM definition study: UNEP/SETAC life cycle initiative. Retrieved May 28, 2006, from http://www.sph.umich.edu/riskcenter/jolliet/Saur%202003.pdf Schaeffer, M. (2003, October 13). Designing and assessing supportability in DoD weapon systems: A guide to increased reliability and reduced logistics footprint. Retrieved June 3, 2004, from http://www.acq.osd.mil/log/logistics_materiel_ readiness/organizations/lpp/assetts/fle/logguide_final%205_29_03.pdf Seawright, K. K. (1994). Product quality in the automobile industry: The United States and Japan. Doctoral dissertation, University of Utah. Retrieved December 2, 2004, from http://80wwwlib.umi.com.ezproxy.apollolibrary.com/dissertations/fullcit/950303312/2/04 Sheskin, D. J. (2007). Handbook of parametric and nonparametric statistical procedures. Boca Raton, FL: Chapman & Hall/CRC.

157 Sieck, L. (2004, March). Experts on project management: Procurement management. PMI GovSIG Magazine, 1822. Retrieved July 15, 2006, from http://www.pmigovsig.org/magazines/PMI%20GovSigMag%20(March%202004).pdf Simon, M. K. (2006). Dissertation and scholarly research: Recipes for success. Dubuque, IA: Kendall Hunt. Smith, M. E., & Zsidisin, G. A. (2002, June), Early supplier involvement at MRD. Best Practices in Purchasing & Supply Chain Management, 5(4), 1-15. Retrieved October 24, 2005, from http://www.sba.pdx.edu/faculty/msmith/Practix602.pdf Society of Automotive Engineers Inc. (1993). Reliability, maintainability, and supportability guidebook. (3rd. ed.).Warrendale, PA: Society of Automotive Engineers Inc. Solis-Galvan, L. E. (1998). An exploratory study of the relationship between quality management and manufacturing competitive capabilities. Retrieved December 2, 2004, from http://80wwwlib.umi.com.ezproxy.apollolibrary.com/dissertations/fullcit/9916209 Solomon, P. J. (2004, May-June). Integrating systems engineering with earned value management. Retrieved August 13, 2005, from www.dau.mil/pubs/dam/05_06_2004/sol-mj04.pdf Southwest Airlines. (2005). Southwest Airlines 2005 annual report. Retrieved July 16, 2006, from http://www.southwest.com/investor_relations/annual_reports.html Space and Missile Systems Center. (2004). Retrieved October 30, 2004, from http://ax.losangeles.af.mil/ Srinivas, T. (2002). Enhancing supply decisions through the use of efficient marginal

158 costs models. Journal of Supply Chain Management, 38(4), 4-11. Retrieved October 24, 2005, from ProQuest database. Statistics Canada. (2006). Range and quartiles. Retrieved September 23, 2006, from http://www.statcan.ca/english/edu/power/ch12/range.htm Stundza, T. (2006). Focus is on total cost of ownership. Purchasing, 135(3), 50-51. Retrieved June 18, 2006, from www.infotrac.com Swink, M. (2006). Building collaborative innovation capability. Research Technology Management, 49(2), 37-48. Retrieved May 21, 2006, from ProQuest database. Tan, S. J., Lee, K. S., & Lim, G. H. (2001). Warranty and warrantor reputations as signals of hybrid product quality. The European Journal of Marketing, 35, 110-132. Retrieved December 2, 2004, from ProQuest database. Tesfatsion, L. (2004). Complex adaptive systems. Retrieved July 26, 2006, from http://www.econ.iastate.edu/classes/econ308/tesfatsion/casdef.pdf Theng Lau, G., Razzaque, M. A., & Ong, A. (2003). Gatekeeping in organizational purchasing: An empirical investigation. The Journal of Business & Industrial Marketing, 18, 82-104. Retrieved October 24, 2005, from ProQuest database. Tibben-Lembke, R. S. (1998). The impact of reverse logistics on the total cost of ownership. Journal of Marketing Theory and Practice, 6(4), 5161. Retrieved October 30, 2004, from ProQuest database. Twining, J. (1999). A naturalistic journey into the collaboratory: In search of understanding for prospective participants. Retrieved August 20, 2006, from ProQuest database.

159 UOP. (2004). Reliability, availability, inspection, and equipment support services. Retrieved October 30, 2004, from ProQuest database. U.S. Department of Defense. (1992, May). Operating and support cost estimating guide. Retrieved October 22, 2004, from http://www.dtic.mil/pae/ U.S. Department of Defense. (2003a, May). The Defense Acquisition System (Directive No. 5000.1). Retrieved October 22, 2004, from http://www.dtic.mil/whs/directives/corres/pdf2/d50001p.pdf U.S. Department of Defense. (2003b, May). Operation of the Defense Acquisition System (Instruction No. 5000.2). Retrieved October 22, 2004, from http://www.dtic.mil/whs/directives/corres/pdf2/i50002p.pdf U.S. Naval Air Systems Command. (2003, July). Total ownership cost. Retrieved July 1, 2006, from http://www.navair.navy.mil/toc/ Veenstra, J. (2000, May). Customers hold the secret to product reliability. iieSolutions, 38-41. Retrieved June 27, 2006, from http://www.iienet.org/membersonly/ backissues/ 500veenstra.pdf Warhoe, S. P. (2004). The basics of earned value management. AACE International Transactions CSC.07, pp. 1-10. Retrieved August 13, 2005, from ProQuest database. Warranty Week. (2004, September). Warranty claims and accruals in financial statements. Retrieved September 18, 2004, from http://www.warrantyweek.com Whittaker, K. (2005). Five keys to deploying activity based costing. Armed Forces Comptroller 50, 6-8. Retrieved August 13, 2005, from ProQuest database.

160 Wicklein, R. C. (1993). Identifying critical issues and problems in technology education using a modified-Delphi technique. Journal of Technology Education, 5, 10451064. Retrieved August 16, 2006, from http://scholar.lib.vt.edu/ejournals/JTE/ v5n1/wicklein.jte-v5n1.html Williams, J. F., & Graveline, W. (2000, August). Defense acquisitions: Air Force operating and support cost reductions need higher priority. Report to the Chairman and Ranking Minority Member, Subcommittee on Readiness and Management Support, Committee on Armed Services, U.S. Senate, pp. 1-38. Retrieved July 15, 2006, from http://www.gao.gov/archive/2000/ns00165.pdf Wilson, E.J., & Woodside, A.G. (1992). Marketing new products with distributors. Industrial Marketing Management, 21, 15-21. Retrieved October 24, 2005, from ProQuest database. Woudenberg, F. (1991). An evaluation of Delphi. Technological Forecasting and Social Change, 40, 131-150. Retrieved August 20, 2006, from ProQuest database.

161 APPENDIX A: LETTER OF INFORMED CONSENT

Dear Participant, My name is Russ King. I am a doctoral candidate at the University of Phoenix, in the Doctorate in Business Administration degree program. My faculty mentor is Marilyn Simon, PhD. The purpose of this qualitative research study is to develop a decision making framework for the total ownership cost management of complex systems. This framework could be used as a starting point to develop implementation strategies for procurement and manufacturing decision makers. This framework could also identify key performance parameters, best practices, and barriers to implementation. You have been recognized as an expert in military and commercial aerospace complex system procurement or development. My goal is to assemble at least 20 recognized experts like yourself to be part of a Delphi Panel. Participation in the study is voluntary and is limited to persons 18 years of age or older. The study will involve experts, like yourself, who will offer advice on the future and current state of total ownership cost management. The benefit to participating in this study is that the information may help the profession to improve the development and implementation of total ownership cost strategies. As a panelist, you will be asked to respond by providing your expert opinion during at least three rounds of survey questions. During the first round you will be asked to respond to a list of questions. The response to these initial questions will be used to develop the Delphi study and the second round of questions. Each member of the expert panel will have the opportunity to contribute additional questions for the survey. During the second round of questions, you will be asked to respond to the collective group response received during the first round. You will be asked to rate your responses on a Likert-type scale. The researcher will then assemble the responses and calculate the group mean for each question. During the third round, you will be provided with the mean of the group response calculated from the Likert-type scale. You will be given the opportunity to revise your response if your previous answers varied from the groups mean. Alternatively, you will be asked to provide a rationale for your answers that are outside the mean. This rationale will provide the researcher with information regarding why an experts response differs from the majority of the group. In some circumstances, this wide array of expert opinion is highly effective in generating a range of alternative solutions to issues and problems facing the researcher. By signing this letter, you are agreeing to participate in the three rounds of the modified Delphi technique. Your anonymity will be preserved. The results of the Delphi study will be provided to each of the participants. The dissertation will be filed in the library at the

162 University of Phoenix, where you will have access to the published copy. You may receive a copy of this study upon its completion. All responses will be stored on a USB drive that will be kept in a locked box. Only the researcher will have the keys to the box. After 7 years, the information will be destroyed. Again, your identity and any data gathered will remain confidential.

Sincerely, Russel J. King

, will be a panelist in the modified I, Delphi study to help develop a decision making framework for the total ownership cost management of complex systems. My signature below indicates that I voluntarily desire to participate in this study. I understand that I am free to withdraw from the study at any time, without penalty or loss of benefit to myself. By signing this form I acknowledge that I understand the nature of the study, the potential risks to me as a participant, and the means by which my identity will be kept confidential. My digital signature on this form also indicates that I am 18 years old or older and that I give my permission to voluntarily serve as a participant in the study described. I realize that the results of this study may be published, but my name and title will not be used in any publication, and my identity will be kept confidential. If you agree to participate, please digitally sign this informed consent form, save the signed form, copy and paste it as an attachment and return it via email to the researcher. Thank you so much for your time and consideration.

Signature of Panelist

Date

163 RESEARCH QUESTIONS

1. 2. 3. 4. 5. 6.

What benefits, if any, are there in implementing total ownership cost strategies in complex system procurement, development, or manufacturing? What key performance parameters are necessary to assess total ownership cost? What methods are available, or could be available, and applicable for guiding complex system total ownership cost decision-making strategies? What are the barriers to the implementation and application of total ownership cost methods in procuring, developing, and manufacturing complex systems? What framework can be developed to characterize and organize total ownership cost methods across the entire life cycle of a complex system? What attributes or special knowledge is necessary to qualify a person as an expert in the development of complex system total ownership cost strategies.

164 APPENDIX B: TOTAL OWNERSHIP COST ASSESSMENT KEY PERFORMANCE FACTORS Capital Taxes Capability upgrades Coordination reviews Licensing Regulatory compliance Security related Diminishing manufacturing sources mitigation Marketing and capture Subcomponents and assemblies Business development Sourcing Spare parts and warehousing Preventive and corrective maintenance General and administrative Energy Testing and test equipment Warranty Safety Installation and tools Integration and lab support Manufacturing and tooling Design and development Raw materials Initial product Research and development Requirements definition Repair Operating Disposal Training Documentation Logistic support shipping System replacement Downtime

165 APPENDIX C: MAIN STUDY ROUND 3 RESULTS Seed Question 1 Rate the attributes or special knowledge required for the development of complex system total ownership cost strategies.
M Able to and comfortable with working across multiple organizations Able to communicate well with others Practical experience in operations and support (O&S) cost estimation Practical experience in the systematic implementation of TOC strategies Practical experience in systems engineering across the life cycle of complex systems Practical logistics management experience Practical experience in program management of complex systems Practical experience in the performance of trade off and return on investment analysis Formal training for systematic implementation of TOC strategies Practical experience with and understanding of financial models Practical experience in business, leadership, accounting and supply chain management 3.76 0.89 2 5 0 3.86 3.76 3.76 0.73 0.77 0.70 3 2 3 5 5 5 0 0 0 3.90 0.83 2 5 0 4.00 3.90 0.77 0.62 3 3 5 5 1 0 4.10 0.54 3 5 0 4.14 0.79 2 5 1 4.33 4.19 0.58 0.98 3 3 5 5 1 0 SD Range Min Max IQR

166

Histogram (with Normal Curve) of C2 by C1

2 16 1 3 4 2 5 6 3 2 3 4 4 5 6
1 Mean 4.333 StDev 0. 5774 N 21 2 Mean 4.190 StDev 0. 9808 N 21 3 Mean 4.143 StDev 0. 7928 N 21

Frequency

16

4 Mean 4.095 StDev 0. 5390 N 21 5 Mean 4 StDev 0. 7746 N 21 6 Mean 3.905 StDev 0. 6249 N 21 7 Mean 3.905 StDev 0. 8309 N 21 8 Mean 3.857 StDev 0. 7270 N 21 9 Mean 3.762 StDev 0. 7684 N 21 10 Mean 3.762 StDev 0. 7003 N 21 11 Mean 3.762 StDev 0. 8891

16

10

11

0 2 3 4 5 6 2 3 4 5 6

C2 Panel variable: C1

167

Seed Question 2 Rate the elements that may be necessary to develop a framework to characterize and organize total ownership cost methods across the entire life cycle of a complex system
M Plan to make TOC a priority from the very beginning of a development program Establish a comprehensive system life cycle work breakdown structure (WBS) Review both the factors being tracked and the data coming out of these systems Formalize a process to pull necessary data from various systems to review/analyze Assign cost targets 4.00 3.71 0.71 0.64 3 3 5 5 1.0 1.0 4.05 0.59 3 5 0.0 4.14 0.79 3 5 1.5 4.86 0.36 4 5 0.0 SD

Range
Min Max IQR

Histogram (with Normal Curve) of C2 by C1

3 1 4 2 5 6 3 20 15 10

1 Mean StDev N 2 Mean StDev N 3 Mean StDev N 4 Mean StDev N 5 Mean StDev N 4 0.7071 21 3.714 0.6437 21 4.048 0.5896 21 4.143 0.7928 21 4.857 0.3586 21

Frequency

5 4 20 15 10 5 0 3 4 5 6 5 0 3 4 5 6

C2 Panel variable: C1

168

Seed Question 3 Rate the importance of each of the following barriers to the implementation and application of TOC strategies.
M TOC is not a program review element and linked with the success of the development Procurement focus on lowest purchase price Data is not available to make TOC strategic decisions Procurement budgets are based on purchase price and do not consider TOC The lack of long-term system planning Acquisition and sustainment budgets are separately identified, acquired, and managed Inconsistency between those responsible for TOC and those able to make strategic decisions Federal budgeting rules that restrict the funding of activities over multiple years Relevant data is not shared across firms or business units 3.71 3.67 0.90 1.02 2 2 5 5 1.0 2.0 4.05 0.67 3 5 0.5 4.10 0.77 3 5 1.5 4.29 4.24 0.85 0.54 3 3 5 5 1.5 1.0 4.38 4.38 4.33 0.67 0.80 0.86 3 2 2 5 5 5 1.0 1.0 1.0 SD

Range
Min Max IQR

169

Histogram (with Normal Curve) of C2 by C1

2 16 1 3 2 4 5 6 3
1 Mean 4. 381 StDev 0.6690 N 21 2 Mean 4. 381 StDev 0.8047 N 21 3 Mean 4. 333 StDev 0.8563 N 21

Frequency

16

4 Mean 4. 286 StDev 0.8452 N 21 5 Mean 4. 238 StDev 0.5390 N 21 6 Mean 4. 095 StDev 0.7684 N 21 7 Mean 4. 048 StDev 0.6690 N 21 8 Mean 3. 714 StDev 0.9024 N 21 9 Mean 3.667 StDev 1.017 N 21

16

0 2 3 4 5 6 2 3 4 5 6

C2 Panel variable: C1

170

Seed Question 4 Rate the methods that may be used to perform a TOC assessment
M Historical review of past performance Activity based cost management for better visibility into cost drivers Cost as An Independent Variable (CAIV) analyses Effective use of earned value management system (EVMS) metrics Weighted system evaluation method / Kepner-Tregoe (K-T) decision analysis 3.05 0.80 2 5 0 3.24 0.83 2 5 1 3.81 3.76 0.87 0.77 2 2 5 5 1 1 4.10 SD 0.83

Range
Min 2 Max 5 IQR 1

Histogram (with Normal Curve) of C2 by C1

2 1 3 2 4 5 6 3 16 12 8

1 Mean StDev N 2 Mean StDev N 3 Mean StDev N 4 Mean StDev N 5 Mean StDev N 3.238 0.8309 21 3.048 0.8047 21 3.762 0.7684 21 3.810 0.8729 21 4.095 0.8309 21

Frequency

4 0 2 3 4 5 6

16 12 8 4 0 2 3

C2 Panel variable: C1

171

Seed Question 5 How would you rate the benefits of implementing TOC strategies in each of the following areas?
M Provides a method to make effective decision based upon total system cost Provides a measure of cost to achieve required performance over the entire system life cycle: Provides quantitative data that may be used to predict cost Provides a measure of the return on investment necessary to assess cost reduction initiatives Provides quantitative data that may be used to assess warranty commitments 3.29 0.85 2 5 1 4.05 0.74 3 5 1.5 4.14 4.05 0.65 0.67 3 3 5 5 1 0.5 4.33 0.80 3 5 1 SD

Range
Min Max IQR

Histogram (with Normal Curve) of C2 by C1

2 1 3 2 4 5 6 3 12 9 6

1 Mean StDev N 2 Mean StDev N 3 Mean StDev N 4 Mean StDev N 5 Mean StDev N 4.048 0.7400 21 3.286 0.8452 21 4.048 0.6690 21 4.143 0.6547 21 4.333 0.7958 21

Frequency

3 4 12 9 6 3 0 2 3 4 5 6 5 0 2 3 4 5 6

C2 Panel variable: C1

172 Seed Question 6 From the following list of factors, rate the importance of those that must be used in the performance of an assessment of TOC on ALL complex system programs.
M Operating cost Maintenance cost The anticipated failure rate or system reliability Cost of repairs Corrective maintenance cost Preventive maintenance cost Initial acquisition cost Sourcing costs The characteristics and environment the system is operating in Operational availability Cost of quality Manufacturing cost Cost of product reliability improvements Spare parts and inventory cost Design cost Development cost Sub-components and assemblies costs Redesign cost Raw material cost 4.67 4.29 4.14 4.10 4.00 4.00 3.95 3.81 3.81 3.76 3.76 3.71 3.62 3.57 3.52 3.52 3.48 3.43 3.38 SD 0.58 0.56 0.79 0.89 0.84 0.63 0.97 0.75 0.98 1.18 0.89 0.85 0.80 0.87 1.03 0.93 0.93 0.87 0.97

Range
Min 3 3 2 2 2 3 2 3 2 2 2 3 3 2 2 2 2 2 2 Max 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 IQR 1 1 1 1.5 1.5 0 2 1 2 2 1.5 1.5 1 1 1.5 1 1 1 1

173

Histogram (with Normal Curve) of C2 by C1

1 1 2 3 4 2 5 6 3 1 2 3 4 4 5 6 5 16 8 0
1 Mean 4.667 StDev 0.5774 N 21 2 Mean 4.286 StDev 0.5606 N 21 3 Mean 4.143 StDev 0.7928 N 21 4 Mean 4.095 StDev 0.8891 N 21 5 Mean 4 StDev 0.8367 N 21 6 Mean 4 StDev 0.6325 N 21 7 Mean 3.952 StDev 0.9735 N 21 8 Mean 3.810 StDev 0.7496 N 21 9 Mean 3.810 StDev 0.9808 N 21 10 Mean 3. 762 StDev 1. 179 N 21 11 Mean 3.762 StDev 0.8891

16

10

Frequency

8 0

11

12

13

14

15

16 8 0

16 8 0 1 2 3

16

17

18

19

C2 Panel variable: C1

174 APPENDIX D: MAIN STUDY ROUND 3 FRIEDMAN NONPARAMETRIC STATISTICS TEST RESULTS Seed Question 1 Rate the attributes or special knowledge required for the development of complex system total ownership cost strategies.
Friedman Test: C3 versus C2 blocked by C1 S = 10.43 DF = 10 P = 0.40 S = 13.62 DF = 10 P = 0.19 (adjusted for ties)

Est C2
1 2 3 4 5 6 7 8 9 10 11

Sum of
Histogram of C3
120

N
21 21 21 21 21 21 21 21 21 21 21

Median Ranks
4.09 4.18 4.09 4.09 4.00 4.00 4.00 3.91 3.91 3.82 3.91 156.0
100

142.0 142.0 132.5 127.5 119.0 120.5 116.5 113.0 109.5 107.5
Frequency

80 60 40 20 0

3 C3

Grand median = 4.00

175 Seed Question 2 Rate the elements that may be necessary to develop a framework to characterize and organize total ownership cost methods across the entire life cycle of a complex system.
Friedman Test: C3 versus C2 blocked by C1 S = 23.42 DF = 4 P = 0.00 S = 35.38 DF = 4 P = 0.00 (adjusted for ties)

Est C2 1 2 3 4 5 N 21 21 21 21 21 Median 5.00 4.00 4.00 4.00 4.00

Sum of Ranks 91.5 64.0 57.5 58.0 44.0

Frequency

Histogram of C3
50

40

30

20

10

4 C3

Grand median = 4.20

176 Seed Question 3 Rate the importance of each of the following barriers to the implementation and application of TOC strategies.
Friedman Test: C3 versus C2 blocked by C1 S = 12.84 DF = 8 P = 0.12 S = 16.99 DF = 8 P = 0.03 (adjusted for ties)

Est C2
1 2 3 4 5 6 7 8 9

Sum of

Histogram of C3
80 70 60 Frequency

N
21 21 21 21 21 21 21 21 21

Median Ranks
4.22 4.22 4.22 4.22 4.22 4.00 3.89 3.44 3.56 120.5 121.5 118.5 115.5 108.5 102.0 95.5 80.5 82.5

50 40 30 20 10 0 2 3 C3 4 5

Grand median = 4.00

177 Seed Question 4 Rate the methods that may be used to perform a TOC assessment.
Friedman Test: C3 versus C2 blocked by C1 S = 14.57 DF = 4 P = 0.01 S = 18.95 DF = 4 P = 0.00 (adjusted for ties)

Est C2
1 2 3 4 5

Sum of
40

Histogram of C3

N
21 21 21 21 21

Median Ranks
4.00 3.80 4.00 3.20 3.00 76.5 73.0 68.5
Frequency

30

20

10

52.5 44.5
0 2 3 C3 4 5

Grand median = 3.60

178 Seed Question 5 Rate the benefits of implementing TOC strategies in each of the following areas.
Friedman Test: C3 versus C2 blocked by C1 S = 14.98 DF = 4 P = 0.01 S = 18.67 DF = 4 P = 0.00 (adjusted for ties)

Est C2
1 2 3 4 5

Sum of
Histogram of C3
50

N
21 21 21 21 21

Median Ranks
4.20 4.00 3.80 4.00 3.00 75.5
Frequency

40

69.0 64.5 67.0 39.0

30

20

10

3 C3

Grand median = 3.80

179 Seed Question 6 From the following list of factors, rate the importance of those that must be used in the performance of an assessment of TOC on ALL complex system programs.
Friedman Test: C3 versus C2 blocked by C1 S = 47.80 DF = 18 P = 0.00 S = 61.07 DF = 18 P = 0.00 (adjusted for ties)

Est C2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Sum of
Histogram of C3
140

N
21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21

Median Ranks
4.58 4.05 4.11 4.00 3.79 3.79 3.63 3.58 3.79 3.58 3.58 3.63 3.53 3.58 3.37 3.42 3.32 3.42 3.26 316.5 270.5
Frequency 120 100 80 60 40

251.5 245.0 227.0 227.5 234.0 207.5 210.5 209.0 210.0 198.0 183.5 180.5 171.5 170.5 163.0 163.0 151.0

20 0

3 C3

Grand median = 3.68

180 APPENDIX E: TOTAL OWNERSHIP COST FRAMEWORK CHECKLIST EXAMPLE

Total Ownership Cost Framework Checklist System Life Cycle Phases Production and Deployment System Development and Documentation Technology Development Refinement and Disposal

Strategic Methods Develop and implement on the job training programs Assign cost targets and penalties for failure to meet targets Link TOC to program success and risk factors Identify and prioritize long range cost goals Review TOC factors and data being tracked Tactical Methods Plan to make TOC a program priority Prioritize purchase decisions and long term plans Implement a process for performing TOC assessment Formalize a process to pull necessary cost development data Plan for supplier TOC assessment Implement cost reduction initiatives based on historical data Operational Methods Implement a comprehensive system lifecycle work breakdown structure Implement a method to share data across business units Track and evaluate design and development cost against targets Track and evaluate support costs against targets Track and evaluate operation, support and warranty cost Track and evaluate obsolescence and product improvement cost Process and review past performance data

Operation and Support

Concept Refinement