Scribd Logo
Hochladen

Willkommen bei Scribd!

  • Marrin Intelligence Analysis Structured Methods or Intuition AIJ 2007 5

    Hochgeladen von

    Stephen Marrin
    2K Ansichten10 Seiten
    Exploring the value of structured methods in intelligence analysis

    Copyright

    © Attribution Non-Commercial (BY-NC)

    Verfügbare Formate

    PDF, TXT oder online auf Scribd lesen

    Stufen Sie dieses Dokument als nützlich ein?

    Sind diese Inhalte unangemessen?

    Dieses Dokument melden
    Exploring the value of structured methods in intelligence analysis

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Als PDF, TXT herunterladen oder online auf Scribd lesen
    Markieren Sie unangemessene Inhalte
    2K Ansichten10 Seiten

    Marrin Intelligence Analysis Structured Methods or Intuition AIJ 2007 5

    Hochgeladen von

    Stephen Marrin
    Exploring the value of structured methods in intelligence analysis

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Als PDF, TXT herunterladen oder online auf Scribd lesen
    Markieren Sie unangemessene Inhalte
    von 10
    Intelligence Analysis: Structured Methods or Intuition?! by Stephen Marrin evently—-after the release ofthe 9/11 Commission BR isc Comision prs stn imount of money and attention has been devoted to the identification and development of structured analytic techniques for capturing, presenting, and evaluating data more effectively; deriving patterns ftom the data; understanding what those patterns mean; and communicating them to others. But less attention has been focused on the utility of structured methods, their fit with the reigning analytic culture, and when they should be used, or not used, by analysts, This article addresses these issues using medical practices to illustrate how the same kinds of dynamics can play out in othe fe, DEFINING TERMS: STRUCTURED METHODS VERSUS INTUITION those techniques which have a formal or structured ‘methodology that is visible to external observers. Examples of structured methods include: key assumption check, analysis of competing typotheses, devil's advocacy, Feam A/Team B, alternative futures. ? S: what are structured analytic methods? They are The primary value of analytic techniques or structured methods is that they provide a way to account forthe analytic judgment; an analytic “audit trail” as it were. ‘According to Rob Hubal—a researcher for RT Inemational—their use provides a mechanism for verifying or documenting “analysis so that any ouside ‘observer can understand the analyte Basis of conclusions."> Allan Schwartz—a teacher of structured techniques throughout the intelligence community—makes a similar point when he observes that “the great virtue” of analytic techniques is that they “make the analytic process transparent.”* While transpareney in terms of methodology may be more useful as a way to structure analytic thinking rather than share with consumers, in a global sense this transparency, according to Schwartz facilitates collaboration, permits meaningful review by managers, and deeper understanding by customers. Given the possibility for failure and the high stakes associated with many judgments, transparency is essential, Without structured techniques, itis virtually impossible to understand the complete bases for critical analytic judgments.” With an American Intelligence Journal Page? analytic audit trail, analysts and their colleagues can discover the sources of analytic mistakes when they occur and evaluate new methods or new applications of old methods. In this way, structured methods make it possible to advance analytic tradecraft “The great virtue” of analytic techniques is that they “make the analytic process transparent.” Systematic analytic techniques are also a way for the intelligence analyst to approximate the scientific method in terms of analytic processes. The scientific method is a relatively simple process to understand in theory, but very ‘complicated to apply effectively to real world problems, which are frequently detailed and messy, So analytic techniques are a short-hand way to approach certain, problems, to make the study of them more methodical. Yet the foundation of most, if not all, of them remains the scientific method. ‘The scientific method—like intelligence analysis—involves, description, explanation, and prediction. So in all fields ‘where the scientific method is applied, methodologists tell practitioners to use good data that is reliable, and validate it if possible. When describing the characteristics of things, decide if the categories are nominal, ordinal, interval, oF ratio, and derive inferences accordingly. When explaining, account for as many independent variables as you can; rank-order them based on how much they affect the dependent variable. Test alternative hypotheses about causation against each other to find out which explains the greatest amount of change in the dependent variable. And so on. The most basic analytic techniques will be those that link or display data so that they can be interpreted or communicated more easily. On the other hand, the more complicated analytic techniques—such as Bayesian analysis—are those that provide assistance in deriving meaning from the accumulated data, In contrast to structured methods are those techniques that are not structured; frequently identified as “intuition.” According to Robert Folker, “non-structured methodology ‘Summer 2007 is intuitive, Inution is a feeling or instinct that does not use demonstrative reasoning processes and cannot be adequately explained by the analyst with the available cevidence.”* He goes on to say that “intuition plays an inescapable role in analysis.” Yet at the same time he argues that “structured methods have significant and ‘unique valve (over intuition) in that they can be easily taught to other analysts as 2 way to structure and balanee their analysis. Iti difficult, if not impossibe, 1s teach an intelligence analyst how to conduct accurate intuitive analysis, Intuition comes with experience.” A problem with Folker’s definition, however, is that it equates intuition with lack of structure, which is nat necessarily the case.” Intuition can be structured, but pethaps in Ways that are not visible to outside observers Folker has since modified his distinction between structured methods and intuition, arguing that the “real difference between the two basic analytie processes is that intuition is an invisible analytic process, and the structured approach is a visible process.” Folker goes on to say that if this distinction is corres, then a comparison between “intuitive” (or invisible) and “visible” analytic approaches is more appropriate than that between structured methods and intuition.” Accordingly, when this paper uses the terms structured versus intuition, it is emphasizing the visible or verifiable aspects of the process versus the invisible or non- verifiable aspects ofthe process rather than how structured versus unstructured the analytic approach might be. ‘The question to be addressed here is bow much of a role should structured methods play in intelligence analysis. ‘The structured versus intuitive approach to intelligenc analysis is something that has been debated in analytic circles for decades, According to Fotker, “a long-standing debate exists within the Intelligence Community about whether more should be invested in structured ‘methodologies to improve qualitative intelligence analysis. ‘At the heart of this controversy is the question of whether intelligence analysis should be accepted as an art (depending largely on subjective, intsitive judgment) or a science (depending largely on structured, systematic, analytic methods) How much of a role should structured methods play? According to Folker, proponents of intelligence analysis as ‘a art “contend that qualitative intelligence analysis deals ‘with an infinite number of variables that are impossible to Swonmer 2007 ‘operationalize because they cannot be adequately quantified or fully collected... Because in many cases the variables are so coinplex, countless, and incomplete, attempting t0 ‘analyze them using scientific methods is pseudo-science. ‘Therefore, any altempt to make predictions based on quantifying these variables is futile. (As a result, these proponents) argue that qualitative intelligence analysis is an art because it is an intuitive process based on istinc, education, and experience.” By way of contrast, proponents of intelligence analysis as a science “assert that science is a necessary tool to use when conducting qualitative analysis. They argue that, athough itis impossible to consider every variable when conducting analysis, one can identify key variables and weigh their importance, And although much may be unknown, idemifying what is known and analyzing it scientifi an effective approach....(because) scientific methods hep analysts determine the relevancy of information and form conclusions, a process that analysts do not perform well on their own.” ‘Afier evaluating each argument, Folker concludes that intelligence analysis is both an art and a science. As he puts it, “The fallacy in the art or science debate may be the “either/or” proposition. If qualitative intelligence analysis is not exclusively an art nor a science, then it may best be considered a combination of both intuitive and scientific method.” So now that terms are defined, the next question is: whic is used more by intelligence analysts; structured methods or intuition? FAILING TO USE STRUCTURED METHODS ‘reat world of intelligence analysis assume that it involves rigorous application of structured methodologies. This perspective is best captured by New York Times columnist David Brooks in an article that atitcized CIA’s analysis as being too structured; how its analysis relies on ‘Sophisticated modeling’ and ‘probability calculations” that results in a“Talse scientism’ that fils to take into account some of the intangibles that are important in determining the actions of an adversary.” And in fact, agvotding to Folker, “many proponents of exploiting scientific methodologies to ad intelligence analysis assume that these methods are regularly used i real-world analysis.” However, Folker goes onto say that “this assumption that intelligence analysts use these various scientific approaches is based not on empirical evidence, but om the belief tnst experience alone does not make an Ss outside observers with litle experience in the American Intelligence Journal expert, and that an expert must have a tool for structuring, knowledge to efficiently solve a problem.” ‘The problem with this assumption, though, is that it is inaccurate. Even though there are over 200 analytic methods that intelligence analysts could choose from,” the intelligence analysis process frequently involves intuition rather than structured methods. As someone who worked at CIA from 1996 to 2000, I possess firsthand knowledge of the kind of analytic approaches used at the time. While | ‘was there, the reigning analytic paradigm was based on ‘generalized intuition; an analyst woufd read a lot, come up with some analytic judgment, and send that judgment up the line without much focus on either the process involved in coming to thst judgment, or making that process transparent to others. No one I knew—except for maybe the economic analysts—used any form of structured analytic process that was transparent to others. No quantitative methods; no special software; no analysis of competing hypotheses; not even link charts The intelligence analysis process ‘frequently involves intuition rather than structured methods. Others have also made similar observations. Folker has said that “structured methodologies are severely neglected” in the intelligence community." In addition, Randy Pherson, a former CIA analyst and National Intelligence Officer who currently teaches analytic techniques across the community, fas said that he “often tel(s) students thatthe traditional CIA method of analysis..(jnvolves three phases): Read as much as you have time to read that day; Think about it and suck an answer out your thumb; Write it down in as ersp a manner as possible.”” ‘This intuitive approach to intelligence analysis—the dominant analytic approach—does not leave much room for structured methodologies. Pherson went on to say that “unfortunately, at CIA most of our energy and on-the-job training as analysts traditionally has gone into phase three—learning how to capture the essence of our analysis in paragraph or page. Substantial resources also have beeen devoted to phase one, but we remain woefully behind ‘what technology offers. And, until recently, we hav largely ignored the need (or used the excuse we don't have the time) to develop the necessary skills to ensure more rigor and scientific method in our analytic process."* Granted, this kind of anecdotal evidence cannot be ‘generalized across the entire CIA—or across multiple ‘organizations because itis likely that certain analytic American Intelligence Journal techniques were used by certain kinds of analysts for specialized purposes. For example, most counterterrorism and counter-narcoties analysts probably ether used or knew about using link charts or social network analysis as a way to display or represent connections between targets of interest and perhaps other analytic techniques were used in other parts of the agency. But the analysts I worked with didn’t use them at all. In other words, intuition frequently dominated the actual process of intelligence analysis instead of more structured methodologies. | was initially appalled at the C1A’s lack of structured analytic methodologies, because I prefér some kind of rigor as a foundation to analysis. So finding out who used analytic methodologies was a particular interest of mine. ‘And I ran into—and identified with—methodologists who advocated greater rigor in analysis or those who experimented with innovative techniques. Foremost among these were Jack Davis and alternative analysis; Stanley Feder and FACTIONS or Policon and other kinds of modeling: the folks at Strategic Assessments Group who ‘worked on scenario-based simulations: and Carole Dumaine and the Global Futures Project, which at the time ‘ended to support future scenario work, among other things.” But fora variety of different reasons, including some good ones, few analysts embraced these techniques. As Folker observes, these approaches were “rejected by analysts ‘because the scientific methods were thought tobe too narrowly focused and not relevant to the questions the intelligence analysts were addressing"? He then quotes Richards Heuer as saying that most analysts tended "to be skeptical of any form of simplification iaherent in the application of probabilistic models."* He also points out that “the complexity of some scientific methodologies (such as Bayesian analysis) prevents them from being regularly exploited by most intelligence analysts.” In addition, he observed that “opponents of the scientific approach....denounce the amount of time it takes to scientifically analyze a problem.” As he points out, “under the accelerating pressures of time, intelligence analysts feel that structured analytical approaches are too cumbersome.” This observation is supported by Arthur Hulnick, a professor at Boston University who spent 35 ‘years as an intelligence analyst and manager, who has said that “methodologists discovered that their new technigues ‘were unpopular with analysts because they had so time to absorb these systems in the face of tight deadlines. Other reasons for the failure by analysts to adopt structured techniques include increased accountability and unproven benefits, In terms of accountability, Alan Schwartz points ‘out that i is natural for analysts to prefer “to have their Summer 2007 intuitive process unexamined.”** Mistakes can be more ceasily explained if the process used is opaque, and credit can be taken for getting the right result for the wrong, reason, Other analysts failed to adopt the techniques because their benefits remained unproven to a skeptical audience. {As Folker puts it, imtelligence analysts “are not convinced that (structured methodologies) will improve their analysis." In an environment of constant change and new analytic cools and techniques being sold—both figuratively and literally—as the next “big thing,” many analysts adopted a cautious if not skeptical posture towards any new tool, technique, or methodology. This skepticism was heightened by the fact thatthe value of many new techniques or methods was asserted or argued rather than proven through scientific evaluation. The value of many new techniques or ‘methods was asserted or argued rather than proven. Exceptions do, of course, exist. For example, Stanley Feder ~a former CIA methodologist~ argues thatthe use of @ specific analytic model produced more precise forecasts than conventional intelligence analysis without sacrificing accuracy." He has pointed out thatthe use of a particular analytic model “helped avoid analytic traps and improved the quality of analyses by making it possible to forecast specific policy outcomes and the political dynamics leading to them.” Feder’s articles about analytic methodology are ‘unusual forthe intelligence community in that they involve an effort to evaluate anki document the effectiveness of a structured methodology against the status quo intuitive approach used by most analysts. Another example is Folker’s Masters thesis atthe Joint Military Intelligence College in which he showed that using one particular analytic method—structured hypothesis testing —produced better analysis than intuition." But few evaluative articles of this kind exist. As a result, the benetits—and limits—of ‘most analytic techniques have not yet been specified. In the end, once there are impediments to incorporating structured methods into how people do analysis, those impediments become embedded in the organizational cculture. Feder said that “despite the advantages of (the models) the vast majority of analysts do not use them.””” Feder speculates that the models are not used because “this kind of systematic analysis does not fit into an ‘organizational culture that sees an “analyst” as someone who writes reports, often evaluating and summarizing available information. In contrast, people who use models Summer 2007 Page 10 and quantitative techniques are considered “methodolagists.” Folker agrees, pointing out that “structured thinking is radically at variance with the way in which the human mind is in the habit of working. Most people are used to solving problems intuitively by trial and error. Breaking this habit and establishing a new habit of thinking is an extremely difficult task and probably the primary reason ‘hy attempts to reform intelligence analysis have failed in the past, and why intelligence budgets for analytical methodology have remained extremely small when compared to other intelligence functions.” But perhaps organizational culture can change if the use of structured methods can be demonstrated to provide clear Benefits to the practitioner, as Feder has attempted to do, And there, Richards Heuer has provided us with a little hope. In an article he wrote regarding the use of quantitative techniques in intelligence analysis, Heuer observed that "the intial attitude of country analysts toward our unconventional proposals typically ranged from skepticism to hostility. Equally typical, however, has been their post-project appraisal that the work was interesting and well worth doing.”* So perhaps itis possible for structured analytic techniques to improve analysis, although the fact that it has been almost 30 years since Heuer made that observation—and ‘ve are stll having the same discussion—does provide a certain cautionary note about prospects for changing culture. In the end, the debate continues between proponents of an a versus science approach to intelligence analysis, and ‘Whether or not structured analytic techniques are more effective than intuition. But the same kinds of debates exist in other fields and can be used to shed Hight on ways to resolve the current debate COMPARING INTELLIGENCE ANALYSIS TO MEDICAL DIAGNOSIS telligence analysis, there is also an art and a science to medicine as well as a similar discussion about the value of structured methods versus intuition. The comparison between intelligence analysis and medicine has run through the intelligence literature for almost 60 years, if not longer. In 1949, Sherman Kent wrote that “intelligence isa simple and self-evident thing. As an activity, itis the pursuit of a certain kind of knowledge Ina small way, itis what we all do every day. ... When & doctor diagnosis an ailment—when almost anyone decides J« as there is both an art and a science to American Intelligence Journal upon a course of action—he usually does some preliminary intelligence work." In 1983, historian Walter Laqueur examined the analogy at Tength, and argued that medicine is more an art than a seience because the process of diagnosis entails the use of Jjudgment as a means to address ambiguous signs and symptoms.” Laqueur also highlighted similarities between medicine and intelligence including similarities in analytic processes, pointing out that “the student of intelligence ‘will profit more from contemplating the principles of medical diagnosis than immersing himself in any other field. The doctor and the analyst have to collect and evaluate the evidence about phenomena frequently not amenable to direct observation. This is done on the basis of indications, signs, and symptoms. ... The same approach applies to intelligence.” As a result of the similarities, references to the medical analogy have recently become ‘much more prevalent in the intelligence literature, in particular Rob Johnston’s observation that the “practice of ‘medicine is an art and a tradecraft” and goes on to describe how medicine relies on a base of science that has net yet been created to support intelligence analysis." Doctor and the analyst have to collect and evaluate the evidence about phenomena frequently not amenable to direct observation. In medicine, the structured approach (o diagnosis is known as ‘differential diagnosis’ which involves a process used to distinguish one disease from another with very similar symptoms, For intelligence practitioners unfamiliar with this process, differential diagnosis is the medical equivalent to Analysis of Competing Hypotheses, Richards Heuer has said that his discussion of the ‘diagnosticity of evidence’ in his book “Psychology of Intelligence Analysis” was drawn, from the medical literature.” So intelligence analysts who use ACH are applying a technique is more or less equivalent fo what docters do when they diagnose patients, Good illustrations of differential diagnosis in practice can be found in the Fox TV show “House"—which is about a ‘master diagnostician—because it shows how hypotheses are developed and tested in a medical context, sometimes intuitively and sometimes rigorously. Frequently, however, physicians don’t work through this process in an observable way. Rather, they do so via structured ‘intuition,’ which may look to outsiders as if they were diagnosing patients off the top of their heads rather than through the use of structured methods. In many ceases, the intuitive approach is used in medicine because of American Intelligence Journal Page 11 the pressure on tite doctor to make a diagnosis and come up ‘with treatment options in a very short period of time. Sometimes doctors who work in large practices are limited to only a few minutes with each patient, This severely restricls how much time they can spend during the diagnostic process, which makes them resistant to using, ‘more structured methods. And those same pressures explain why intelligence analysts tend to resist incorporating structured methods into their work processes as well ‘There's an ongoing debate in the medical community over ‘he value of structured methodologies—just as there is in intelligence~-and some people have criticized the unstructured aspects of medical diagnosis—part of the art of medicine—for causing diagnostic failure. In medicine, the dominant approach to diagnosis is intuition frequently structured, but relying on the practitioner's knowledge, memory, and reasoning skills, and these competencies can vary greatly between individuals, For example, according to Jeffrey Pfeffer and Robert Sutton in an article in the Harvard Business Review, medical decisions are frequently based on “obsolete knowledge gained in school, long- standing but never proven traditions, patterns gleaned from experience, the methods they believe in and are most skilled in applying and information from hordes of vendors with products and services to sell." For example, Jerome Groopman—a professor at Harvard Medical School and author of the book “How Doctors Think” —has said that “conservatively about 15 percent of | all people are misdiagnosed” because of “errors in thinking” primarily involving shortcuts: As he says, “most doctors, within the first 18 seconds of seeing a patient, will...generate an idea in his mind (of) what's ‘wrong. And too often, we make whats called an anchoring mistake —we fix on that snap judgment” which he says, could be based on anything. He also describes other kinds ‘of cognitive errors that physicians make in the diagnostic process. And he points out that there is a movement afoot to make the diagnostic process more rigorous. Some have argued that medical diagnosis can be twrned from an art into a science if physicians use structured ‘methods to assist them in the process of making diagnoses. ‘According to Groopman, “To establish a more organized structure (for their diagnoses), medical students and. residents are being taught to follow preset algorithms and practice guidelines in the form of decision trees. ..The trunk of the clinical decision tree is a patient's major symptom or laboratory result, contained within a box. Arrows branch from the first box to other boxes, For example, a common symptom like “sore throat” would begin the algorithm, followed by a series of branches with Summer 2007 yes" of “no” questions sbout associated symptoms. 1s there a fever or not? Are swollen Iymph nodes associated with the sore throat? Have other family members suffered from this symptom? Similarly, a laboratory test like a throat culture for bacteria would appeat farther down the trunk of, ‘he tree, with branches based on " * answers 10 the results of the culture. Ultimately, following the branches to the end should lead to the correct diagnosis and therapy. Whether this kind of diagnostic process isan improvement over the intuitive approach hasn't yet been decided in medicine, either. Groopman thinks that this more structured approach to diagnosis may be effective for simpie problems, but “quickly falls apart when doctors need to think outside the box, when symptoms are vague, oF multiple and confusing, or when test results are inexact. In such cases—the kinds of cases where we most need a discerning doctor—algorithms discourage physicians from thinking independently and creatively. Instead of expanding a doctor's thinking, they can constrain i.” Algorithms discourage physicians from thinking independently and creatively. ‘This discussion about the use of structured methods in medicine is taking place within an even larger context of evidence-based medicine, which is a push to make medical Practice in general less intuitive; less of an art, and more of 4 science. For example, an evidence-based approach to medicine would involve the implementation of evaluation processes within existing diagnostic procedures. A couple ‘of years ago, the New York Times published an article describing how some radiologists had instituted a feedback Joop in their diagnosis of breast cancer which provided tem with the opportunity to track eases over tts find out Where their mistakes were, and use that knowledge to become better at diagnosing problems." I've been told by ‘medical professionals that this example is unysual—even in medicine—because of the rigor involved in the process Le these processes are found to be more effective, an argument could be made to incorporate them into existing diagnostic doctrine Medicine has an established doctrine-—a formalized process-—for diagnosing and treating patients; a doctrine that is taught to new physicians in medical school. Doctrine does not necessarily reflect “best practices,” although in an ‘optimal word it might, But doctrine atone may nex provide the answer to improving analytic quality across the board because there might not be a single best way t0 approach each specific problem, According to Hubal, “experts often don't agre. ‘Summer 2007 Page 12 ‘There isa goal (e.g. 10 diagnose a patient's presenttion) but two experts may go about acquiring information, forming relationships, and drawing eonelusions quite differently — shat is, they may use different structured approaches to reach the goal. A doctrinal process is simply “a way’ to go aboot the business tha s formally seta “the way’, even though there may be other perfectly valid ways to succeed. And doctrine must really be used as a guide, because initial conditions will vary and individuals will have to adapt. Yet even if doctrine has limits in application and as a result is not the only answer, it can still act as a mechanisin to inculcate best practices into new practitioners. So how might this discussion about the art and science of medi assist intelligence practitioners in improving practices in their own field? SHOULD INTELLIGENCE ANALYSIS BECOME MORE LIKE MEDICINE? diagnostic doctrine to become less like an art and ‘more like a science, the same discussion is taking place segarding intelligence analysis as well. Still an open question, however, is whether there 18 an equivalent kind of doctrine for intelligence analysis. Je as medicine is debating how to adjust its In 1987, Washington Platt observed that “in intelligence production as yet we find litle study of methods as such, Yet a systematic study of methods in any given field by an ‘open-minded expert in that field nearly always leads to worthwhile inoprovement." But it wasn’t until the mid 19903 that CTA’s Product Evaluation Staff put together its analytic tradecraft notes in an effort to formalize analytic doctrine, and in 1999 its Center for the Study of Intelligence released Richards Heuer's “Psychology of Unteltigence Analysis” which provides the articulation, rationale, and description of the technique known as Analyses of Competing Hypotheses, which js fast becoming 4 part of informal analytic doctrine Bur is there a mechanism to inculcate this doctrine into new practitioners? The medical profession has formal practices for developing and teaching doctrine; primarily ‘through medical schools and continuing education programs that are a required compotient of the medical profession, Medicine is a mix of the art and the science, which involves both professional knowledge as well as craftsmanship. But medicine also possesses educational requirements for entry into the profession, fm medicine, a doctor spends a substantial amount of time in school to learn some of the knowledge that has been accumulated thus far on medical science—in other words, how 10 diagnose and treat the various diseases and problems that American Intelligence Sownal affect human health—with skills development acquired through an apprenticeship program and on-the-job training Intelligence analysis has similar educational and apprenticeship requirements for proficiency, but lacks the infrastructure and formal requirements to acquire the requisite knowledge and skills. Intelligence analysis has, for the most part, been practiced more as a craft than a profession; or as Hubal has said, “the educational and experiential processes have not (yet) been formalized." As a reset, professional practices are still pretty rudimentary. If there is some benefit to making intelligence analysis more like a formal profession, then greater professionalization is necessary. Professionalization—or the process of shifting a profession from less formal to more formal practices—is a process that relates to the interaction ofthe art and the science in the field. Ifyou think about it a craftsman is really an artist whose skill i, developed through training and experience, while a professional is someone who has been formally educated in the ‘science’ of his or her field, and then uses that knowledge in an applied way. Crafts rely primarily on the skill ofthe individual practtioner—which does not improve very much from generation to generation without the accumulation of knowledge that the formalization of education in the field entails—while professions build on the knowledge of past practitioners and relay it to new professionals through their educational process. ‘There isa growing consensus that some kind of improved professionalization process is necessary, but an open question is the mechanism for doing so. Some of the pieces of this process are being put together at the national level by the staff of the Office of the Director of National Intelligence (ODND, as they try to define analytic ‘competencies and standards for intelligence analysts across the many analytic disciplines of the intelligence ‘community. Others~such as myself—are advocates for a less hierarchical approach through the efforts of a professional association, and a possible model could be the ‘American Medical Association (AMA). Medicine wasn't always a profession. 150 years ago it was actually a craft very much like intelligence analysis is (oday. As my co-author—Dr. Jonathan Clemente—has pointed out, the mechanism that pushed medicine from mostly a craft to mostly a profession was the American Medical Association through its effort from about 1850 to about 1910. We've argued that the most effective way to professionalize intelligence analysis is through an over- arching professional association modeted on the AMA.” This association would then work on developing the adoption of centralized knowledge accumulation efforts and American Intelligence Journal Page 13 the development of formal personnel practices such as: a structured selection process built around analytic competencies; training, education and development programs; performance standards; and a code of ethics. In audition, the creation of a centralized focal point for knowledge regarding best practices—or as Hubal pus it, “ “centr fr lessons learned” that maintains a repository of best structured practices, as well as meta-data describing when, why, how to apply those practices"** would enable intelligence analysis as an occupation to learn and improve over time in a way that itis currently unable to. Just as medicine is debating whether or not to become more structured, the intelligence community has been examining the value of structured methods in greater detail. According to Randy Pherson, there is “a growing realization that we have to instill more rigor into the analytic process and that we can no longer afford to say we are too busy to get it right" In addition, some writers have advocated a more methodological approach to intelligence analysis. Two of the more prominent are Rob Johnston's call for the incorporation of methodologists into teams of substantive ‘experts,® and Tim Smith's suggestion that to increase the rigor of analysis the community should create “knowledge factories” explicitly using scientific methodology to produce all-source finished intelligence *' This paper won the 2096 DNI Galileo essay contest, signaling the community's interest in making the analytic process more rigorous and structure. We can no longer afford to say we are too busy to get it right. But as analytic standards and doctrine ate developed, questions will be raised about whether to promote certain analytic techniques over others; whether to teach—or ‘mandate the use of—certain techniques such as ACH or link analysis. For example, parts of the intelligence community have been encouraging analysts 10 use more structured methods in their analysis. At the community level, the ODNI has created an “Analysis 101” course as a ‘way to introduce structured techniques to all new analysts, in the community *= Additional courses are being introduced at the agency level, Pherson observes that “the CIA has been engaged in a ‘major training effort to introduce structured analytic techniques into its work process including mandatory training of all analysis managers in the core techniques.” In addition, his company is teaching structured analytic techniques to analysts at various intelligence community organizations, including 1,400 FBI analysts in 2006, and is “engaged at several agencies in helping analysts apply ‘Summer 2007 these techniques to everyday challenges." Finally, the National Security Agency's David T. Moore has written a book on critical thinking which he used to develop a course 6m critical thinking and structured methodology that is being taught at a number of intelligence community organizations." Additional efforts to develop structured analytic techniques and teach them to practitioners are also underway. For example, the softsvare version of Richards Hieuer’s Analysis ‘of Competing Hypotheses is being adapted by CTA to accommodate multiple users. According to Heuer, itis being “designed for use in a collaborative environment...t0 facilitate collaboration across organizational boundaries, and through all the firewalls, within the Intelligence ‘Comnamity.” 1 is hoped thas the ability to collaborate ‘online in a common virtual workspace will “change how the interagency coordination process works."*” Additional ‘work on ACH—being done at Mereyhurst College—~is also ongoing.** Some might perceive the teaching of these techniques as just the next step in the professionalization process, To a degree it is. Knowledge advancement is an iterative process and its impartant to teach new practitioners the more structured cuting edge approaches 3s well as those which ane more traditional, such as intuition, Essentially, the use of structured methods, if only as a guide or framework for addressing a problem, is better than no structure at all, But before certain structured analytic techniques become de rigeur throughout the intelligence community, I think it is important to take a step back and ask how structured intelligence analysis should be, and whether or not that structure will actually lead to improved analysis. At this point—given the general intuitive approach that analysts use and the relative paucity of data showing that structured techniques would improve accuracy—f'm agnostic about the value of mandating use of more structured metitods,{ think itis important to teach them, because if there is value in their application then it would bbe good for practitioners 10 know how to use them. But people think in different ways, and until the intelligence community can demonstrate that these techniques are an improvement over intuitive models forall practitioners, I don’t think their use should be mandsted, Instead, more time, money, and effort should be devoted to developing the capacity to evaluate the utility ofthese approaches rather than just ta developing and teaching them, As Rob Johnston observed in 2003, “litle work has been done ‘comparing stractored techniques 10 intuition "* A step in the right direction was the January 2007 ODNI ‘conference on “improving intelligence analysis” intended to “examine a new way of doing business in which analytic Summer 2007 practices are based not on tradition, personal experience, or intuitive plausibility but on solid evidence of what works.” Institutionalizing this kind of effort would be even better, and a recommendation by Steve Rieber would accomplish this through the creation of @ National Institute for Analytic Methods—modeted on the National Institutes of Health—to sponsor research on the effectiveness of analytic, tools and techniques. If this kind of research infrastructare is created, i-witl be possible to gain knowledge—scientifically—about the effectiveness of different approaches,® rather than rely primarity on aneedotal evidence as is currently the case, And as Folker has observed, “if empirical evidence shows that such an improvement will take place, then analysts may consider it worth the investment of time, effort, and risk required on their part co cegularly explait structured methodalogies.”™ Fotker had it right when he observed shat “he basic research question remains unanswered: Will the use of structured methods improve qualitative imelligence analysis... If qualitative intelligence analysis isan art, then efforts to improve it should focus on measuring the accuracy of one's intuition, selecting those analysts with the best track record, and educating them to become experts ina given field. If, on the other hand, qualitative intelligence analysis isa science, then analysts should be ‘ined fo select the appropriate method for a given problem from a variety of sciemtfic methodologies and exploit it to guide them throwgh the analytical process In the end, since inteigence analysis is both an art anda science, that meas efforts to improve it should incorporate both intuition and structured methods.* But the same kind of rigor that advocates for structured methodologies argue should be used in intelligence analysis should also be applied to the evaluation of the utility of those very techniques. Stephen Marrin—a former analyst with the CIA anil the ‘congressional Government Accountability Office—Is an assistant professor in Mereyhurst College's Intelligence Smudies Department. He is a doctoral candidate atthe University of Virginia, and has writen many articles on various aspects of intelligence studies, including one that led to the creation of CIA University. In 2004, the National Journal described him as one of the country’s top ten ‘experts on intelligence reform. He is alsa a member of the editorial advisory board of the Inrernational Journal of Inyellgence and Counterintlligence, the editorial review board of the National Miltary Intelligence Association's American Intelligence Journal, 9 member of American Military University’s Inelligence Advisory Council, and on the Board of Directors ofthe International Association for Intelligence Education. Page 14 American Intelligence Journal Notes The core concepts behind this paper were derived from presentations given atthe Naval War College in March 2006 (Making Better Communicators”) and the Eighth Annval Meteyhurst College International Colloquium on Intelligence in June 2006 ("Non-Intelligence Analytical Approaches") 2 For more information on particular structured analytic techniques, se: “A Tradeeraft Primer: Structured Analytic “Techniques for Improving Intelligence Analysis." CIA Sherman Kent Sehool. Vol. 2. No. 2. June 2005; “Handbook of Analytic ‘Tools and Techniques.” Pherson Associates, LLC. 2005; Kristan 4. Wheaton, Emily F. Mosco, and Diane E. Chido. “The Analyst "s Cookbook.” Mereyhurst College Institut of Intelligence ‘Studies Press. 2006 » Rob Hubs Email. December 26, 2006. ‘Alan Schwartz. Email. 25 December 2006 * Robert Folker.“Inteligence Analysis in Theater Joint Intelligence Centers: An Experiment in Applying Structured Methods.” Occasional Paper Number Seven: Center for Suategic Intelligence Research, Joint Military iatelligence College. January 2000. 5 * Folker, 14, 7 Lam beholden to Rob Hubal for suggesting this distinction. This flue to carefully define the concept of intuition wit rigor is something that Reb Johnston also referenced ia “Foundations for Meta-Analysis: Developing a Taxonomy of | Intelligence Analysis Variables.” CIA Studies in Intelligence V. 4. No. 3. (2003) * Robert Folker. Email. 1S June 2007 » Fokker, 6 ' Folker, 6-7 citing Richard K. Bets, “Surprise, Scholasticism, and Sirtegy: A Review of Ariel Levite’s Intelligence and Strategic Surprises (New York: Columbia University Press, 1987)," Intemational Studies Quarterly 33, no. 3 (September 1989): 338, and John L. Peterson, “Forecasting: It's Not Possible.” Defense Intelligence Journal 3,N2 (Fall 1994): 37-38, " Folker, 9-10. " Folker, 13. David Brooks. The CIA: Method and Madness. The New York ‘Times. February 3, 2004, A 23, Folker, 11 'S Rob Johnston, Integrating Methodologiss into Teams of Substantive Expests. Studies in Intelligence. Vol. 47. No. 1 " Folker, 1a " Randy Pherson Email. December 24, 2006. » Randy Pherson Email. December 24, 2006. °* For more information on Jack Davis and his involvement with Altemative Analysis and other aspects of analytic methodology, see: hip:www.oss.nevextea/page! action=page_show&id=21 &module_instance=4 For more information on alternative analysis in general, sce: Roger Z. George. “Fixing the Problem of Analytical Mind-Sets: ‘Alternative Analysis." Intemational Journal of Intelligence and Couaterintelligence. Vol. 17. No. 3. (2004). 385-404, » Stanley A. Feder. “FACTIONS and Policon: New Ways to ‘Analyze Politics.” In Inside CLA’s Private World: Declassified Ailes from the Agency's Internal Journal, 1955-1992, ed, H. Bradford Westerfield, New Haven: Yale University Press. 274- 292. American Intelligence Journal Page 15 2 According (othe following website, the Strategic Assessment Group's “methodology ranges from the pooling of experts to the construction of seenarios.” Sec: hutp»/home.gwu.edu/~esialsf/Final%20Report%20- 620Fall%202003 pat ® For a dated description of the Global Futures Partnership, see T. Irene Sanders and Judith A. McCabe. “The Use of Complexity Science." Washington Center for Complexity and Public Policy. October 2003. 36, hipfvww.complexsys.ore/pddl ComplexityScienceSurvey:pdf Also see a description of the program contained in the mirror of an olé CIA webpage at: htp:/ cxyptome.orp/cia-d htm » Folker, 9 % Richards I. Heuer, Je, “Adapting Academic Methods and Models to Government Needs,” in Quantitative Approaches to Political Intelligence The CIA Experience, ed. Richards J. Heuer, Jr. Boulder: Westview Press, Inc, 1978), 6. ® Folker, 8 ® Fotker, 14 Arthur S. Hulnick. Fixing the Spy Machine: Preparing ‘American Intelligence forthe Twenty-First Century. Praeger; Wesiport, CT. 1999. 53. These and the Following quotes ate from Stephen Marrin. “Homeland Security and the Analysis of Forcign Intelligence.” Markle Foundation Task Force oa National Security in the Information Age Background Paper. October 2002, Also reprinted under the same tide in: The Intelligencer: Tournal of US. Intelligence Studies. V13. N2. (2003). 25-36. ° Alan Sehwartz. Email, December 25, 2006, » Folker, 15. Stanley A. Feder. “Forecasting for Policy Making in the Post- ‘Cold War Pesiod.” Annual Review of Political Science. Vol. 5 (2002), 111-125, ‘npilpoliseiannualreviews.org/egivconten/Full//1/111 Stanley A. Feder. “FACTIONS and Policon: New Ways to ‘Analyze Politics." In Inside CIA's Private Weeld: Declasified Articles from the Agency's Internal Joural, 1955-1992, ed. H. Bradford Westerfield, New Haven: Yale University Press. 275, Robert D. Folker, Js. Exploiting Structured Methodologies © Improve Qualitative Intelligence Analysis. Masters Thesis. Joint Military Intelligence College. July 1999. 2. Feder. “Forecasting for Policy Making inthe Post-Cold War Period” > Folker, 14 ° Richards J. Heuer. Adapting Academic Methods and Models to Goveramental Needs: The CIA Experience. Strategic Studies Institute, US Army War College. 31 July 1978. 5. Heuer has since said that the analysts were resistant to quantitative techniques but more receptive to qualitative approaches, ° Sherman Kent. Strategic Ineligence for American World Policy. Princeton University Press; Princeton, NJ. 1951. (reprint of 1949 version). vi. » Walter Laqueur, The Question of Judgment: Intelligence and Medicine,” The Journal of Contemporary History, Vol. 18. 1983, (533-548). $35; as quoted in Stephen Marrin and Jonathan Clemente, “Improving Intelligence Analysis by Looking to the Medical Profession.” International Journal of Intelligence and Counterintelligence. Vol. 18, No. 4. (2005) (707-729). 708. In particular, see: Rob Johnston, “Foundations for Meta- Analysis: Developing a Taxonomy of Intelligence Analysis Variables.” CIA Studies in intelligence V. 47. No. 3. 2003). ® Richards J. Heuer. Email, January 5, 2008. Summer 2007 © Jeffrey Pfeffer and Robert I, Sutton, “Evidence-Based Manogement,” Harvard Business Review, January 2006, As cited in the invitation for the 2007 ODNI Conference on Improving. Intelligence Analysis, 4" Jecome Groopman, quoted in “The Doctor's In, But Is He Listening?" National Public Radio Morning Edition March 16, 2007 ‘hupsiwww-npr orpitemplates/story!story-phpstoryld-8946558 © Jerome Groopman. Excerpt from: “How Doctors Think.” National Public Radio Morning Edition March 15, 2007, hup:/ww npr orgitemplates/story/story php?storyld8946558 © Michael Moss, “Mammogram Team Leams From It Errors.” [New York Times, Sune 28, 2002. * Rob Hubal; Email, December 26, 2006, ‘Washington Plat, Strategic Intelligence Production: Basic Principles. Frederick A. Praeger: New York. 1957. 15 * Rob Hubal; Email. December 26, 2006, Stephen Matrin and Jonathan Clemente. “Modeling an Intelligence Analysis Profession ou Medicine.” Intemational Journal of Intelligence and Counterintelligence. Vol. 19. No. 4 (Winter 2006-2007). 642.665. “Rob Hubal; Email. December 26, 2006, “© Randy Pherson Email. December 24, 2006 ® Bale Iohnston. Integrating Methodotogists into Teams of Substantive Experts. Studies in imeltigence. Vol, 47. No. 1 © Timothy J. Smith, “Predictive Waming: Teams, Networks, and Scientific Method." Roger Z. George and James B. Bruce, Getting it Right: Expanding Frontiers for Incelligence Analysis. Georgetown University Press. 2007 (forthcoming) > Mary Louise Kelly, “Intelligence Community Unites for “Analysis 101.” NPR Moming Edition. ? May 2007, -upslwvwnprorgtemplates story/story. php?storyld=10040625 ® Randy Pherson Email. December 24, 2006, % Randy Pherson Email. December 24, 2006 David T. Moore. Critical Thinking and Intelligence Analysis. ‘Occasional Paper Kumikes Fourteen (Washington, DC: National Defense Intelligence College, 2006). It includes the sylisbus fo [SA's critical thinking and structured analysis class in the appendix, pages 97-115, Available at htp:/wwrw2 pare convist/projects‘ach/ach html © Richards J. Hever, Jr, “The Future of “Altemative Analysis” Presentation notes forthe Director of National Intelligence conference on Improving Intelligence Analysis: What Works? How Can We Tell? Lessons from Outside the Intelligence Community, Chantilly, VA, January 9 - 10, 2007. 4 ® Kristan Wheaton and Diane Chido, “Strtured Analysis of Competing Hypotheses. Improving a Tested Intelligence Methodology.” Competitive Intelligence Mugazine, November? December 2006. hupsriwww.momanis-monsalve.com/assets/publications! intoligence-methodology-1-07-chido pdf Also see: “Structured Analysis of Competing Hypotheses: Theory and Application.” Analytic Methodologies Project: Mercyhurst College Institue of Intelligence Studies Press. 2006, Rob Johnston, “Foundations for Meta-Analysis: Developing a ‘Taxonomy of Intelligence Analysis Variables.” CIA Studies in Intelligence V. 47. No. 3. 2003) " Conference materials. Improving Intelligence Analysis: What ‘Works? How Can We Tell? Lessons from Outside the latelligence Community." 9-10 January 2007. Chantilly, VA. Swemer 2007 © Steven Rieber and Neil Thomason. Creation of a National Institute for Analytic Methods. Studies in Intelligence, Vol. 49. No.4. (2005). 71-77 © Examples ofthis kind of research iat analytic methods include: Leonard Adelman, Paul E, Lehner, Brant A. Cheikes, and Mark F, Taylor, “An Empirical Evaluation of Structured Argumentation Using the Toulmin Argument Formalism. “IEEE Transactions on Systems, Man, and Cybernetics —Pan A: Systems and Humans, Vol. 37, No. 3. (May 2007). 340-347, Also see: Paul E. Lehner, Leonard Adelman, Beant A. CCheikes, and Mark Brossn, “Confirmation Bias in Complex Analyses.” IEEE Transactions on Systems, Man, and Cybernetics. Forthcoming. The former paper reports the results of an experiment assessing the utility ofa strciured argumentation technique for intelligence analysis. and the latter paper reports the results of an experiment assessing the ability of ACH 10 mitigate confirmation bias. © Folker, 15, Folker, 8 * Bole, 6. “For another perspective on impsoving the at and science of intelligence analysis, see: Stephen Martin. “Adding Value to the Intelligence Product.” Handbook of Intelligence Studies. (Ed. Loch Johnson} Routledge, 2006. 199-210, Interested in publishing an article in the American Intelligence Journal? Submit a manuscript for consideration to Anthony Mc Ivor

    Das könnte Ihnen auch gefallen