You are on page 1of 21

Can We Generalize from Student Experiments to the Real World in Political Science, Military

Affairs, and International Relations?
Author(s): Alex Mintz, Steven B. Redd, Arnold Vedlitz
Reviewed work(s):
Source: The Journal of Conflict Resolution, Vol. 50, No. 5 (Oct., 2006), pp. 757-776
Published by: Sage Publications, Inc.
Stable URL: http://www.jstor.org/stable/27638520 .
Accessed: 15/11/2011 11:48
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.

Sage Publications, Inc. is collaborating with JSTOR to digitize, preserve and extend access to The Journal of
Conflict Resolution.

http://www.jstor.org

from Student
Can We Generalize
to the Real World in
Experiments
Political Science, Military Affairs,
and International Relations?
ALEX MINTZ
of Political

Department

Science

Texas A&M University
and Interdisciplinary

Center-Herzliya,

Israel

STEVEN B. REDD
of Political

Department
University

Science

of Wisconsin-Milwaukee

ARNOLD VEDLITZ
Institute for Science,
George

Bush School

Technology

and Public

of Government

Policy

and Public

Service

Texas A&M University

The authors conducted an experiment with a group of military officers and replicated itwith a group
in the United States. The experimental
of students at a public university
scenario dealt with a decision
The authors found that while more than one-third of students rec
problem in the area of counterterrorism.
ommended
doing nothing, the overwhelming
majority of military officers (more than 90 percent) recom
and more satisfacing decision
mended doing something. Also, military officers exhibited
less maximizing
than students.The
results
making
real-world national security policy
ent. Based on student samples, it
of elite decision makers and reject
be assigned to experiments where

Keywords:

national

show that relying on experiments with students "playing" the role of
makers may bias the results. The two groups are, in fact, very differ
is possible to accept propositions
that would not be found with samples
it is possible that students can
that may be right. However,
propositions

security;

they represent

decision

and not elites.

the "public"

making;

counterterrorism;

student experiments;

uncertainty

AUTHORS'
NOTE: All data used in this article can be found at http://jcr.sagepub.com/cgi/content/
We thank Letitia Alston, Roger Channing, Mark Davis, Donald Green, and Xinsheng
full/50/5/757/DCl/.
to this project. This research was supported by the National Defense
Liu for their valuable contributions
University under contract no. DABJ29-03-P-0084.
Opinions, findings, and conclusions or recommendations
expressed

do not necessarily

EDITOR'S

reflect

the views

of the National

Defense

NOTE: This article, by the Associate
Editor
Journal of Conflict Resolution
(JCR) and coauthors, experienced
two rounds of readers' reports and the JCR Editor.

University.

of
for Experiments
and Simulation
the customary process of harassment

the
by

JOURNAL OF CONFLICT RESOLUTION, Vol. 50 No. 5, October 2006 757-776
DOI: 10.1177/0022002706291052
? 2006 Sage Publications
757

Do students make 2.. Are Do the same the same information relations 4. Do students use 3. and (5) the effect of certainty on choice and process. where often blurred. (3) the decision strategy they employed.Green and Gerber (2002) and Tetlock (1983) provide a useful review of strengths and weaknesses associated with experimental research (see also Kinder and Palfrey 1993. commanders process-tracing their The article attempts to answer the following 1. and (Mintz et al.2 The scenario stated that the military was faced with a decision about which counterterrorism nario is highly rorism. 2002b).0 simulator choices elites' represent accurately of an experiment It uses students.1 Questions about the validity of results using students in experiments are often raised (see. variables exogenous choices with conducted Board 4. (2) the amount of information students and officers used. of elite decision making in the war on ter the boundaries between military and civilian decision making are Subjects were presented with four real-world alternatives and were asked to select among the following: Border-Crossing Environmental Local Emergency Sensors Monitors Responders Do Nothing 1. The sce representative. The experimental task and the decision in detail inAppendix A. Stoll 2004 and Sears's [1986] famous study on the "sophomore effect"). as such and with experiment 2004a) decision processes.g. see McDermott 2002a.758 JOURNALOF CONFLICTRESOLUTION 1 numerous studies in political science. e. 2. technologies to pursue in fighting the war on terror. Mintz make. of course. military affairs. 1997. McDermott 2002a. (4) the effect of framing on choice and process. but no empirical test has been conducted to assess article a whether compares replication using the paring effect the decision a students of and officers computerized. Virtually all of these studies use students as the subject pool. on certainty in com choices and processes. the Decision using and students results makers? decision as actual strategies and of students patterns decision acquisition decision makers? officers similar in international scenarios? such as framing and uncertainty in the same way? leaders factors and military we Accordingly. military respective framing This strategies. and international relations use experiments (for reviews. scenario are described . THE EXPERIMENT In this a national experiment. 2002b). security scenario dealing with a combating terrorism decision was used to introduce alternatives and dimensions into the Decision Board software. as elite choices questions: students' compared and affect officers' the choices choices and strategies and decision of students strategies along the following dimensions: (1) actual choices.

Ostrom and Job 1986). Morgan and Bickers 1992. Riggle Astorino-Courtois Mintz and Geva (1997). Lau 1995. 556) state. Hence. Steenbergen et al. As Mintz et al. 2002) have used process-tracing techniques in studies of foreign policy decision making.. evaluated (see Mintz et al. Mintz et al. e.Mintz et al. we report results of an experiment conducted with students and military officers using the Decision Board 4. (2000). Taber and 1994. (1997. and Redd (2000. / FROM STUDENT EXPERIMENTS TO THE REAL WORLD 759 The dimensions employed in the scenario represent decision criteria that were found to be relevant in other studies in international relations (see James and Oneal 1991. The computerized board records key features of The ." Larson (1985) and Khong (1992) used process-tracing techniques in studies of the origins of the containment doctrine and on decision making during the Vietnam War. the dimensions included in the Decision Board were as follows: Military Economic Political The Decision then compared tracing Board software recorded the "moves" of subjects on the board. respectively (see also George and Bennett 2005 for a more general discussion of process-tracing methods have methods thus far in been case most study often research). tracing identifies what information is being accessed to form a judgment and the order in which the information is accessed (Ford et al. This knowledge can then be used tomake inferences about which deci sion strategies were employed en route to a choice. used in Experimental studies of voter process-tracing choice (see. 1997). and Riggle and Johnson 1996. We students' and officers' decision and strategies choices using process techniques. Mintz 1995). (1997). 1989). PROCESS TRACING tracing is a methodology designed to help identify and classify decision Process making processes. Johnson THE DECISION BOARD PLATFORM core structure of the Decision is depicted in Board platform. In this article.0 software. Process "[Process tracing's] main strength is its ability to identify specific strategies used by decision makers and to test theoretically derived implications of situational and per sonal variables on the decision and process its outcome.g. 1997. Mintz and Geva 1997. which a on are is matrix of the alternatives and dimensions which alternatives Figure 1.

and for understanding and management. in teaching courses in international relations. has been used and economic trends and events. of California-Davis. School of Management of Canterbury. more typically. The com puterized Decision Board records (1) the sequence inwhich decision makers acquire the information. These features are subsequently used to identify the decision strategies of policy makers.760 JOURNALOF CONFUCT RESOLUTION Decision Economic Political Board Dimension Dimension * Do Environmental Crossing Emergency ?? Select f? 0) oeiect f) Select f? Final Choice: Select f? Select Select f? Select Select f) Select O Nothing Select Add Select Add Select Add c Figure 1: Decision Board Platform the decision-making process. for modeling poliheuristic consumer It behavior and choice. These information bins can be opened to reveal their contents by the click of a mouse. (2) the number of items that respondents view for every alternative along every dimension. A decision problem typically consists of the selection of an alternative from a set of available alternatives. theory). Israel. New Zealand.0 of the Decision Board also automatically displays the "decision portraits" of decision makers and calcu versus satisficing lates holistic versus nonholistic search patterns and maximizing one can the decision rules. It has been 3. Using process-tracing strategy selec techniques.0 has been used for research. prospect theory. multiple dimensions. China. The "values" in the matrix represent the evalu ation of a given alternative on a given dimension. teaching. cybernetic theory. for process tracing of political voting games and electoral campaigns. University in Foreign Policy Decision and the George Bush School of and the Program Making University. and training. The Decision in research to test theories of decision making (expected utility. China Foreign Affairs University Force the Air Yale at Tel Aviv University. Board Simulator 4.S. identify tion and decision model of decision makers. Academy. U. The Board has been used at twelve universities. public policy and public administration. Version 4. of Wisconsin-Milwaukee. whereas decisions are made by clicking on the choice box of a desired alternative (Mintz et al. and (3) the amount of time that elapses from the time respon dents begin the task until they make their choice. and Public Service at Texas A&M University. 1997). The choice set is evaluated along a single dimension or. including the University University in Beijing.3 A major category of these features relates to the sequence in which the decision maker accesses the information. Government used .

dimension-based a decision maker across dimensions Dimension-based processing. the student undergraduate were subjects recruited from several courses. The two factors were as follows: (1) certainty effects (certain vs. armed and National Guard.0 Board all represented forces and several branches of the Reserve as a "decision platform process to one of four experimental domly assigned branches four of the U.S. The Decision Board recorded the amount of information used by the officers and students. METHOD SUBJECTS Fifty military officers and forty-six undergraduate students at the University of Wisconsin-Milwaukee (UWM) participated in the experiment. officers Decision 4. THE INDEPENDENTAND DEPENDENT VARIABLES The independent variables in this study consisted of (1) the framing of the likeli hood of funding from Congress for the new antiterrorism technologies and (2) the degree of certainty that the proposed antiterrorism technologies would work as designed. uncertain that the antiterrorism technology would function) and (2) framing of the likelihood of funding the antiterrorism tech nologies (positive vs." The subjects were ran conditions. thirteen colonels. We also Alternative-based reviews proceeds items in of this used measures strategies information manner of within for alternative versus imply a process whereby a subsequent given alternative alternatives. the values were inserted in the decision matrix. These values consisted of a descriptive statement and a summarizing numeric value (on a scale from -10 to +10) (see Appendix B). DESIGN A 2 x 2 between-groups factorial design was employed. sequentially and then strategies simply signify that decision makers process information within a dimension across alternatives and (Payne 1976). thirty-one lieutenant colonels. These strategies were calculated using Billings Scherer's (1988) search index (SI). and five These captains. The index ranges from -1 (purely dimensional to +1 (purely alternative-based processing) processing). The study used the tracer. As in many experi mental studies in political science. / FROM Mintz STUDENT TO THE REAL WORLD EXPERIMENTS 761 Following the definition of the four alternatives and three dimensions. military affairs. and international relations. The scoring of subjects' . The military officers who participated in this study included a brigadier general. negative). The dependent variables in this study consisted of (1) information acqui sition patterns and (2) the choices subjects made. science political The military commanders were recruited from a leadership course taught at the National Defense University (NDU).et al.

allows This the military to coordinate a more for provides For evaluation. 2002). Congress committed rorism. and (4) negative framing?uncertain technology. . RESEARCH MATERIAL In addition to comparing the choices that students and military officers made and strategies." to and Kahneman . the manner an in which 1981). Subjects could also open any information bin by on it. capacity. of Framing Manipulation refers Framing 1993. Subjects the present because time. maximizing As in previ as an the information was presented 2002). garded alternative-based in Billings As while moves. It also produces a distribution of subject choices. framing?uncertain positive technology.g. a by local versus satisficing local emergency "Using example. we have assessed the effect of framing and certainty on deci their decision sion and processes choice. two Consequently. 'compre antiterrorism shield for the United States. there is approxi a 90% chance that Congress will fund at least one of these options. given multimedia interface. Each clicking subject dealt with only one of the four conditions: (1) positive hensive' framing?certain technology. Tversky were mately has "At told. which contained infor Board matrix. Frisch framing condition terrorism." Subjects indicated their choice by clicking on the choice button underneath the corresponding alternative. and shifts (s) and uses the equation SI = (a.762 JOURNALOF CONFLICTRESOLUTION moves was determined by the SI scores (see also Mintz et al. and the ability to record the cognitive "moves" of decision makers. (3) negative framing?certain (2) technology. a decision portrait of each subject. because . were factors and framing manipulated: certainty (see Appendix A). there is approximately from public told. 1997. Iwould rate this alternative as 7. (Mintz et al. Dimensions) to the pertaining Decision Board evaluation has several a of alternative given advantages: a along a user-friendly dimension.. Redd 2000. e. THE RESEARCH INSTRUMENT The Decision mation The in this experiment consisted of a 4 x 3 (Alternatives x representing twelve information bins. The index tallies the number of dimensional moves (d). and scores. negative shifts were disre and Scherer. holis tic versus nonholistic ous studies statement evaluative antiterrorism followed activities with scores. of issue is presented in the positive the war on (see. 1997. Mintz 2004a. Redd numerical corresponding responders communities. alterna tive moves (a). numbers imply dimensional from this index. "At ter the pre a 10% chance . sent time. In contrast.d)l(a +d)to define the index. search Positive numbers indicate moves. to do whatever those it takes in the negative of the recent war to protect framing the American condition were in Iraq.

Many in the scientific community are encouraged by the progress made so far and are hopeful that these options and the technology associated with them will actually work. 41) . and political. Following three different dimensions: military. a the evaluations of those alternatives along economic. Subjects were informed that they would be presented with a specific scenario con cerning various technologies being developed to combat terrorism and with a deci sion matrix containing alternatives and national security decision. followed by a detailed debriefing. / FROM Mintz STUDENT EXPERIMENTS TO THE REAL WORLD 763 that Congress will not fund any of these options.009.00) and negative frame significant. F(l." of Certainty Manipulation Subjects in the certain condition were told. all of the subjects were subjected to time pres sure manipulation. Congress may be constrained in its spending because of a weak domestic economy and the ongoing war in Iraq and the cost of the war approaching $100 billion." In contrast. in the uncertain those condition were told. The subjects were the among the available options. However. For the student subjects. the framing manipulation was = = 7." To increase the "mundane reality" of the experiment (Aronson and Carlsmith 1968). instructed to make the best choice postdecision questionnaire was administered. which consisted of instructions indicating that there was a time constraint. the subjects in both subject pools were not actually restricted in the amount of time available. "At this stage there is a high level of certainty that these future technologies will be successful and will work as conceived of and designed.et al.43. positive frame (M 8. RESULTS MANIPULATION CHECKS It is important to note that the manipulations for framing and certainty worked in both subject populations. "At this stage there is a high level of uncertainty about whether these future technologies will be successful and will work as conceived of and designed. p < . The instructions and decision were scenarios on displayed the computer screen. Many in the scien tific community are skeptical about the progress made so far and doubt that these options and the technology associated with them will actually work. PROCEDURE The experiment was administered with military officers at the National Defense University and replicated with students at the University ofWisconsin-Milwaukee. and since many national security and foreign policy decisions are made under time and informational constraints.

with the students again = 11.003.04) and military subjects (M= 8.19. significant.03. compared with only 8 percent of the military officers the Do Nothing choosing option and 92 percent choosing one of the other (Do Something) options (see Table 1). As will be processes. positive frame (M = 8. = = 9. we found a significant difference between the two subject pools (z = 3.39. we found that military officers were likely to select the Border-Crossing Sensors alternative while student subjects were more likely to choose the Do Nothing option (see Table 2). and officers.82. p < .57). Using a chi-square test.p < .75).0002. Significant dif ferences were also obtained for the certainty condition. This is probably the result of students being less familiar with the policy alternatives and their implications. p < .50). significant. students also opened (M 9. First we compared student subjects' decision to choose the Do Nothing response versus choosing one of the other Do Something options with the officers' decisions to do the same. p < . The certainty manipulation for the student subjects was also = = = 10. 45) The certainty manipulation officers was also for the military = = = 7.24). Using a difference of proportions test for the two different subject pools (z-test). students (M = 11. F(l. We also found significant differences in the amount of information accessed as a function of condition (see Tables 4 and 5). they were also obtained for the uncertainty condition.63. the difference was significant. We then analyzed differences between the two groups in the amount of informa tion they accessed. students (M = 11.86) and uncertain (M 3.88.57) and military officers (M = 9.003. STUDENTS-MILITARY COMMANDERS COMPARISON We compared students' choices.96).001). while 65 percent chose one of the other three alternatives.11.12). and decision strate gies with the military officers' choices. Under negative framing.01). Specifically.008. F(l.33) compared to the military commanders accessing more information (M = = 7. p < . 47) = 9. = (M 6.13. 94) = 14. Furthermore. p < .009. the framing manipulation = 7. Students viewed more all conditions. We found that 35 per cent of the student subjects chose the Do Nothing option. 41) was likewise For the military officers.27) than military subjects (M 8. 47) = 6. we compared the choices of the two groups for the four specific options. Specifically. explained information there were significant of patterns acquisition in differences students and the choices. we found significant differences between the two = groups (%2 12.12) and negative frame F(l. and decision strategies and generated the following results. certain (M 6. p < .84). search information patterns. p < . significant.42. The data reveal that the two groups differed in the overall number students accessed signifi of cells viewed. F(l. F(l.30) than their military counterparts (M cantly more cells (M Table 3). 45) below. F(l. F(l.20. 45) = 5. Under the condition of positive framing. F(l. 45) = = more information bins (M 11. p < .56) and uncertain (M 4.55. information than military officers across .764 JOURNALOF CONFLICTRESOLUTION = {M 6. information search patterns.27).04) (see 11. p < . certain (M 5.009.01.

We found a significant result in officers' and students' maximizing = < decision of the . whereas 82 percent of the officers chose a satisficing strategy and only 18 strategy (see Table 6).39.4 Specifically. p ).et al.04 WHO ISMORE RATIONAL?MILITARY OFFICERS OR STUDENTS? We also compared officers' and students' differences in choices of decision strate and gies. The strong findings showing sat percent chose amaximizing rules decision isficing employed by most subjects correspond to Herbert Simon's of notion bounded (2004b) poliheuristic (1955) theory.01 40 2.30 9. / FROM Mintz STUDENT TO THE REAL WORLD EXPERIMENTS 765 TABLE 1 Differences between Students and Military Commanders in Choosing "Do Nothing" versus "Do Something" (in Percentages) Choice Subject Do Something Do Nothing Group 65 Students 35 8 Military 92 TABLE 2 Differences between Students and Military Commanders in Choosing among All Options Choice Border- Local Environmental Crossing Emergency Responders Do Nothing n (Percentage) n (Percentage) Sensors Monitors Subject Group n (Percentage) n (Percentage) Students 15(33) 30(60) Military 3(7) 3(6) 12(26) 13(26) 16(35) 4(8) TABLE 3 Number of Cells Accessed Cells Accessed (Mean) Students Military 11. percent satisficing strategies (z a student subjects chose a maximizing while chose 60 strategy percent satisficing strategy. rationality and Mintz's .

this alternative to its negative score on the military dimension. there was a marked 82 difference the between groups.12 8. 5. the choice is "good enough" (Simon 1955. Weights refers to the importance levels decision makers attach to different dimensions.33 Military 9. with and more satisficing decision making exhibiting less maximizing was the highest rated overall. Specifically. the Do alternative Although Nothing was a nonstarter for the majority of the military officers due the officers than students. one 4. threshold. merely that is. We then compared the weights that officers and students assigned to the dimen sions on the Decision Board. Satisficing implies that a decision that satisfies some a priori minimum maker's chosen alternative need not be an optimal one.766 JOURNALOF CONFLICTRESOLUTION TABLE 4 Number (Mean) as a Function of Framing of Cells Accessed Framing Group Positive Subject Students Negative 11.27 11. .84 TABLE 5 Number (Mean) as a Function of Certainty of Cells Accessed Certainty Group Certain Subject Students Uncertain 11.5 The data show that military officers' weights for different dimensions were significantly lower than those of students.96 TABLE 6 Differences Maximizing between Students and Military Commanders in Choosing versus Satisficing Decision Strategies (in Percentages) Decision Subject Students GroupMaximizing Strategy Satisficing 40 60 18 Military Interesting.57 11.24 8.04 Military 9. 1957).

at 5. whereas the military officers rated the politi cal and military dimensions in the 6-point range but rated the economic dimension lower. F(l. We a also found specific difference between students and officers evaluating the impor tance of the economic dimension. 93) = 10.45) (see Table 8).45 TABLE 9 Differences between Rating Students and Military Commanders the Do Nothing Alternative Alternative Subject Group Students Do on (Mean) Nothing 4.99) (see Table 7). p < .99 TABLE 8 Differences between Students and Military Commanders on Evaluating the Importance of the Economic Dimension Weighting of Economic Dimension Students (Mean) Military 5. p < .82 1. 283) 5.002. wherein student subjects weighted the dimensions = higher (M 6.45. This finding indicates that in recommending Do Something.02.62 5. military officers were less concerned with the financial costs associ ated with the different alternatives.70. .Mintz et al.61 Military there was a difference between the two subject pools in weighting the dimensions = overall.4 and 6.8. / FROM STUDENT EXPERIMENTS TO THE REAL WORLD 767 TABLE 7 Overall Weighting Weight Assigned of Dimensions to Dimensions Students (Mean) Military 6. F(l. It is interesting to note that the students rated all three dimensions between 6.46.80) and officers (M = 5. students (M = 6.62) compared to military commanders (M 5.

dimension-based the 10 for a when than -. F(l.63). student fashion. F(l.768 JOURNALOF CONFLICTRESOLUTION TABLE 10 versus Alternative and Choice Dimension-Based in the Context Information Processing of the Certainty Manipulation Certainty Group Certain Subject Students Military Positive NOTE: dimension-based numbers Uncertain 0. processing.05.97.418). the officers military more used presented with uncertainty ated in a more antiterrorism mation dimensions. Using ANOVA methods and the Billings and Scherer (1988) SI. = < for the .415) = when presented with uncertainty (M -. p < .415 -0.165) of uncertainty. 91) = 31.60. .82) and NDU (M = 1. p < .010). We THE EFFECT OF FRAMING AND UNCERTAINTY ON PROCESS AND CHOICE We then examined the effect of and framing on uncertainty information process ing and choice. 91) p (M 5. they tended to dimension-based to resort that strategies strategies. Specifically. 42) certainty manipulation. 46) = 6.61)?a huge difference (see Table 9). when presented with certainty. whereas negative denote numbers also found a significant difference between the two groups in their overall the alternatives?specifically. much better than the Do Nothing alternative. they oper summary of this respect to the operation of the decided to gather more infor information about each alternative technologies across = were the political. We found a significant result for the military subjects.55) (M 6. tended . certain economic. of the Border-Crossing Sensors and Do Nothing rating alternatives. and military in the .010 -0. = = = < students and officers 3.0001. 3. When to evaluate commanders by processing told them (M procedures manner about these technologies an alternative-based military (M = alternative-based It seems that when finding). the results were in the opposite direction to the student compared subjects. Military officers liked the Border-Crossing Sensors alternative.165 0. for the certainty manipulation. have been shown to help alleviate cognitive strain (Russo and Dosher 1983).07. however.418 denote alternative-based processing. when presented with certainty. officers fashion. In the uncertainty condition. The scale ranges from -1 to 1. F(l. to succeed.56. Under conditions Table (see facing uncertainty with the military technologies. UWM (M = 4. we found a close to conventionally significant result for the student population.01. Specifically. This finding suggests that subjects operated in a more dimension-based when the student subjects were faced with a more cognitively demanding uncertain condition. students more used alternative-based (M procedures = than .44. p F(l.

Numerous experimental studies of political decision making involve voter participation experiments (Palfrey and Rosenthal 1985). The external validity of such experiments is questionable. 1997). However. to perform in political therefore the public: marketing and advertising experiments (Astorino-Courtois 1996. For example. from 6. while the average age of the student subjects was twenty-two. the military samples and experience are often environment. and the compared military profession tends to be more male dominated. problem is one or we of may incorrect accept that propositions and inference. We would like to set the boundary conditions of our findings. compared Wisconsin-Milwaukee. experimental research on framing takes the that students are an educated segment of society. not all experiments successfully science and international relations focus on elite decision making. gender differences effects on conflict and aggression (McDermott and Cowden 2001). therefore. deal less about the sophisticated for example. in evaluating theo retical and empirical claims based on experiments with student samples. as empirically inappropriate. sample. 84 percent were male and 16 percent female. officers' represent accurately Because of these and expertise student limitations. For the military officers. tell less a representative educated. officers ought to be more accountable or responsible for failed decisions to students. using student samples. Such a the sample of mil subject pool is younger and more mixed than that of elite decision makers. With actually on other.6 The experiment reported in this article demonstrated the difficulty of generalizing from experiments with students to the behavior of real-world national security deci sion makers. public atti tudes toward affirmative action (Kuklinski et al. political choice. 1992). Whereas the general tendency of officers' and students' results was largely similar. with an appropriate a great design. caution could be wrong. to 57 percent male and 43 percent female for the subjects from the University of . students may not Moreover. then. is warranted the Ultimately. Military officers are also typically older than students. Experimental studies in political science are typically conducted with student subjects. Sears.et al. When the real-world "equivalent" of a student sample is the "public" rather than the leader or the elite. we may accept propositions that would not be found in samples using elite decision makers. it should (Mintz and Geva may experiments also work 1998). the evaluation of public figures (Lau. itary officers in our study consisted of subjects with an average age of forty-three. statistically significant differences were found on most indicators of information processing and It is unrealistic to expect that undergraduate students will be able the role of military elites. they can lead to divergence in subject population results. and Centers 1979). Therefore. Specifically. Iyengar 2000). / FROM Mintz STUDENT TO THE REAL WORLD EXPERIMENTS 769 WHEN CANWE GENERALIZE FROM STUDENT EXPERIMENTS IN POLITICAL SCIENCE AND INTERNATIONALRELATIONS? Experiments using students to test propositions about military elite decision mak ing ignore the fact that students are typically novices while officers are more often experts in national and security counterterrorism of aspects decision making. if framing position "works" groups on student us research behavior students. Specifically. and an experimental analysis of a political stock market (Forsythe et al. of the public.

There were also marked differences between the military officers and students in their choices. we compared the choices student subjects made to those of mili tary officers. while student subjects were much more likely to do so (35 percent). as those dealing with personal Finally. in international choice tended to acquire significantly also Officers relations. and financial and so on?may social decisions. than military decision officers en route and strategy student subjects to their choices. We also the compared information search and processes of both groups. while the The military commanders were rather homogeneous student subjects were more evenly split between the Do Nothing option and other alternatives. "Experiments tions . and (4) the effect of exogenous on conditions Specifically. to students-as-subjects amenable consumer research.770 JOURNALOF CONFLICTRESOLUTION geographically and demographically diverse schools. experiments. while officers used tary affected decision makers' selection of decision strategies. We used a decision scenario thatmimics real-life problems of officials in area the of counterterrorism. (2) the amount of information used by the two groups. We have done so in an experimental setting using theWeb version of the Decision Board Platform 4. it affected the uncertainty two subject pools in opposite ways: student subjects responded to uncertainty with more dimension-based commanders while the military employed processing alternative-based procedures when faced with greater uncertainty. Also. whereas of results where experiments students in the area of national security should be taken play the role of elite decision makers cautiously. more assigned information lower weights to dimensions than students. (3) the decision strategy employed by each group. Future work should compare students' and actual decision makers' choices and strategies in experiments where decision dilemmas . one may be able to infer about the behavior of the public on a variety of issues. Our results reveal significant statistical differences between students and actual decision makers in (1) the choice. Military commanders were very reluctant to choose the Do Nothing option (8 percent). As a general rule. in order testing the same model to determine the external should be conducted validity of any given on multiple experimental popula par adigm" (McDermott 2002b.. and the groups varied in their decision strategies: mili more satisficing strategies than students (Simon 1957). while we do not have evidence to support the external validity of experiments with students repre senting the public. we believe that experiments can be conducted when students this represent of segment society. be dilemma market issues.0 with two different subject students populations: and military commanders. CONCLUSION In this article. in their choices.. 40). other types of research questions?such economic behavior. We then analyzed how uncertainty and decision-making the framing of options affect decision makers' responses and choices with respect to policy options dealing with counterterrorism.

in the future. because will at least one fund to do whatever alternatives Border-Crossing to detect order have Sensors: whether or biological these monitors sample options. the proposed tech decision your of verbally terrorism. level of professional responsibility.. The what tech advanced (WMD)." . we compared were specific differences significant found results statistically on most indicators. A implemented choice must and political The military dimension deals with how implications. stations. multiple 771 science and international relations decision security the making?as groups are very different in their sociodemographic characteristics. and other significant factors. there is a 10% chance that Congress will not fund any of these options. 7. expertise. Research concern public decision making cross-validate STUDENT area the of national should also to It is unrealistic issues. Negative frame reads as follows: "At the present time. cers. Congress to protect the American public identified: been it takes sensors Introduce terrorists into the country. to detect nologies a chief been weapons to develop looking of mass destruction all of the technologies administration official. can crossings or biological in trace whether set off. The administration.et al.7 at border chemical that and choice a 90 percent has committed from soil. The political dimension deals with how will be received and the American by Congress public. because of the recent war in Iraq. centers and As made. economic. / FROM Mintz our results using scenarios multiple on students to play the role of elites in political expect not least experiments?at in TO THE REAL WORLD EXPERIMENTS (e. on the effect of framing). of the war on terrorism.g. designed not However. While some general tendencies were when similar. there is approximately time. 2001. students between and offi APPENDIX A The Scenario COMBATING TERRORISM on the administration's the past several weeks the media focused have During heavily to pursue advanced decision its choice whether various upcoming regarding technologies at combating in the future. weapons Environmental Monitors: chemical of these Introduce are that can be used attempting to smuggle environmental monitors be has military. Using have been EPA monitoring weapons the air for traces of chemical and biological toxins. Congress may be con approximately strained in its spending because of a weak domestic economy and the ongoing war in Iraq and the cost of the war approaching $100 billion. has laboratories. can be you must several new futuristic decide to do. on American uses of WMD aid the armed services in dealing with potential in terms of research dimension addresses the total costs of each technology would nologies The economic as well development the new technology At chance and the present that Congress in writing The following as implementation. of September aimed terrorism the Since the events 11. and terrorist acts has been undertaking administration and combat terrorists steps to detect here in the United in cooperation committed with various research States.

Remember that you can once." .772 JOURNALOF CONFLICTRESOLUTION Local Provide local emergency Emergency Responders: detection equipment. Many this stage there and will work cessful encouraged associated so far and are hopeful the progress made them will actually work. not to proceed Do Nothing: Decide with implementing time. Many whether will be successful and will work as conceived are skeptical about the progress made so far and doubt that these options and in the scientific community the technology associated with them will actually work.8 Board will indicate how each of these by with The Decision evant These dimensions. written responders any with particular radiological system at this will be suc technologies are in the scientific community that these options and the technology future rel is evaluated along various options as a rating on a 21-point summarized a neutral 0 implies position. unfavorably. At is a high that these level of certainty as conceived of and designed. 8. Uncertainty manipulation reads as follows: "At this stage there is a high level of uncertainty about these future technologies of and designed. that information. are also evaluations that an option is evaluated (-10 implies very 10 implies a very favorable evaluation of the option). the time time it takes you to review "box" to start of is a trade-off to make a decision the evaluations information the decision may to explore simulation the evalua then determine your choice. A decision has to be made! Please the computer begin the various and tions of the alternatives dimensions along scale and As you with consider Taking only access Press all "real-life" and too much a particular CONTINUE there decisions. of information the amount between based be on costly. process.

S.APPENDIX B DECISION MATRIX FOR COMBATING TERRORISM Border Military Environmental Crossing Sensors Dimension sensors at "Deploying border crossings is a positive step in the prevention of terrorism in the United States." afford. rather them.-A (--Ij rate this alternative Iwould as -8 rate this alternative I would as rate this a . billions and billions "This option is mor effective since loc nities and actors c bear the cost of fig war on terrorism." than simply detecting I would rate this alternative I would States. The military 'compr shield antiterrorism tends to favor United 'preventing' terrorist activities." of dollars." rate this a as as as Emerge Responders 1 Economic "This option would most likely be cost prohibitive. simply 'monitoring' the release of chemical/ "Using local emer responders allows tary to correlate a ism activities with communities. The U. however. environmental "Deploying monitors throughout the United States in most major with cities would has long borders and both Mexico Canada and trying to cover of miles with the thousands each is more than we can be entirely too running into the expensive. too late." rate this alternative "Deploying environmental monitors could help in the detection of terrorist activity." I would as . This also provides the military with a concrete loca tion inwhich to concentrate its antiterrorism I would Local Monitors activities. This for a more toxins may be too biological little.

" "This will show that the U." "This option will n looked upon favora local governments have to contrib resources to the wa will t terror." I would as rate this alternative I would as rate this alternative I would as their rate this a . put undue burdens public within communities.S.N B (continued) APPENDIX Crossing Sensors "Protecting very visual our borders Monitors is a sign of the war on terror and will make U. is seriously committed war on terrorism. Environmental to the monitors show the public that we will it takes to do whatever protect Emergen Responders our people. citizens Local Environmental Border feel safer.S. Likewise.

research: From shopping malls designs for political communication at the Experimental Methods 11-12. Age-related Johnson. In and Ira Katznelson. Experimental methodology in political science. Gerber. Schechtman. Ann Arbor: University of Michigan Press. Richard R. -. Brian M. James. of social psychology. Kuklinski. B. and J. Origins of containment: Philip E. for framing effects.. L. 1988.. Patrick. Reasons Frisch. Tetlock. and Ellen D. University 1995. and Jonathan Cowden. edited by Helen New York: W. Scherer. Organizational Robert Billings. Journal of Politics 58:1035-54. 2005. political science. Journal of Conflict Resolution Mintz. Experimental to the Internet. 2002.Merrill Carlsmith.. into national realities: Allison. Schmitt. Racial prejudice and attitudes American Journal of Political Science 41:402-19. and neglected research questions. and Thomas R. 3rd ed. Organizational 43:75-117. edited by Milton Lodge and Kathleen judgment. Experimental methods in political science. Experimentation Aronson. Forrest Nelson. Palfrey. of stakes and threat on foreign policy decision-making. Atlanta. 1979. Reclaiming Political science: The state of the discipline. 1991. Elliot. in a crisis simulation 2001. McDermott. Princeton.. Case studies and theory development the experimental Green. Doherty. Paul M. and Human Decision Processes Forsythe. Sniderman. The "positivity bias" in evaluations of public Lau. 1996. Neal Ford. Mitzi M. Kathleen Knight. R. 325-42. Princeton. Lawrence. Press. May MA. 1989. David O. W. Political Psychology 21:489-510. A psychological explanation. The effects peace in Jordan. Gordon toward affirmative action. Oneal. Lau. Hults. 2004a. tradition in political V. Riggle. Public Opinion Quarterly 43:347-58. Sears. Alexander sciences. The effects of response mode and importance on decision Processes Behavior and Human Decision strategies: Judgment versus choice. 1985. Political Analysis 10: 2002b. George R. Deborah NJ: Princeton Welch. edited by Gardner Lindzey and Elliot Aronson. Robert. Process Susan L. J. 2002a. at war: Korea. Cambridge. Kinder and Thomas R. making 41:1-19. Thomas Piazza. Foreign policy decision making in familiar and unfamiliar settings: An experimental 48:91-104. Rose. Shanto. On behalf of an experimental Kinder. The effects of sex and uncertainty Rose.. Richard R. S. James H. Paper presented in the social Press. Dien Bien Phu. G A. Deborah. and the Vietnam decisions Yuen Foong. information for in processing differences 1994. . Annual Review of Political Science 5:31-61. Milner science. McGraw. American Economic Review 82:1142-61. MA: Addison-Wesley. of 1965.. Journal of Conflict Resolution 35:307-32. Reading. agreements -. S. Marketing Arab-Israeli 2000. Ann Arbor: University of Michigan Press. Conference. game. 1992. methodology M. and Alan S.. and John R. figures: Evidence against the instrument artifacts. International McDermott. Interactions 27:353-80. Cambridge. and Richard Centers. 1997. Transforming international Astorino-Courtois. and Barbara Meilers. Palfrey. Larson. Anatomy of an experi and Human Behavior Decision 54:399-429. Neumann. 1993. mental political stock market.et ai / FROM Mintz STUDENT EXPERIMENTS TO THE REAL WORLD 775 REFERENCES In The handbook and social psychology. NJ: Princeton University science. Kevin. Processes and Jack Wright. tracing methods: Contributions. and Mary L. Behavior problems. 1968. Information search during an election campaign: Introducing a processing-tracing for political In Political scientists.. Alex. Norton. study of high-ranking military officers. The influence of domestic and international politics on the pres ident's use of force. Donald P. and Lisa L. In 1993. April. Analogies Press. MA: MIT Bennett. 2000. Experimental of political foundations attitude-laden Khong. decisions. Donald R. Paper presented at the Cognitive Aging Conference. and Andrew George. Organizational 1992. Munich. Iyengar. edited by Donald R.

choice sets on political decision making: An analysis using the Decision Board platform.. New York: John Wiley. on foreign influence of advisers 2002. Science 211:453-8. sion making. Bickers. Charles W. Alex. and Howard Rosenthal. Mor. Palfrey. T. Glenn Palmer. Computational Taber. Alex Mintz. 1957. Psychology: Learning. Richard J. 1955. Political Science Review 79:62-78. experiments edited by Milton Political judgment. Age difference 18:99-118. in the study of conflict management and substance Integrating methodology in inter InMultiple paths to knowledge and conflict resolution. of personality Hills. Texas A&M University. In Conflict the study of crisis. in electoral behavior. -. 1992. Political Behavior 1983. war and peace. Quarterly Journal of Economics Experimental Sears.. David O. and the external use of force. Journal of Conflict In Geva. Conclusion: Multiple paths to knowledge? and conflict resolution. and Richard J. -. and Daniel Tversky. Journal of Conflict Resolution 36:25-52. edited by Frank P. American Political Science Review 91:553-66. Clifton. Steven B.. Simon. Stoll. and Barbara Anne Dosher. in the laboratory: Influences of a narrow data base on social 1986. 1983. and Amy Carnes. Lodge and Kathleen M. The framing of decisions and the psychology of choice. in political Ellen D. 1997. The effect of dynamic and static Mintz. Johnson. How 2004b. PhD diss. in decision making: An information and contingent processing Payne. Resolution Mintz. B. do 48:3-13. Amos. A behavioral model of rational choice. edited by Nehemia Geva and Alex CO: Lynne Rienner. Journal of and Cognition 9:676-96. New York: St. 2000. 1981. Clifton Morgan. and Marco R. and Nehemia Decisionmaking Mintz. J. 2004. Stoll. 1995. Herbert A. Voter participation American and strategic uncertainty. McGraw. Models of man. Task complexity 16:66-87. Strategies for evaluating political candidates. T.. Lanham. Martin's. 1985. Alex. The president and the political use of force. Kahneman. MD: Lexington Books. 1986. leaders make decisions? A poliheuristic perspective. Domestic Morgan. Organizational Redd. 1997. Jr. 1976. In Review research on foreign policy: A methodological Philip E. . Edward.. Tetlock.776 JOURNALOF CONFLICTRESOLUTION -. edited by Ladd Wheeler and Phillip Shaver. College sophomores and Social Psychology 51:515-30. 4.. overview. S. Ostrom. decision making: choice. and Mitzi M. Beverly and social psychology. The Resolution policy decision making. A prospect-based analysis of war termination. Redd. and Brian L. binary Riggle. The influence of advisers on decision strategies and choice in foreign policy deci -. Memory. Job. Ann Arbor: University Michigan In of Press. Behavior and Human Performance search and protocol analysis. Journal of Personality psychology's 69:99-118. John W. on war and peace: The cognitive rational debate. Strategies for multiattribute Russo. Harvey and Ben D. Thomas R. Psychological vol. Boulder. Steven B. discontent and Kenneth N. view of human nature. American Political Science Review 80:541-66. 1996. Steenbergen. Charles S. CA: Sage. Journal of Conflict 46:335-64. edited in the study of conflict management national relations: Methodology by Zeev Maoz. The poliheuristic theory of foreign policy decision-making. in in world politics: Advances 1998. Nehemia Geva.