GAME-BASED LEARNING IN INTELLIGENCE: TEACHING INTELLIGENCE ANALYSTS GEOGRAPHY USING GAMES

DANIEL Q. BURCH

A Thesis Submitted to the Faculty of Mercyhurst University In Partial Fulfillment of the Requirements for The Degree of MASTER OF SCIENCE IN APPLIED INTELLIGENCE

INSTITUTE FOR INTELLIGENCE STUDIES MERCYHURST UNIVERSITY ERIE, PENNSYLVANIA MAY 2012

INSTITUTE FOR INTELLIGENCE STUDIES MERCYHURST UNIVERSITY ERIE, PENNSYLVANIA

GAME-BASED LEARNING IN INTELLIGENCE: TEACHING INTELLIGENCE ANALYSTS GEOGRAPHY USING GAMES

A Thesis Submitted to the Faculty of Mercyhurst University In Partial Fulfillment of the Requirements for The Degree of MASTER OF SCIENCE IN APPLIED INTELLIGENCE

Submitted By: DANIEL Q. BURCH

Certificate of Approval:

___________________________________ Kristan J. Wheaton Assistant Professor Department of Intelligence Studies

___________________________________ Shelly L. Freyn Assistant Professor Department of Intelligence Studies

___________________________________ Phillip J. Belfiore Vice President Office of Academic Affairs May 2012

Copyright 2012 by Daniel Q. Burch All rights reserved.

iii

DEDICATION
This work is dedicated to my family and friends. I am forever thankful for the love of my parents, Henry and Lily Burch, the support of my brothers, Alex Burch, Richie Neale, Stan Topa, and Josh Gamse, and for the beautiful smile of my goddaughter, Lila Neale. I am also thankful for the guidance Professor Kristan Wheaton has provided and for the progress others have made in the field of game-based learning. Most of all, I am thankful to God, without Whom none of this would be possible.

iv

ABSTRACT OF THE THESIS

Game-Based Learning in Intelligence: Teaching Intelligence Analysts Geography Using Games A Critical Examination By Daniel Q. Burch Master of Science in Applied Intelligence Mercyhurst University, 2012 Professor Kristan J. Wheaton, Chair

Initially, the purpose of this study was to use a controlled experiment to test the hypothesis that entry-level intelligence analysts who play Jericho, a web-based computer game, will perform higher on assessments measuring their ability to recall a countrys geographic information than analysts who learn through traditional methods. However, due to flaws in the experimental design, the study actually produced a comparison between implicit and explicit learning. Students in the experimental group learned Jerichos geography implicitly through one exposure while students in the control group learned Jerichos geography through continuous exposure over a period of twenty minutes. Since the control group was able to review the information continuously, it is no surprise that they outperformed the experimental group. It is important to note however, that with just one exposure, students in the experimental group answered questions correctly 70.91% of the time. Further research should seek to determine whether

increasing the number of exposures to Jericho significantly increases the scores for the experimental group.

vi

TABLE OF CONTENTS
DEDICATION ................................................................................................................... iv ABSTRACT OF THE THESIS .......................................................................................... v LIST OF FIGURES ........................................................................................................... ix LIST OF ABBREVIATIONS ............................................................................................. x INTRODUCTION .............................................................................................................. 1 LITERATURE REVIEW ................................................................................................... 4 Theoretical Framework ................................................................................................... 4 Traditional Game-Based Learning .................................................................................. 7 Digital Game-Based Learning ...................................................................................... 10 Edutainment vs. Serious Gaming .................................................................................. 13 Game-Based Learning in the Military .......................................................................... 15 Intelligence Analysis ..................................................................................................... 17 Geography and Games .................................................................................................. 18 METHODOLOGY ........................................................................................................... 22 Research Design............................................................................................................ 22 Variables ....................................................................................................................... 22 Selection of Participants ............................................................................................... 23 Recruitment of Participants........................................................................................... 24 Process .......................................................................................................................... 25 Instrumentation ............................................................................................................. 25 Data Analysis Procedures ............................................................................................. 27 RESULTS ......................................................................................................................... 28 Initial Assessment ......................................................................................................... 29 Text-based Portion (Questions 1 10) ..................................................................... 29 Map-based Portion (Question 11a. 11f.) ................................................................ 34 Post Assessment ............................................................................................................ 37 Text-based Portion (Questions 1 10) ..................................................................... 38 Map-based Portion (Question 11a. 11f.) ................................................................ 39 Interest Level ............................................................................................................ 40 CONCLUSION ................................................................................................................. 41 Recommendations for Future Research ........................................................................ 43 REFERENCES ................................................................................................................. 46 Annex ................................................................................................................................ 53

vii

Annex A: Institutional Review Board Proposal Form .................................................. 53 Annex B: Control Group Consent Form ....................................................................... 58 Annex C: Experimental Group Consent Form.............................................................. 59 Annex D: Initial Assessment Form ............................................................................... 60 Annex E: Debriefing Statement Form .......................................................................... 62 Annex F: Post Assessment Form .................................................................................. 63 Annex G: Statistical Data.............................................................................................. 65 Initial Assessment ..................................................................................................... 65 Post Assessment ...................................................................................................... 176 Annex H: Jericho Screenshots .................................................................................... 218

viii

LIST OF FIGURES
Page Average Score (In Percent) for Group and Figure 1.1 Educational Level Average Percent Of Correct Answers For TextFigure 2.1 Based Section Average Percent Of Correct Answers For MapFigure 3.1 Based Section Average Percent of Correct Answers for TextFigure 4.1 Based Section Average Percent of Correct Answers for MapFigure 5.1 Based Section 35 34 30 26 25

ix

LIST OF ABBREVIATIONS
IARPA IIS-MU IPB Intelligence Advanced Research Projects Activity Institute for Intelligence Studies at Mercyhurst University Intelligence Preparation of the Battlefield

INTRODUCTION
Although game-based learning has existed in various forms throughout history, it was not until the 1960s that researchers became truly interested in its application within the classroom. By the 1970s however, interest in game-based learning dropped significantly, as researchers and educators alike discovered that game-based learning was not the panacea they had believed it would be. Game-based learning then endured a period of ignominy, before the advent of computer technology and digital gaming breathed new life into game-based learning and saw to the creation of the game-based learning industry. As discussed in the literature review, many researchers and educators have successfully implemented game-based learning as a teaching tool in the classroom. The widespread use of computers has helped considerably in this regard. Todays young men and women have grown up in a digital age and are highly accustomed to the various technologies educational game designers employ. Despite the widespread (but not universal) acclaim for game-based learning, no research has statistically measured the efficacy of using game-based learning to teach the concepts and skills that are necessary for conducting intelligence analysis. It is for this reason that I selected an experimental methodology for this study. The purpose of this study is to measure the degree to which game-based learning can be used to assist entry-level intelligence analysts in familiarizing themselves with the geography of an unfamiliar region. This study utilizes experimental data and statistical analysis to determine whether or not Jericho, a webbased computer game that takes place in the fictional country of Jericho, can be an effective tool for teaching entry-level intelligence analysts to quickly recall key

information concerning the geography of a fictional country. The experimental group played this game, and its results were compared to those of the control group. The control group did not play the game, and was instead exposed to Jerichos geography through the traditional method of reviewing an informational page containing a map and key information. In conducting this study, the researcher hoped to determine whether or not Jericho can assist entry-level analysts in memorizing the geography of Jericho. The ability to quickly understand the geography of a region is useful for providing military assessments, such as the United States Armys Intelligence Preparation of the Battlefield, and the United States Navys Threat Model. It is also important for intelligence analysts in general, who are often tasked with producing assessments on countries or regions with which they might be unfamiliar. The faster these analysts are able to understand how a countrys geography shapes its civilian and military planning, the faster they will be able to produce high quality intelligence products. This study also seeks to lay the groundwork for future studies. If the results of this study indicate that playing Jericho is at least as effective as the traditional method of reading a map and a list of associated information, then educators in the field of intelligence will have statistical data that supports the use of game-based learning in the classroom. This would, in turn, pave the way for future studies analyzing other skills, such as the ability to shift through large volumes of information, the ability to perform under pressure, or the ability to effectively communicate ones findings. The relatively small number of participants (55 individuals) and limited resources (in terms of funding, available time, and computer programming ability) serve to hinder

this study. As an individual who has grown up in the digital age, it is also possible that the researcher suffers from biases in favor of game-based learning. Regardless, this study will hopefully help provide the foundation and inspiration for future researchers who seek to use game-based learning to teach the skills and practices necessary for conducting intelligence analysis.

LITERATURE REVIEW
Game-based learning has undergone a relatively recent revival, particularly with the advent of digital gaming. Modern game-based learning relies heavily on the principles of implicit learning, experiential learning, and case-based reasoning, which together form the core of this learning modalitys theoretical foundation. Although gamebased learning was first introduced in the early 1960s, it is still prevalent in education today. The United States military in particular has effectively incorporated game-based learning into its training efforts for new personnel. Few researchers however, have examined the potential for this modality in teaching skills and concepts to entry-level intelligence analysts. Theoretical Framework Although few proponents of game-based learning have explicitly stated so, gamebased learning relies heavily on principles found in implicit learning, experiential learning, and case-based reasoning. Researchers have been unable to agree on a proper definition for implicit learning. Reber, one of the first researchers in implicit learning, defines the term as the acquisition of knowledge that takes place largely independently of conscious attempts to learn and largely in the absence of explicit knowledge about what was acquired (Reber, 1993). Some, such as Cleeremans, believe that implicit learning is the process of acquiring implicit knowledge (1996). For such researchers, implicit knowledge is knowledge that can influence processing with possessing in and of itself the properties that would enable it to be an object of representation (Cleeremans, 1996). Other authors however, believe that individuals have learned implicitly if their

performance [in accomplishing a given task] indicated greater knowledge than they thought they possessed or could enunciate (Dorken & Whittlesea, 1993). Each of these definitions possesses varying levels of similarity. For the purposes of game-based learning however, it is enough to refer to implicit learning as a process that occurs without the intention to learn, which results in knowledge that is not completely accessible to consciousness (Poletick & van den Bos, 2009). Researchers have extensively debated the nature of implicit learning. Some researchers have wholeheartedly denied the existence of implicit learning. For these individuals, learning is an inherently conscious process with no room for unconscious learning (Perruchet & Vinter, 2002). Others believe that learning takes place on a spectrum where explicit and implicit learning form the spectrums ends (Dunn & Kirsner, 1989, Reber, 1993; Cleeremans & French, 2002). This document operates on the assumption that implicit learning will occur in game-based learning, regardless of whether the distinction between implicit and explicit learning is a sharp divide or a flexible spectrum. An individual playing a strategy game may learn that if he does not send scouting units to determine the location and composition of his opponents forces, he will be unprepared for incoming enemy attacks. In games with a wider strategic view, a player may find that upon completion of the game, he or she may be able to recall the locations of important cities. In most cases, players do not sit down and attempt to memorize the rules of a game. To do so would likely be boring and tiresome. Individuals play a given game because they find enjoyment in playing the game. In the process however, they learn the rules of the game and

eventually are able to act within the limits of those rules to accomplish whatever objectives the game has created. Game-based learning also incorporates the concept of experiential learning. Whitehead observed that people learn from experience and process that experience within the confines of their own mind (1929). Supporters of experiential learning often find themselves at odds with those who support traditional education, where fragmentary topics are taught, quickly memorized, tested, and then forgotten (Lesgold, 2001). This research operates from the premise that both experiential and traditional learning methods have significant roles to play in education. One does not learn how to drive a car for example, by studying reports on the mechanics of vehicular movement. An individual learning to drive must physically drive a vehicle, ideally under the supervision of a more experienced driver, before he or she is capable of obtaining a drivers license. Similarly, one must understand the physical properties of chemistry before performing experiments with potentially dangerous chemicals. Games, by their very nature, present scenarios that can represent either real world or fantastical situations (Salen & Zimmerman, 2004). Through these simulations, players are able to experience and learn from dynamic systems that construct representation through play (Salen & Zimmerman, 2004). Finally, the application of game-based learning fully incorporates the concept of case-based reasoning. According to case-based reasoning theory, in life an individual will encounter and solve countless problems. When faced with a problem, the individual will assess the situation, recall a solution for a similar situation, and then apply that solution to the circumstances at hand (Lesgold, 2001). If educators use a game to teach their students how to solve a given problem, they hope that the students, when encountering a similar

problem in their lives, will be able to apply their experiences from the game in solving the problem they face. Traditional Game-Based Learning Many authors have examined game-based learning and its applications for education, businesses, the military, and government organizations. To date, however, scholars have been unable to reach a consensus on what constitutes game-based learning, or even on a definition for a game. In an effort to create a universal definition, Salen and Zimmerman examined eight scholars definition of the word game and found that the most common factor was that a game proceeds according to rules that limit players (2004). Five of those eight scholars also stated that games are goal or outcome-oriented (Salen & Zimmerman, 2004). For the purposes of this research, the term game-based learning will refer to the use of games to teach concepts or skills. The term game will refer to a structured activity where factors such as roles, goals, activities, constraints, and consequences drive a players actions. Some researchers feel it is important to distinguish further the difference between games and simulations (Cruickshank & Telfer, 2001). Scholars have provided varying and sometimes conflicting definitions of the terms (Heitzman, 2001), but these terms are not mutually exclusive (Cruickshank & Telfer, 2001). For this research, there is no need to draw a distinction between the two. For centuries, societies have used game-based learning to teach concepts and theories to individuals. One of game-based learnings oldest forms, war-gaming, spread its roots in the days of the Roman gladiators, the Gupta Empire in India, and medieval European knights (Carlson, 1969). The governments and militaries of major and minor

world powers utilized war-gaming in World Wars One and Two, and it has been argued that the Japanese attacked Pearl Harbor in 1941 based on faulty assumptions made during their war games (Carlson, 1969). Large businesses such as Proctor & Gamble, Boeing, and IBM have employed games within their companies (Carlson, 1969). Harvard, the University of Chicago, and the University of Pennsylvania have all devoted resources to developing games within their graduate business programs (Carlson, 1969). The United States and its allies regularly use joint training exercises to reinforce the training of their military forces. Educators have successfully utilized game-based learning across a broad range of fields to help teach a wide variety of topics. However, research in games-based learning did not start until 1962 (Boocock & Schild, 1968). In its infancy, games-based learning research underwent three stages (Boocock & Schild, 1968). In the first stage, which occurred from 1962 to 1963, researchers accepted game-based learning on faith and employed it in the classroom with wild enthusiasm (Boocock & Schild, 1968). From 1963 to 1965 however, researchers began to realize that games were not a cure for all the problems that existed in the education system and that there was a significant lack of empirical evidence to support the use of games in the classroom (Boocock & Schild, 1968). After 1965, researchers approached games-based learning with a more cautious optimism, in hopes of determining what specific benefits games-based learning could provide (Boocock & Schild, 1968). Initially, much of the research in game-based learning relied on anecdotal evidence. Researchers and educators alike postulated that games could teach investment strategies, collective bargaining techniques, and methods for boosting employee morale

(Carlson, 1969). Other research suggested that games create an environment where students want to learn and are able to obtain immediate feedback for their actions (Carlson, 1969). Boocock and Coleman found that games, when structured appropriately, could motivate players, teach specific information, and impart a broader social perspective on young learners (1966). There were also claims that games provide structure while allowing students to connect subject matter to emotional learning and to understand group dynamics (Adams, 1973). There were also beliefs that games could force students to approach learning material in new ways (Taylor, 1972). Finally, researchers claimed that games could create unique opportunities in classrooms with students of various intellectual levels, where higher-level students are able to perform without negatively affecting their less gifted peers; while the less gifted are able to learn from their peers and play the game in a way that is most interesting to them (Taylor, 1972). Most modern research in game-based learning relies heavily on Csikszentmihalyis so-called flow state, where a participant is deeply focused on the task at hand and, as a result of his mastery of the task, receives immense enjoyment from it (1991). For a game to create a flow state in its players, it requires constant, intense and enjoyable focus on the tasks it asks players to complete (Koster, 2005). Contemporary research also highlights the ability of games to provide experiences similar to those that learners will encounter in their own lives (Cruickshank & Telfer, 2001). Through these experiences, gamers are able to use their own abilities to solve difficult problems, attain immediate feedback, and shorten the learning period, all in a safe and enjoyable environment (Cruickshank & Telfer, 2001). Using these concepts, game designers have

created simulation games for a variety of customers, including the New York City Health Department, the New York Police Department, the United States Forestry Service, Mexico City University, and even a NATO country (Dunnigan, 2011). As many researchers have noted however, employing games in the classroom will not automatically address the various problems that exist in the educational system. Despite its benefits, games-based learning does present unique challenges when applied to a classroom setting. Games are time-consuming, costly and require teachers to reevaluate the manner in which they run their classrooms (Taylor, 1972; Cruickshank & Telfer, 2001). They also have varying rules and structures, and there is little agreement on the level of assistance that teachers should provide (Fletcher, 1971). Additionally, educational game designers have trouble balancing the entertaining and educational aspects that a game can provide (Cruickshank & Telfer, 2001; Cheng, Hong, Hwang, Lee, Lin, & Lu, 2009). Other criticisms focus on the lack of empirical evidence to support researchers claims regarding games-based learning. Such critics cite the lack of standardized independent and dependent variables in games-based learning studies and the inability of researchers to utilize standardized methods of measurement (Fletcher, 1971). Regardless of these criticisms, games-based learning has many applications in a variety of learning environments and has continued to grow with the advent of digital gaming. Digital Game-Based Learning Digital game-based learning refers to the synthesis of serious learning and video games to create a fun and engaging learning environment in which students can learn new concepts and strategies (Prensky, 2001). As early as 1982, researchers studied the appeal

10

of video games and how they could be applied to learning environments (Bowman, 1982). Bowman claimed that the underlying design of the game Pac-Man, which provided clear goals and immediate feedback, created a state of intense concentration in players (1982). He believed that this principal could be adapted to the classroom by providing learners with clear tasks, roles and responsibilities, as well as providing them with a degree of choice and the ability to balance their skills with progressive challenges. Other researchers claimed that the absence of severe negative consequences for risks taken in video games (Provenzo, 1991), or the level of challenge, fantasy, and curiosity associated with video games were concepts that could be applied in an academic environment (Malone, 1981a, 1981b). The advent of the digital age revitalized the publics interest in game-based learning. Some researchers and educators, sensing the benefits that video games could potentially provide, began to create educational games to teach concepts in the classroom. However, these efforts have not always met with success. More often than not, there is little funding available for educational games and the individuals who design them have little access to the technology that modern games require (De Castell & Jenson, 2003). These developers also tend to follow an extremely linear pathway where players have few options to solve monotonous problems that are in no way related to the narrative of the game itself (De Castell & Jenson, 2003). Challenges, exploration, exposure to stimuli, and character identification are strong factors that create immersive gaming environments that draw in players both young and old (Salen & Zimmerman, 2004). Through these factors, games have the ability to create patterns that players want to learn, practice, and use, because games make this process fun (Koster, 2005). For these reasons, some

11

researchers have suggested that educational game designers must learn from commercial games or disappear so that educators may incorporate commercial games in the classroom (Dickee, 2005; Gee, 2005; Ranalli, 2008). More recently, the push for digital game-based learning has come from authors such as Marc Prensky and James Gee. In his book, The Digital Game-Based Learning Revolution, Prensky claims that the current generation of American workers learns in ways that are fundamentally different from the ways in which previous generations have learned (2001). He also believes that growing up in a digital world has significantly transformed the learning preferences and abilities of this new generation (Prensky, 2001). These changes, combined with the growing promise of the $10.5 billion video game industry (based on 2009 statistics) have created new opportunities to revolutionize the learning process (Entertainment Software Rating Board, 2009). Similarly, Gee claims that although todays students are capable of passing tests concerning the material they have learned, they are unable to apply the knowledge they have gained to solve real-world problems (2005). Gee takes game-based learning in a different direction, by stating that the best video games already incorporate effective learning principles that are supported by current research in cognitive science (2005). These learning principles allow learners to immerse themselves in the material they are learning, gain immediate feedback on their progress and work with information that is connected in a manner where one piece of information can have a strong impact on the system as a whole (Gee, 2005). This is especially important in todays digital world, where individuals must sift through a massive amount of information in order to get a clear grasp on the situations in which they find themselves.

12

Digital game-based learning has gained increasing popularity, due in large part to the significant technological advances associated with modern video games. Modern video games create environments where players utilize a variety of tools and resources to overcome challenges, co-operate with fellow gamers, increase their engagement with the material in the game and develop their problem-solving strategies (Gros, 2007; Squire, 2005). These games can also improve players ability to comprehend spatial patterns, read and understand diagrams and multi-task to accomplish objectives (Gros, 2007). In video games including multiple players, participants must obtain and analyze information from a variety of sources, discuss that information with their fellow players and collaborate to solve in-game problems that are meaningful to them (McCreery & Schrader, 2008). These are all valuable skills that young individuals can apply not only to academic settings, but to almost every work environment after graduation. Most importantly, video games transform the tedious traditional methods of learning dry and technical concepts into a fun learning process that makes learners want to increase their knowledge (Prensky, 2001). These factors can help to ensure that each student more fully comprehends the material he or she is learning. Edutainment vs. Serious Gaming Edutainment refers to the combination of [educational and entertainment] on a variety of media platforms (Egenfeldt-Nielsen, n.d.). With the advent of computer gaming however, edutainment began to intersect with game-based learning. Today, a significant portion of edutainment focuses on the use of educational computer games to teach children as they play. Edutainment games are created to provide simple and straightforward gameplay that demonstrates a clear in-game reward structure (Egenfeldt-

13

Nielsen, n.d.). Much like edutainment, serious gaming is the melding of education and entertainment (in the form of games). Serious gaming is specifically defined by its explicit and carefully thought-out educational purpose and [is] not intended to be played primarily for amusement (Abt, 1987). Initially, it may appear that edutainment and serious gaming are one in the same. When edutainment is limited to computer gaming, both schools of thought focus on allowing individuals to learn while they play. However, while edutainment and serious gaming may initially appear very similar, they have two significant differences. The most readily visible difference between edutainment and serious gaming is the academic focus. Edutainment primarily focuses on simple concepts such as basic math, spelling and geography (Charsky, 2010). However, due to budgetary constraints and outdated learning theories, edutainment often provide little intrinsic motivation, lack an integrated learning experience, focus on memorization rather than comprehension, are overly simple and do not involve teachers or parents (Egenfeldt-Nielsen, n.d.). These shortfalls have led to an overall negative attitude towards edutainment titles (EgenfeldtNielsen, n.d.). In contrast, serious games focus on education, industrial and governmental training, planning, research, analysis, and evaluation (Abt, 1987). Although less visible than academic focus, the style of learning forms used in edutainment and serious gaming is far more important. As mentioned above, edutainment games teach students to memorize the material in front of them rather than comprehend it. Students playing edutainment games will be able to memorize two plus two equals four, but they will be unable to explain why two plus two equals four (Charsky, 2010). In serious gaming players must not only understand the rules, objectives, and consequences

14

of a game, but they must understand why those factors exist as well (Abt, 1987). The lack of higher-order thinking resulted in a significant amount of harsh criticism for edutainment and was a significant factor in the promotion of serious gaming. It is important to note however, that edutainment still has applications in education, as it is particularly effective for teaching lower order thinking skills, facts, concepts, and procedures [that are] essential to [virtually all] fields of study (Charsky, 2010). Game-Based Learning in the Military The United States military has been a particularly strong supporter of serious game-based learning. One of its early initiatives, SIMNET (SIMulater NETworking), replaced both live exercises and costly high-end stand-alone simulators in the early 1980s (Nieborg & van der Graaf, 2003). It also focused on collective training rather than individual training. In the mid-1990s, the federal government began to explore whether commercials games could be incorporated into government training. Marine Doom, a 1996 modification of the popular DooM II video game, provided United States marines with the opportunity to train and develop military skills and decision making with a four-member fire team (Nieborg & van der Graaf, 2003). The United States Army also utilizes Battlefield 1942, a first-person shooter while the Air Force uses Falcon 4.0, a flight simulator (Nieborg & van der Graaf, 2003). Other games, such as Full Spectrum Warrior, can help players familiarize themselves with military technology (Nieborg & van der Graaf, 2003). As of 2005, the United States military had invested at least five billion dollars in hardware, software, and research and development in the realm of game-based learning (Branom, 2005). In fiscal year 2008, the United States Armys Program Executive Office for Simulation, Training, and Instrumentation spent over 3

15

billion dollars on products and services for training, mostly by contracting with commercial companies (Atkinson-Bonasio, 2008). Two games in particular, Americas Army and Asymmetric Warfare Environment have illustrated how gaming can be utilized for military training. Americas Army is a publicly available first-person shooter originally designed for recruitment purposes. In playing the game however, individuals can quickly strengthen their grasp of military concepts. Before a player can begin playing the game, he must undergo a short basic rifle marksmanship tutorial. Upon completion of this tutorial, the player is able to assume the role of an infantryman. If he wishes to play as another class however, such as a medic or Special Forces, he will have to undergo further specialized training. Through these training exercises, the player is able to gain an appreciation of chain of command, familiarization with the use of a high-power rifle, the ability to estimate distances and basic medical techniques (Nieborg, 2004). Unlike Americas Army, Asymmetric Warfare Environment is a massively multiplayer environment that is available only to select members of the United States military. Asymmetric Warfare Environment allows players to prepare themselves for circumstances that they may encounter while conducting operations in an urban environment. In the game, players represent themselves with a highly customizable virtual avatar that they can modify in real time during the simulation (Kusumoto, Mayo, & Singer, 2006). Asymmetric Warfare Environment also has a large library of avatar templates with skin, body types, and clothing for Iraqi civilian men and women, Iraqi police, and U.S. Army soldiers (Kusumoto, Mayo, & Singer, 2006). These templates allow the Army to run many types of simulations. One such simulation requires players

16

to man checkpoints where they follow rules of engagement, often making split-second decisions on which rules apply based on their situational awareness (Kusumoto, Mayo, & Singer, 2006). This often involves observing and interacting with the local population through the assistance of translators and civil affairs officers (Kusumoto, Mayo, & Singer, 2006). Intelligence Analysis Much like game-based learning, it has been difficult for individuals to agree on a proper definition for intelligence. According to Mark Lowenthal, intelligence is information that meets the stated or understood needs of policy makers and has been collected, processed, and narrowed to meet those needs (2009). Robert Clark defines intelligence as a target-centered process designed to reduce uncertainty for decision makers in a conflict (2010). Mercyhurst University however, has developed a more comprehensive definition of intelligence: an externally focused process designed to reduce the level of uncertainty for a decision maker using information derived from all sources (Chido, 2006). To date, few scholars have attempted to use game-based learning to teach intelligence concepts or practices. The work of the Intelligence Advanced Research Projects Activity (IARPA), a research agency that answers to the Office of the Director of National Intelligence, and Professor Kristan Wheaton are the only examples of such attempts. In early 2011, IARPA posted a solicitation for its Sirius Program, which seeks research projects centered on the use of games to mitigate the cognitive biases that commonly affect all types of intelligence analysis. Specifically, the solicitation requested that the project examine confirmation bias, fundamental attribution error, blind

17

spot bias, anchoring bias, representativeness bias and projection bias (IARPA, 2011). At the time of writing this document, IARPA had scheduled the Sirius Program to begin in 2012 and had estimated that the project would take four years to complete. Wheaton, as a professor at Mercyhurst University, has used game-based learning to augment traditional teaching methods in an intelligence studies classroom. In his research, Wheaton tasked students with playing specific games before each class. He then instructed the students to come to a defensible conclusion about how the game related to the topic they were studying, which they would discuss in class with their fellow students (Wheaton, 2011). In later classes, Wheaton combined this form of teaching with supplemental reading material that hinted at connections between the games and the course topics. The results of this study indicated a significant increase in the students ability to see patterns and connections buried deeply in unstructured data sets, produce high-quality intelligence products and retain an understanding of the concepts learned (Wheaton, 2011). While much of the evidence in this study was anecdotal, it helps to provide a strong framework for future research in the application of game-based learning to intelligence studies. Geography and Games The importance of geography in intelligence analysis cannot be understated. Concepts such as contour lines and the map tables of World War Two were quantum leaps in the realm of cartography and have since been replaced by computer and video technology (Doty, 2005). These resources, which the National Geospatial Intelligence Agency utilizes today to provide geospatial intelligence, help intelligence consumers in the Department of Defense and the United States government to understand the

18

conditions on the ground (Doty, 2005). Geospatial intelligence describes, assesses and visually depicts physical features and geographically referenced activities on the earth (Miller, 2009). Intelligence analysts operate within a very limited timeframe. The more time that is spent learning new geographic regions, the less time analysts have to apply methodological tools to analyze raw information (Marrin, 2003). Many analysts in the intelligence community focus on one region of the world for only one to two years before their employers assign them to another region (Hitz, 2001). While Hitz argues that analysts should be given assignments based on their familiarity with specific regions of the world, the needs of the intelligence community do not typically allow for this (2001). This constant need to reassign analysts only highlights how important it is for analysts to have the ability to quickly familiarize themselves with regions of the world that are outside their current area of expertise. It is also important to understand the geography of smaller regions. These features of these areas, sometimes referred to as the microphysical geography, include both terrain features and weather patterns of an area (White, 1996). This concept is best captured in the Intelligence Preparation of the Battlefield (IPB), which the United States Army developed to inform its planning efforts. The IPB examines the battlefield terrain, the effects the terrain may have on friendly and enemy units and uses this information to evaluate opposing forces and their likely course of action (U.S. Army Field Manual, 1994). On a tactical level, the IPB can outline complex route intersections, interchanges, bridges and buildings, which in turn improves situational awareness for commanders and allows staffs to develop courses of action and plans based on accurate data (Miller,

19

2009). The United States Navy employs a similar concept in their Threat Model, which incorporates terrain, hydrology, and weather (Intelligence Analysis, 1990). Improving the ability of entry-level intelligence analysts to memorize geographic locations would significantly assist in creating geospatial intelligence products, especially given the need to continually update the information prior to and during any military operation (U.S. Army Field Manual, 1994). This information is still valuable after a decision maker has committed resources, as it allows him or her to plan further strikes, either on the same target or on additional targets (Doty, 2005). Additionally, the geography of a region is likely to have a significant impact on how modern armies conduct knowledge-based warfare in the future. Knowledge-based warfare is defines as warfare in which combat power is best concentrated through information transmission (Eden, 1997). Modern military tactics focus on attacking the enemy where he cannot effectively fight, but this is likely to change as technology improves both weaponry and communications systems (Eden, 1997). These improvements will create new opportunities for ground forces to fight more effectively by communicating digitally with sensors, artillery, air support, electronic warfare systems and other ground units (Eden, 1997). As military forces employ these new technologies, any geographic features that adversely affect communication will severely reduce the ability to conduct effective warfare. The ability to quickly recall information concerning key geographic features will be an asset to any military operation that an analyst is supporting. Multiple researchers have already used games to teach geography. Admiraal, Akkerman, Huzienga, and ten Dam found that students who played a location-based

20

game on their mobile phones were able to learn more about medieval Amsterdam than students receiving traditional project-based education (2009). These results may have been impacted however, by the fact that students playing the game spent an entire day learning the material while students learning through traditional methods spent roughly two hours (Admiraal, Akkerman, Huizenga, & ten Dam, 2009). Brom, Sisler, and Slavik found that students who played Europe 2045, a game that introduces players to the geography of several different European states, reported having learnt new information about Europes geography (2008). It is important to note however, that this study did not measure the level of knowledge students actually gained. Tarng and Tsai on the other hand, found that students who utilized a game-based learning system in conjunction with traditional methods to study geography performed on assessments that students using only traditional methods (2010). Although these studies do not provide irrefutable proof that game-based learning can be used to effectively teach geography, they do help to provide a framework for this research. Initially, this study had intended to test the hypothesis that subjects who play a web-based computer game will perform higher on assessments measuring ones ability to recall a countrys geographic information than students who learn through traditional methods. Due to flaws in the experimental design however, a more accurate hypothesis for this study is that subjects who play a web-based computer game will be able to recall a majority of key geographic information encountered through the course of the game through one exposure, as compared to a control group with continuous exposure.

21

METHODOLOGY
Research Design This study was conducted using an experimental research design in which participants were tasked with memorizing information associated with geographic locations in Jericho, a fictional country. Subjects were directed to review an informed consent form. Separate forms were provided for the control and experimental groups. Participants were then directed to a website constructed expressly for the purposes of this study. Members of the control group reviewed a web page that provided a map and information associated with Jerichos geography. Members of the experimental group played a web-based computer game that exposed them to the same information as the control group. After twenty minutes, participants in both groups completed an assessment designed to measure their ability to recall the information they encountered. Upon completion of the assessment, the researcher provided subjects with a debriefing statement. The entire process was designed to be completed in approximately forty minutes. Variables The independent variable in this experiment was the medium through which participants were exposed to the geographic information. Two mediums were employed in this design: one was a web-based computer game with a graphic and text, while the other was a web page with a graphic and text. The dependent variable in this experiment was the degree to which participants were able to recall information concerning the fictional country. The information recall was measured by the participants score on the assessment, administered immediately after the subject was exposed to the information.

22

Several of the participants also completed a post assessment designed to measure the degree to which participants retained the information they encountered after a period of approximately one month to two months. Attempts were made to control for three variables that had the potential to significantly impact the outcome of the experiment. The experiment utilized a fictional country to ensure that no subject had prior knowledge of the countrys geography. If an actual country had been used it would have been difficult to account for the participants varying levels of familiarity with that country. The experiment was also designed to provide near-identical testing conditions for each session. The experiments were conducted in the same testing room during each of the five time slots. Four of the five experiments occurred from 6 to 7 PM Monday through Thursday. An additional time slot was available Monday from 7 PM to 8 PM. Finally, the instrumentation for both the control and experimental groups was delivered via computer. This accounted for the possibility that differences in delivery methods could have created differences in results. Selection of Participants This study utilized a convenience sample comprised of undergraduate and graduate students in the Institute for Intelligence Studies at Mercyhurst University (IISMU). IIS-MU students, who obtain an undergraduate degree in Intelligence Studies or a masters degree in Applied Intelligence, typically pursue an intelligence analysis career in one of three fields: business, law enforcement, or national security. Thus, these students were selected to serve as a representative sample of entry-level intelligence analysts.

23

Recruitment of Participants The researcher obtained permission from six professors within IIS-MU to visit their classes and recruit participants. Approximately two weeks before the experiments, the researcher visited the classes, informed the students of the experiments purpose and time frame and distributed a sign-up sheet for interested individuals to provide their information. The sign-up sheet required individuals to provide their name, email address, academic level and their professors name. The sheet also asked participants to rank their preference in terms of four one-hour time slots (6 PM to 7 PM, Monday through Thursday) during which they could participate. If students did not have a preference they selected an option that allowed the researcher to assign them a day. An additional time slot was created for Monday (7 PM to 8 PM) to ensure that there were enough computers available for the participants. Several professors also offered extra credit for participating in the experiments; this provided incentive for some students who may not have normally participated or volunteered. The experiment was open to all IIS-MU undergraduate and graduate students. A total of 55 students participated in the experiment: 28 undergraduate and 26 graduate. Of the 28 undergraduate participants, there were: 20 seniors, 4 juniors, 2 sophomores and 2 freshmen. Of the 26 graduate students, 14 were first year graduate students and 12 were second year graduate students. One final participant did not identify his educational level. The control group was comprised of: 7 second year graduate students, 7 first year graduate students, 10 seniors, 1 junior and 1 freshman. The experimental group was comprised of: 5 second year graduate students, 7 first year graduate students, 10 seniors, 3 juniors, 2 sophomores and 1 freshman.

24

Process The researcher submitted a proposal to the Mercyhurst University Institutional Review Board, as required for any student researchers who wishes to conduct an experiment involving human subjects. In order to obtain approval, the researcher had to demonstrate that there were no potential risks to participants health or safety throughout the course of the experiment. The proposal also had to include consent forms, which participants had to sign before participating in the experiments. Instrumentation This study utilized two separate instruments (a web-based computer game and a web page) to delivery geographic information concerning the fictional country of Jericho. A fictional country was chosen in order to ensure that participants had no previous knowledge of the country. Both delivery methods utilized Google Sites, a commercially available web design utility. The text for both delivery methods was typed directly into Google Sites. Additionally, both methods utilized a single map created using GIMP 2, a commercially available image manipulation program. In the web-based computer game Jericho (see Appendix H for screenshots of the game), the small nation undergoes a military coup, during which President Hanson and his senior generals are executed. Players assume the role of a senior bodyguard tasked with protecting Helen Narro, who has succeeded Hanson to the office of the president. Through a series of choices, the player must guide his character, Narro, and her security detail in their attempt to escape Jericho. If at any point the players choices result in Narros death, the game guides the player to the correct choice and continues. Through these choices, players are exposed to Jerichos major cities and the characteristics that

25

make each city unique. A map is displayed at the top of each page throughout the entirety of the game. The web page exposes subjects in the control group to the same map that is used in the web-based computer game. The information associated with Jericho and its cities is displayed below this map in text form. The experimental group was directed to play the game while the control group was directed to review the information available on the web page. Both groups were informed that after twenty minutes, they would complete an assessment measuring their ability to recall the information they encountered. The assessment, which was created using Microsoft Word, was designed to measure the degree to which participants recalled the information that they had been exposed to. The assessment was comprised of eleven questions. Participants were provided with a word bank with which to answer these questions. Words found in the word bank could be reused while completing the assessment. Six of the words in the word bank were correct answers while the remaining ten words were incorrect answers. Three of these ten incorrect answers were similar to the correct answers. The word bank was provided so that subjects would be required only to recognize the information they were exposed to rather than recall the correct answer unassisted. Questions one through ten provided information that the subjects had been exposed to and asked them to recall which city was associated with that information. The eleventh and final question provided participants with a similar map as described above, with the names of each location removed. Subjects were required to label each location on the map using the word bank. Please see Appendix D for a copy of this assessment.

26

Data Analysis Procedures The results of the assessment were entered into a Microsoft Excel spreadsheet. This information was kept separate from the participants names and contact information, in order to preserve each subjects anonymity. The researcher then obtained averages for the number of questions the control and experimental groups correctly answered on the assessment. This included an average of each groups correct answers for the assessment as a whole, the text-based (questions 1 through 10) and map-based (questions 11a through 11f) portions, as well as an average for each individual question. Averages were also found for the post assessment. In order to determine statistical significance, the researcher first applied several normality tests. If normality was satisfied, T tests were conducted in order to determine whether the results were statistically significant (given that both the experimental and control groups numbered less than thirty participants each). If normality was not satisfied, the researcher used Mann-Whitley non-parametric tests. Given the exploratory nature of this research, P values below 0.1 indicated that the data was statistically significant and unlikely to result from chance. If the P values were above 0.1 then the data was not statistically significant and chance could not be ruled out.

27

RESULTS
Overview This study included an initial assessment conducted over a period of four days and a post assessment conducted over a period of 25 days. The post assessment was administered 22 to 57 days after the initial experiments. The initial assessment relied on a sample of 55 participants (29 in the experimental group and 26 in the control group) while the post assessment included 16 participants (8 in the experimental group and 8 in the control group). As a whole, the control group outperformed the experimental group on the initial assessment. The control group scored 19.48% higher on the assessment overall, with scores 24.32% higher on the text-based portion of the assessment and 11.41% higher on the map-based portion of the assessment. P values of 0.000 for the total and text score and 0.016 for the map score indicated that these results were statistically significant and unlikely to result from chance. Additionally, within the control group, undergraduate participants had higher scores than graduate participants on their text-based, map-based and overall scores. The reverse was true for the experimental group (see chart below). In the control group, this comparison of undergraduates and graduates scores had P values of 0.808 for total score, 0.801 for text score, and 0.182 for map score, and thus was not statistically significant. In the experimental group, this comparison had P values of 0.695 for total score, 0.629 for text score, and 0.745 for map score and thus was not statistically significant. These results indicate that educational level had no impact on results in both the experimental and control group.

28

Figure 1.1: Average Score (In Percent) for Group and Educational Level

The post assessment indicated that for the total score, members of the control group better retained the information they encountered during the initial experiments. Results for the text and map-based sections were generally inconclusive. The post assessment included all the questions asked in the initial assessment, with the addition of a final question that asked respondents to rate their interest level while participating in the study. Initial Assessment Text-based Portion (Questions 1 10) The following questions comprised the text-based portion of the assessment: 1. Which city is Jerichos capital? 2. Which city is near Jerichos airport? 3. Which city is home to Jerichos automotive industry? 4. Which city is home to Jerichos casinos? 5. Which city is Jerichos port city? 6. Which city is Jerichos military stronghold? 7. Which citys population is made of pale-skinned individuals? 8. Which city has the most internationally diverse population? 9. Which city is home to Jerichos oil reserves? 10. Which city was going to house Jerichos nuclear power plants?

The purpose of the text-based portion was to assess the degree to which participants were able to recall key information encountered throughout the game for the

29

experimental group or on the informational page for the control group. The control group scored an average of 88.46% correct on the text-based portion while the experimental group scored 64.1%. Approximately 88.46% of the control group scored a 70% or above on the text-based portion, while 58.62% of the experimental group scored a 70% or above. The control group scored higher on every question in the text-based section, as indicated in the graph below. As discussed above, these higher scores were statistically significant and unlikely to result from chance.

Figure 2.1: Average Percent of Correct Answers For Text-Based Section

Question 1 asked Which city is Jerichos capital? As indicated in the graph above, 92.31% of the control group answered this question correctly while 58.62% of the experimental group answered this question incorrectly. A P value of 0.005 indicated that this was statistically significant. All of the undergraduates control group members and 85.71% of the graduate control group members answered this question correctly. In the experimental group, 50% of undergraduates and 75% of graduates answered this question correctly. For the comparison of undergraduates and graduates, P values of 0.181 for both

30

the control and experimental groups indicated that both observations were not statistically significant and again, indicated that education level had no impact on results. Question 2 asked Which city is near Jerichos airport? As indicated in the graph above, 96.15% of the control group and 82.76% of the experimental group answered this question correctly. A P value of 0.115 indicated that this was not statistically significant (it does still approach statistical significance, however). All of the undergraduates and 92.86% of graduates in the control group, as well as 81.25% of undergraduates and 91.67% of graduates in the experimental group answered correctly. For the comparison of undergraduates and graduates, P values of 0.355 for the control group and 0.444 for the experimental group indicated that both observations were not statistically significant. Question 3 asked Which city is home to Jerichos automotive industry? As indicated in the graph above, 84.62% of the control group and 68.97% answered this question correctly. A P value of 0.117 indicated that this was not statistically significant (it does still approach statistical significance, however). Approximately 83.33% of the undergraduates and 85.71% of graduates in the control group, as well as 68.75% of undergraduates and 66.67% of graduates in the experimental group answered correctly. For the comparison of undergraduates and graduates, P values of 0.869 for the control group and 0.909 for the experimental group indicated that both observations were not statistically significant. Question 4 asked Which city is home to Jerichos casinos? As indicated in the graph above, 92.31% of the control group and 68.97% of the experimental group answered this question correctly. A P value of 0.032 indicated that this was statistically significant. All of the undergraduates and 85.71% of the control group, as well as 75% of

31

undergraduates and 58.33% of graduates in the experimental group answered correctly. For the comparison of undergraduates and graduates, P values of 0.181 for the control group and 0.359 for the experimental group indicated that both observations were not statistically significant. Question 5 asked Which city is Jerichos port city? As indicated in the graph above, 96.15% of the control group and 82.76% of the experimental group answered this question correctly. A P value of 0.115 indicated that this was not statistically significant (it does still approach statistical significance, however). All of the undergraduates and 92.86% of graduates in the control group, as well as 75% of undergraduates and 91.67% of graduates in the experimental group answered this question correctly. For the comparison of undergraduates and graduates, P values of 0.355 for the control group and 0.263 for the experimental group indicated that both observations were not statistically significant. Question 6 asked Which city is Jerichos military stronghold? As indicated in the graph above, 80.77% of the control group and 58.62% of the experimental group answered this question correctly. A P value of 0.079 indicated that this was statistically significant. In the control group, 75% of undergraduates and 85.71% of graduates answered this question correctly, along with 50% of undergraduates and 66.67% of graduates in the experimental group. For the comparison of undergraduates and graduates, P values of 0.498 for the control group and 0.397 for the experimental group indicated that both observations were not statistically significant. Question 7 asked Which citys population is made of pale-skinned individuals? As indicated in the graph above, 76.92% of the control group and 58.62% of the

32

experimental group answered this question correctly. A P value of 0.152 indicated that this approached statistical significant. In the control group, 75% of undergraduates and 78.57% of graduates answered this question correctly, along with 50% of undergraduates and 66.67% of graduates in the experimental group. For the comparison of undergraduates and graduates, P values of 0.899 for the control group and 0.397 for the experimental group, indicating that both observations were not statistically significant. Question 8 asked Which city has the most internationally diverse population? As indicated in the graph above, 80.77% of the control group and 42.28% of the experimental group answered this question correctly. A P value of 0.013 indicated that this was statistically significant. This indicated that the control group had more accurate results than the experimental group. In the control group, 75% of undergraduates and 85.71% of graduates answered this question correctly, along with 50% of undergraduates and 41.67% of graduates in the experimental group. For the comparison of undergraduates and graduates, P values of 0.498 for the control group and 0.676 for the experimental group, indicating that both observations were not statistically significant. Question 9 asked Which city is home to Jerichos oil reserves? As indicated in the graph above, 100% of the control group and 58.62% of the experimental group answered this question correctly. A P value of 0.000 indicated that this was statistically significant. Again, indicating better results with the control group as compared to the experimental group. All members of the control group, as well as 62.5% of undergraduates and 50% of graduates in the experimental group answered this question correctly. For the comparison of undergraduates and graduates, P values of 0.498 for the

33

control group and 1.000 for the experimental group, indicating that both observations were not statistically significant. Question 10 asked Which city was going to house Jerichos nuclear power plants? As indicated in the graph above, 84.62% of the control group and 55.17% of the experimental group answered this question correctly. A P value of 0.019 indicated that this was statistically significant; showing a higher control group accuracy than experimental group results. Approximately 83.33% of undergraduates and 85.71% of graduates in the control group, as well as 50% of undergraduates and 58.33% of graduates in the experimental group, answered this question correctly. For the comparison of undergraduates and graduates, P values of 0.869 for the control group and 0.676 for the experimental group, indicating that both observations were not statistically significant. Map-based Portion (Question 11a. 11f.) The purpose of the map-based portion was to assess the degree to which participants were able to recall the names of the cities on the map that both groups encountered. The control group scored an average of 93.59% correct on the map-based portion while the experimental group scored 82.18%. Approximately 92.31% of the control group scored a 70% or above on the text-based portion, while 72.41% of the experimental group scored a 70% or above. The control group also scored higher on every question in the map-based section, as indicated in the graph below. As discussed earlier, the higher control group score for the map section was statistically significant.

34

Figure 3.1: Average Percent Of Correct Answers For Map-Based Section

Question 11a asked participants to label Fallon, one of Jerichos cities, on the map. As indicated in the graph above, 96.15% of the control group and 68.97% of the experimental group answered this question correctly. A P value of 0.01 indicated that this was statistically significant with the control group having more accurate scores as compared to the experimental. All of the undergraduates and 92.86% of the graduates in the control group answered correctly, as compared to 75% of undergraduates and 58.33% of graduates in the experimental group. For the comparison of undergraduates and graduates, P values of 0.365 for the control group and 0.359 for the experimental group, indicating that both observations were not statistically significant. Question 11b asked participants to label Oteak. As indicated in the graph above, 92.31% of the control group and 82.76% of the experimental group answered this question correctly. A P value of 0.293 indicated that this was not statistically significant. All of the undergraduates and 85.71% of graduates in the control group answered correctly, as compared to 87.5% of undergraduates and 83.33% of graduates in the

35

experimental group. For the comparison of undergraduates and graduates, P values of 0.181 for the control group and 0.759 for the experimental group, indicating that both observations were not statistically significant. Question 11c asked participants to label Jericho Airport. As indicated in the graph above, 92.31% of the control group and 89.66% of the experimental group answered this question correctly. A P value of 0.735 indicated that these values were not statistically significant. All of the undergraduates and 85.71% of graduates in the control group answered correctly, as compared to 93.75% of undergraduates and 91.67% in the experimental group. For the comparison of undergraduates and graduates, P values of 0.181 for the control group and 0.835 for the experimental group, indicating that both observations were not statistically significant. Question 11d asked participants to label Basilogne, Jerichos capital city. As indicated in the graph above, 92.31% of the control group and 82.76% of the experimental group answered this question correctly. A P value of 0.293 indicated that this was not statistically significant. All of the undergraduates and 85.71% of the graduates in the control group answered this question correctly, as compared to 81.25% of undergraduates and 91.67% of graduates in the experimental group. For the comparison of undergraduates and graduates, P values of 0.181 for the control group and 0.444 for the experimental group indicated that both observations were not statistically significant. Question 11e asked participants to label Huntington. As indicated in the graph above, 92.31% of the control group and 82.76% of the experimental group answered this question correctly. A P value of 0.293 indicated that this was not statistically significant.

36

All of the undergraduates and 85.71% of graduates in the control group answered correctly, as compared to 81.25% of undergraduates and 91.67% of graduates in the experimental group. For the comparison of undergraduates and graduates, P values of 0.181 for the control group and 0.444 for the experimental group indicated that both observations were not statistically significant. Question 11f asked participants to label Caidana. As indicated in the graph above, 96.15% of the control group and 86.21% of the experimental group answered this question correctly. A P value of 0.204 indicated that this was not statistically significant. All of the undergraduates and 92.86% of the graduates in the control group answered correctly, as compared to 81.25% of undergraduates and 91.67% of graduates in the experimental group. For the comparison of undergraduates and graduates, P values of 0.365 for the control group and 0.444 for the experimental group indicated that both observations were not statistically significant. Post Assessment The post assessment indicated that the control group was able to recall information more accurately than the experimental group. The control group answered questions correctly on the text-based portion 57.5% of the time, questions on the mapbased portion 50% of the time, and on the assessment as a whole, 54.69% of the time. By contrast, the experimental group answered questions correctly on the text-based portion 28.75% of the time, questions on the map-based portion 33.33% of the time, and on the assessment as a whole, 30.47% of the time. P values of 0.083, 0.317, and 0.131 for the overall assessment, text-based portion, and map-based portion respectively, indicate that

37

the difference in overall score was statistically significant while the difference in scores for the text and map-based portions were not. Text-based Portion (Questions 1 10) As with the initial assessment, the following questions comprised the text-based portion of the post assessment: 1. Which city is Jerichos capital? 2. Which city is near Jerichos airport? 3. Which city is home to Jerichos automotive industry? 4. Which city is home to Jerichos casinos? 5. Which city is Jerichos port city? 6. Which city is Jerichos military stronghold? 7. Which citys population is made of pale-skinned individuals? 8. Which city has the most internationally diverse population? 9. Which city is home to Jerichos oil reserves? 10. Which city was going to house Jerichos nuclear power plants?

The purpose of the text-based portion was to assess the degree to which participants were able to recall key information encountered throughout the game for the experimental group or on the informational page for the control group. The control group scored an average of 57.5% correct on the text-based portion while the experimental group scored 28.75%. The control group scored 70% or above on three of the questions on the text-based portion while the entire control group scored 50% or below on all textbased questions. The control group scored equal to or higher than the experimental group on every question in the text-based section, as indicated in the graph below. Only three of these observations (questions 7, 8, and 10) were statistically significant and unlikely to result from chance. Question 1 had a P value of 0.117 and thus approached statistical significance.

38

Figure 4.1: Average Percent of Correct Answers for Text-Based Section

Map-based Portion (Question 11a. 11f.) The purpose of the map-based portion was to assess the degree to which participants were able to recall the names of the cities on the map that both groups encountered. The control group scored an average of 50% correct on the map-based portion while the experimental group scored 33.33%. Both the control and experimental group scored a 70% or above on only one question (11c). None of these results were statistically significant.

39

Figure 5.1: Average Percent of Correct Answers for Map-Based Section

Interest Level The post assessment also included a question that asked respondents to rate their interest level on a scale of -2 to 2 while participating in the study. A -2 indicated that the participant was very uninterested in the study, a -1 that the participant was uninterested, a 0 that the participant felt neutral while participating, a 1 that the participant was interested, and a 2 that the participant was very interested. On average, the control group rated their experience at 0.875 while the experimental group rated their experience at a 0.625. A P value of 0.405 indicates that this observation is not statistically significant and could have resulted from chance.

40

CONCLUSION
Initially, the purpose of this experiment was to determine if game-based learning was a more effective tool than rote memorization for teaching entry-level intelligence analysts to quickly familiarize themselves with an unfamiliar geographic region and its key information. As mentioned previously however, due to flaws in the experimental design, the experiment actually produced a comparison of implicit and explicit learning. Subjects in the experimental group learned implicitly through a single exposure to the information in Jericho, while subjects in the control group learned through continuous rote memorization. The results of this study partially support the hypothesis that subjects who play a web-based computer game will be able to recall a majority of key geographic information encountered through the course of the game through one exposure, as compared to a control group with continuous exposure. The experimental group was able to recall a majority of the geographic information, but its scores were still lower than the control groups scores. As detailed in the methodology section, members of the experimental group were asked to play Jericho and members of the control group were asked to review an informational web page. Both groups had access to the same map of Jericho, but as an unintentional consequence of the experimental design, the experimental group was unable to review previously visited cities. While the map was available regardless of the players location, as soon as he or she passed through one city into the next, he was unable to review the key information associated with the previously visited city. Research has shown that repetition has a significant impact on recall (Castel, Kornell, Rhodes, & Tauber, 2011). The control group on the other hand, was able to simply scroll up and

41

down the page to review the important information. This ability to revisit previous information in the control group, and the lack thereof in the experimental group, likely had a significant impact on the scores each group attained on the assessment. For this reason, the comparison of the control and experimental group may be a comparison of apples and oranges. Regardless, it is important to note that with just a single exposure to each city, the experimental group was able to correctly answer 70.91% of the questions on the assessment. In other words, by simply reading the information once, players were able to correctly recall key information over 70% of the time. In addition to this, they were exposed to extraneous information that helped to inform the decisions they made throughout the game. If the game had dealt with a real country, then players would have been exposed to cultural and societal information about the country in question, in addition to learning about the countrys geography. This is especially important for intelligence analysts, who must understand geography, politics, culture, and a wide variety of other factors when producing an assessment. In the case of Jericho, players learn that in the event of the death of Jerichos president, the vice president succeeds to the office of the president. Obviously, this is a relatively easy concept to grasp. If in the game the vice president also killed however, then the game could center on protecting whoever is next in the line of succession. This concept could easily be adapted to reflect cultures that are more difficult for westerners to comprehend, such as different lines of succession that divides Shia Islam into several sects.

42

The control group also outperformed the experimental group on all questions in the post assessment, although these results were not statistically significant and could have resulted from chance. The small sample sizes (8 participants per group) and the wide span of time over which participants responded to the post assessment (22 to 57 days after the initial experiments) makes it very difficult to draw any significant conclusions from the post assessment results. The experimental group answered 28.75% of the assessment correctly, as compared to 57.5% for the control group. It is still remarkable however, that the experimental group was able to recall just over a quarter of the information 22 to 57 days after they encountered it, even though they only reviewed the material once. The post assessment also asked respondents to recall and rate their interest level while participating in the study. It is interesting to note that the control group collectively reported a higher interest rate (.875) than the experimental group (.625). This observation was not statistically significant however, and could have resulted from chance. Both the literature and common sense suggest that playing a game should be more fun and engaging and thus produce a greater interest level, than simply reading a page and memorizing its contents. Recommendations for Future Research This study was amongst the first ones examining the application of game-based learning to intelligence studies. While it has hopefully helped in laying the foundation for future studies, there is still much ground to cover. This study for example, creates an opportunity for future researchers to determine whether the experimental groups scores will increase if subjects play Jericho multiple times. Such a study could seek to

43

determine the number of plays required for the experimental group to equal or surpass the control group. Other studies centered on game-based learning, geography, and intelligence analysis could attempt to expose control groups to a greater amount of irrelevant information in order to more accurately simulate the chaotic real-world conditions found in intelligence analysis. Such studies could also utilize larger sample sizes to increase the chances of obtaining statistical significance. Finally, these studies could analyze demographic information, such as age or gender, to identify any potential trends. There is also a large breadth of potential game-based learning research topics outside of geography. As mentioned in the literature review, IARPAs Sirius Program examining cognitive bias and Professor Wheatons work on pattern recognition, intelligence production and conceptual comprehension are the only other bodies of research on the use of game-based learning in intelligence studies. The ability of analysts to create an Intelligence Preparation of the Battlefield (IPB), as discussed in the literature review, could present a potential topic for future research. Such a study could expose an experimental group to a military strategy game and determine whether this group could conduct a more thorough or accurate IPB than the control group. A recent video game, L.A. Noire, may also present an additional topic for future research. In this game, players assume the role of a Los Angeles detective who often encounters suspects who are actively trying to deceive him. Researchers could examine whether this game allows individuals to better detect deception, as compared to individuals who read literature on detecting deception. Ultimately, research has shown that game-based learning alone is not the cure-all for the problems found in American education. Research seeking to

44

determine the proper combination of game-based and traditional learning methods will likely present the most valuable findings for the field of intelligence studies. These ideas are but a fraction of the potential research topics that await individuals who wish to examine the nexus of game-based learning and intelligence studies.

45

REFERENCES
Abt, C.C. (1987). Serious games. Lanham, MD: University Press of America, Inc. Adams, D.M. (1973). Simulation games: An approach to learning. Worthington, OH: Charles A. Jones Publishing Company. Admiraal, W., Akkerman, S., Huizenga, J., & ten Dam, G. (2009). Mobile game-based learning in secondary education: Engagement, motivation and learning in a mobile city game. Journal of Computer Assisted Learning, 25, 332-344. Boocock, S.S., & Coleman, J.S. (1966). Games with simulated environments in learning. Sociology of Education, 39(3), 215-236. Boocock, S.S., & Schild, E.O. (1968). Simulation games in learning. Beverly Hills, CA: Sage Publications, Inc. Bowman, R.F. (1982). A pac-man theory of motivation: Tactile implications for classroom instruction. Educational Technology, 22(9), 14-17. Bckle, M., Ebner, M., & Schn, M. (2011). Game based learning in secondary education: Geographical knowledge of Austria. Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2011. 15101515. Branom, M. (2005). Military simulations let warriors learn their craft in peace. U.S. Army PEO STRI. Retrieved from http://web.archive.org/web/20050108134507/http://www.peostri.army.mil/PAO/p ressrelease/Sniper.jsp

46

Brom, C., Sisler, V., & Slavik, R. (2008). Implementing digital game-based learning in schools: Augmented learning environment of Europe 2045. Multimedia Systems, 16(1), 23-41. Candler, C. & Meeuwsen, H. (2002). Implicit learning in children with and without developmental coordination disorder. The American Journal of Occupational Therapy, 56(4), 429-435. Carlson, E. (1969). Learning through games. Washington, DC: Public Affairs Press. Castel, A.D., Kornell, N., Rhodes, M.G., & Tauber, S.K. (2011). The ease-of-processing heuristic and the stability bias: Dissociating memory, memory beliefs, and memory judgments. Psychological Science. Retrieved from http://pss.sagepub.com/content/22/6/787 Charsky, D. (2010). From edutainment to serious games: A change in the use of game characteristics. Games and Culture, 5(2), 177-198. Chido, D.E et al. (2006). Structured analysis of competing hypotheses: Theory and application. Erie, PA: Mercyhurst College Institute of Intelligence Studies Press. Ciavarro, C., Dobson, M., & Goodman, D. (2006). Alert hockey: An endogenous learning game. 2006 Authors & Canadian Games Study Association Symposium. Ciavarro, C., Dobson, M., & Goodman, D. (2008). Implicit learning as a design strategy for learning games: Alert Hockey. Computers in Human Behavior, 24(6), 28622872. Cleeremans, A. (1996). Principles for implicit learning. Sminaire de Recherche en Sciences Cognitives, 1-12.

47

Cleeremans, A. & French, R.M. (2002). Implicit learning and consciousness: An empirical, philosophical and computational consensus in the making. New York, NY: Psychology Press. Cheng, C., Hong, J., Hwang, M., Lee, Y., Lin, C., & Lu, C. (2009). Playfulness-based design in educational games: A perspective on an evolutionary contest game. Interactive Learning Environments, 17(1), 15-35. Clark, R.M. (2010). Intelligence analysis: A target-centric approach. Washington, DC: CQ Press. Cruickshank, D.R., & Telfer, R. (2001). Classroom games and simulations. Theory Into Practice, 19(1), 75-80. Csikszentmihalyi, M. Flow: Psychology of optimal experience. New York, NY: Harper Perennial. De Castell, S., & Jenson, J. (2003). Serious play. Journal of Curriculum Studies, 35(6), 649-665. Dorken, M.D., & Whittlesea, B.W.A. (1993). Incidentally, things in general are particularly determined: An episodic-processing account of implicit learning. Journal of Experimental Psychology, 122(2), 227-248. Doty, J.M. (2005). Geospatial intelligence: An emerging discipline in national intelligence with an important security assistance role. Defense Institute of Security Assistance Management Journal. Dunn, J.C., & Kirsner, K. (1989). Implicit memory: Theoretical issues. Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers. Eden, S.J. (1997).Knowledge-based warfare implications. Military Review, 77(2).

48

Egenfeldt-Nielsen, S. (n.d.). Making sweet music: The educational use of computer games. Center for Computer Games Research. Entertainment Software Rating Board. (2009). How much do you know about video games?. Retrieved from http://www.esrb.org/about/video-game-industrystatistics.jsp Fletcher, J.L. (1971). The effectiveness of simulation games as learning environments: A proposed program of research. In Coleman, J.S., & Edwards, K.J. (Eds.), Simulation & games: An international journal of theory, design, and research (pp. 425-454). Beverly Hills, CA: Sage Publications. Gee, J.P. (2005). Good video games and good learning. Phi Kappa Phi Forum, 85(2), 3337. Gros, B. (2007). Digital games in education: The design of games-based learning environments. Journal of Research on Technology in Education, 40(1), 23-38. Heitzmann, W.R. (2001). The validity of social science stimulations: A review of research findings. Social Science Stimulations, 170-173. Hitz, F.P. (2001, October 21). Not just a lack of intelligence, a lack of skills. The Washington Post. Retrieved from http://www.washingtonpost.com/ac2/wpdyn/A24912-2001Oct20? Intelligence Advanced Research Projects Activity. (2011). Solicitations office of incisive analysis. Retrieved from http://www.iarpa.gov/solicitations_sirius.html Koster, R. (2005). A theory of fun for game design. Scottsdale, AZ: Paraglyph Press, Inc.

49

Kusumoto, L., Mayo, M., & Singer, M.J. (2006). Massively multi-player (MMP) environments for asymmetric warfare. Journal of Defense Modeling & Simulation, 3(3), 155-166. Lesgold, A.M. (2001). The nature and methods of learning by doing: American Psychologist, 964-973. Lowenthal, M.M. (2009). Intelligence: From secrets to policy (4th ed.). Washington, DC: CQ Press. Malone, T.W. (1981a). Toward a theory of intrinsically motivating instruction. Cognitive Science, 4, 333-369. Malone, T.W. (1981b). What makes computer games fun? BYTE. Marrin, S. (2003). CIAs kent school: A step in the right direction. International Journal of Intelligence and Counterintelligence, 16, 609-637. doi: 10.1080/08850600390198779 McCreery, M., & Schrader, P.G. (2008). The acquisition of skill and expertise in massively multiplayer online games. Education Tech Research Dev, 56, 557-574. Doi: 10.1007/s11423-007-9055-4 Miller, F.A. (2009). Irregular warfare perhaps not so irregular. USAWC Strategy Research Project. Nieborg, D.B. (2004). Americas army: More than a game?. Transforming Knowledge into Action through Gaming and Simulation. Nieborg, D.B., & van der Graaf, S. (2003). Together we brand: Americas army. Level Up: Digital Games Research Conference, 324-38.

50

Poletiek, F.H., & van den Bos, E. (2009). Structrual selection in implicit learning of artificial grammars. Psychological Research, 138-151. doi: 10.1007/s00426-0090227-1 Prensky, M. (2001). Digital game-based learning. New York, NY: McGraw-Hill Companies, Inc. Provenzo, E.F. (1991). Video kids: Making sense of nintendo. Cambridge, MA: Harvard University Press. Reeber, A.S. (1993). Implicit learning and tacit knowledge. New York, NY: Oxford University press, Inc. Salen , K., & Zimmerman, E. (2004). Rules of play: Game design fundamentals. Cambridge, MA: Massachusetts Institute of Technology. Squire, K. (2005). Game-based learning: Present and future state of the field. Madison, WI: University of Wisconsin-Madison Press. Tarng, W. & Weichian, T. (2010). The design and analysis of learning effects for a gamebased learning system. World Academy of Science, Engineering and Technology, 61, 336-346. Taylor, J.L., & Walford, R. (1972). Simulation in the classroom. Middlesex, England: Penguin Books Ltd. United States Army. (1990). Intelligence analysis. Washington, DC: Department of the Army. United States Army. (1994). Intelligence preparation of the battlefield. Washington, DC: Department of the Army.

51

Wheaton, K.J. (2011). Teaching strategic intelligence through games. Erie, PA: Mercyhurst University. White, P.J. (2008). Geography: A new domain for the united states air forces crosscultural competency framework. Maxwell Air Force Base, AL: Air Command and Staff College Air University. Whitehead, A.N. (1929). The aim of education. Cambridge, England: Cambridge University Press.

52

Annex
Annex A: Institutional Review Board Proposal Form Date Submitted: 10/14/2011 Investigator(s): Daniel Burch Investigator Address: 10309 Gainsborough Rd, Potomac, MD 20854 Investigator(s) E-mail: dburch11@lakers.mercyhurst.edu Investigator Telephone Number: 240-486-4806

53

Advisors Name (if applicable): Professor Kristan Wheaton Advisors E-mail: kwheaton@mercyhurst.edu Advisors Signature of Approval: [X] Place X here if advisor has approved research Title of Research Project: Game-based Learning in Intelligence Date of Initial Data Collection: Dec 5, 2011 (approximate estimate)

54

Please describe the proposed research and its purpose, in narrative form: The proposed research will examine whether entry-level intelligence analysts can learn the details of a geographic region and the particulars of a given situation more rapidly by playing a web-based computer game than through traditional lecture and discussion..

Indicate the materials, techniques, and procedures to be used (submit copies of materials): I have attached the documents below to the en d of this proposal: 1. Informed Consent Form Control Group 2. Informed Consent Form Experimental Group 3. Debriefing Statement Form 4. Proposed Methodology 5. Proposed Assessment 6. Screenshots of game a. Please note that the game is not yet complete. I have attached the screenshots to provide a basic understanding of how the game will be played.

1. Do you have external funding for this research (money coming from outside the College)? Yes[ ] No[x] Funding Source (if applicable): N/A 2. Will the participants in your study come from a population requiring special protection; in other words, are your subjects someone other than Mercyhurst College students (i.e., children 17-years-old or younger, elderly, criminals, welfare recipients, persons with disabilities, NCAA athletes)? Yes[ ] No[x] If your participants include a population requiring special protection, describe how you will obtain consent from their legal guardians and/or from them directly to insure their full and free consent to participate. N/A Indicate the approximate number of participants, the source of the participant pool, and recruitment procedures for your research: This study will include approximately 100 students from the Mercyhurst College Department of Intelligence Studies. I will seek permission from professors to visit their classes and ask for volunteers. I will also request that the Department of Intelligence Studies send an email over its mailing list.

55

Will participants receive any payment or compensation for their participation in your research (this includes money, gifts, extra credit, etc.)? Yes[x] No[ ] If yes, please explain: Pizza and soda will be available for the participants. I will also determine if professors would like to award extra credit to those who participate in the study. 3. Will the participants in your study be at any physical or psychological risk (risk is defined as any procedure that is invasive to the body, such as injections or drawing blood; any procedure that may cause undue fatigue; any procedure that may be of a sensitive nature, such as asking questions about sexual behaviors or practices) such that participants could be emotionally or mentally upset? Yes[ ] No[x] Describe any harmful effects and/or risks to the participants health, safety, and emotional or social well being, incurred as a result of participating in this research, and how you will insure that these risks will be mitigated: N/A 4. Will the participants in your study be deceived in any way while participating in this research? Yes[ ] No[x] If your research makes use of any deception of the respondents, state what other alternative (e.g., non-deceptive) procedures were considered and why they werent chosen: N/A 5. Will you have a written informed consent form for participants to sign, and will you have appropriate debriefing arrangements in place? Yes[x] No[ ] Describe how participants will be clearly and completely informed of the true nature and purpose of the research, whether deception is involved or not (submit informed consent form and debriefing statement): I will provide an informed consent form and a debriefing statement form for all participants. Please include the following statement at the bottom of your informed consent form: Research at Mercyhurst College which involves human participants is overseen by the Institutional Review Board. Questions or problems regarding your rights as a participant should be addressed to Mr. Timothy Harvey; Institutional Review Board Chair; Mercyhurst College; 501 East 38th Street; Erie, Pennsylvania 165460001; Telephone (814) 824-3372. b. Describe the nature of the data you will collect and your procedures for insuring that confidentiality is maintained, both in the record keeping and presentation of this data:

56

Students names will not be associated with any data that is collected. I will assign each participant a letter/number combination indicating whether he or she is in the control or experimental group. Upon completion of the individuals participation, he or she will complete a geography-test. Initially each participants test results will be on paper. I will personally safeguard these test results until returning to my apartment. I will then record the results on my personal computer and shred the original paper results. My computer is password protected and I am the only one with access to its files. I will share the names of students with professors from whom they are seeking extra credit. No data aside from the students names will be shared with the professors. Identify the potential benefits of this research on research participants and humankind in general. I designed this study to test the hypothesis that through game-based learning, entry-level analysts will be able to develop a more rapid familiarization with new terrain and a new situation, in a manner that allows them to provide better intelligence to the customers they support. If the results support my hypothesis, the Mercyhurst College Department of Intelligence Studies will have experimental results that support the use of games in teaching the skills necessary for intelligence analysis. Additionally, such results would create opportunities for further study using other skills that are necessary for conducting intelligence analysis. Participants will also gain an increased familiarity with gamebased learning options, an opportunity to further the academic development of intelligence studies, and an appreciation of the thesis process for Mercyhurst Colleges Masters of Science in Intelligence Studies program. b.

57

Annex B: Control Group Consent Form The purpose of this research is to test whether or not playing a computer game can help entry-level intelligence analysts learn the geography of an unfamiliar region more rapidly than the traditional method of memorizing geographic features on a map. Your participation involves a short period during which you will review a map, followed by an assessment of your ability to recall information concerning key locations on the map. This process should take no longer than 30 minutes. Your results will not be associated with your name throughout the entire process. There are no foreseeable risks or discomforts associated with your participation in this study. Participation is voluntary and you have the right to opt out of the study at any time for any reason without penalty. By signing this informed consent form, you hereby acknowledge that your involvement in this research is voluntary and agree that your data can be used for the purpose of this research. In addition, by signing this form, you are confirming that you are at least 18 years of age or have parental consent to participate in this survey. If you have any further questions about this research feel free to contact me at dburch11@lakers.mercyhurst.edu Research at Mercyhurst College which involves human participants is overseen by the Institutional Review Board. Questions or problems regarding your rights as a participant should be addressed to Tim Harvey; Institutional Review Board Chair; Mercyhurst College; 501 East 38th Street; Erie, Pennsylvania 16546-0001; Telephone (814) 8243372. tharvey@mercyhurst.edu

_____________________________________ Signature _____________________________________ Name

58

Annex C: Experimental Group Consent Form The purpose of this research is to test whether or not playing a computer game can help entry-level intelligence analysts learn the geography of an unfamiliar region more rapidly than the traditional method of memorizing geographic features on a map. Your participation involves a short period during which you will a web-based computer game, followed by an assessment of your ability to recall information concerning key locations encountered during the game. This process should take no longer than 30 minutes. Your results will not be associated with your name throughout the entire process. There are no foreseeable risks or discomforts associated with your participation in this study. Participation is voluntary and you have the right to opt out of the study at any time for any reason without penalty. By signing this informed consent form, you hereby acknowledge that your involvement in this research is voluntary and agree that your data can be used for the purpose of this research. In addition, by signing this form, you are confirming that you are at least 18 years of age or have parental consent to participate in this survey. If you have any further questions about this research feel free to contact me at dburch11@lakers.mercyhurst.edu Research at Mercyhurst College which involves human participants is overseen by the Institutional Review Board. Questions or problems regarding your rights as a participant should be addressed to Tim Harvey; Institutional Review Board Chair; Mercyhurst College; 501 East 38th Street; Erie, Pennsylvania 16546-0001; Telephone (814) 8243372. tharvey@mercyhurst.edu

_____________________________________ Signature _____________________________________ Name

59

Annex D: Initial Assessment Form

Educational level: ___Freshman ___Sophomore ___Junior ___Senior Control Group ___ ___G1 ___G2

Experimental Group ___

Please use the word bank below to complete this assessment. Please note that some or all of the words below may used in response to multiple questions. Word Bank Huntington Alvaro Horus Narro

Berrington Fallon Plaisan Guadal

Caidana Jericho Curiseau Basilogne

Ferrus Airport Rogal Jericho Airport Oteak

1. Which city is Jerichos capital?

2. Which city is near Jerichos airport?

3. Which city is home to Jerichos automotive industry?

4. Which city is home to Jerichos casinos?

5. Which city is Jerichos port city?

6. Which city is Jerichos military stronghold?

7. Which citys population is made of pale-skinned individuals?

8. Which city has the most internationally diverse population?

60

9. Which city is home to Jerichos oil reserves?

10. Which city was going to house Jerichos nuclear power plants?

11. Please label the following map using the names from the word bank.

b. a.

c.

d.

e.

f.

11a.

11b.

11c.

11d.

11e.

11f.

61

Annex E: Debriefing Statement Form Thank you for participating in this research process. I appreciate your contribution and willingness to support the student research process. It is of utmost importance that you do not discuss the information presented in this experiment, or your answers to the assessment, with any possible future participants. Discussion with any prospective participants may manipulate or alter their results. The purpose of this study was to test whether or not playing a computer game can help entry-level intelligence analysts learn the geography of an unfamiliar region more rapidly than the traditional method of memorizing geographic features on a map. Both the control and experimental groups are completing the same assessment. The control group prepares for this assessment by reviewing a map with geographic locations and associated information. The experimental group prepares for this assessment by playing a web-based computer game that exposes participants to the same locations and associated information. I will compile your responses to the assessment with other participants responses in the control and experimental groups. I will then compare the results of each groups assessments to see whether or not playing the web-based computer game was more effective in assisting entry-level intelligence analysts become familiar with the geographic region used in this experiment. Educators and researchers have devoted extensive amounts of time and effort to gamebased learning. This research is important because to date, no formal research has examined theuse of game-based learning in teaching skills necessary for intelligence analysis. If this study reveals that game-based learning can be an effective teaching tool for intelligence studies, more opportunities will exist for future research focusing on the application of game-based learning to intelligence studies. Thank you once again for your participation. If you have any further question about my research you can contact me at dburch11@lakers.mercyhurst.edu

62

Annex F: Post Assessment Form

Educational level: ___Freshman ___Sophomore ___Junior ___Senior Control Group ___ ___G1 ___G2

Experimental Group ___

Please use the word bank below to complete this assessment. Please note that some or all of the words below may used in response to multiple questions. Word Bank Huntington Alvaro Horus Narro

Berrington Fallon Plaisan Guadal

Caidana Jericho Curiseau Basilogne

Ferrus Airport Rogal Jericho Airport Oteak

1. Which city is Jerichos capital?

2. Which city is near Jerichos airport?

3. Which city is home to Jerichos automotive industry?

4. Which city is home to Jerichos casinos?

5. Which city is Jerichos port city?

6. Which city is Jerichos military stronghold?

7. Which citys population is made of pale-skinned individuals?

8. Which city has the most internationally diverse population?

9. Which city is home to Jerichos oil reserves?

10. Which city was going to house Jerichos nuclear power plants?

63

11. Please label the following map using the names from the word bank.

b. a.

c.

d.

e.

f.

11a.

11b.

11c.

11d.

11e.

11f.

12. Please indicate your level of interest while participating in this study. Very uninterested __ Uninterested __ Neutral __ Interested __ Very interested __

64

Annex G: Statistical Data Initial Assessment Is there a difference between experimental and control for total score? Null: there is normality for the experimental and control groups for total score. Alternative: there is not normality for the experimental and control groups for total score. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Total Experimenta .178 29 .914 29 .020 .021 Score l Control .348 26 .590 26 .000 .000 b. Lilliefors Significance Correction

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

Most points are close to the line, no significant curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the experimental group.

65

Most points are close to the line (except one), no significant curvature, and evenly distributed along the line, thus assumption of normality is satisfied for the control group.

For the experimental group there are no outliers, which is conducive for normality. For the control group there are 3 outliers, which is not conducive for normality. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test for independent samples.

Test Statisticsa Total Score 170.500 605.500 -3.606 .000

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2tailed) b. Grouping Variable: Group

Mann-Whitney test statistic = -3.606 (Mann-Whitney test P-value = 0.000) < ( = 0.1) thus reject the null hypothesis.

66

Conclusion: There is a significant difference between the experimental and control groups for total score. --Is there a difference between the experimental and control groups for text score? Null: there is normality for the experimental and control groups for text score. Alternative: there is not normality for the experimental and control groups for text score. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Text Experimental .168 29 .923 29 .035 .036 Score Control .385 26 .590 26 .000 .000 b. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

Most points are close to the line, no significant curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the experimental group.

67

Most points are close to the line (except one), slight curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the control group.

For the experimental group there are no outliers, which is conducive for normality. For the control group there are 5 outliers, which violates normality. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test for independent samples.

Test Statisticsa Text Score 166.000 601.000 -3.703 .000

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2tailed) b. Grouping Variable: Group

Mann-Whitney test statistic = -3.7.3 (Mann-Whitney test P-value = 0.000) < ( = 0.1) thus reject the null hypothesis.

68

Conclusion: There is a significant difference between the experimental and control groups for text score. --Is there a difference between the experimental and control groups for map score? Null: there is normality for the experimental and control groups for map score. Alternative: there is not normality for the experimental and control groups for map score. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Map Experimental .357 29 .692 29 .000 .000 Score Control .532 26 .308 26 .000 .000 b. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

Most points are not close to the line (except one) and not evenly distributed along the line, thus the assumption of normality is not satisfied for the control group.

69

Most points are not close to the line, slight curvature, and not evenly distributed across the line, thus normality is not satisfied for the control group.

For the experimental group there is one outlier. This is acceptable for normality. For the control group there are 2 outliers. Thus normality is not satisfied for group control. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test for independent samples.

Test Statisticsa Map Score 271.000 706.000 -2.400 .016

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2tailed) b. Grouping Variable: Group

Mann-Whitney test statistic = -2.4 (Mann-Whitney test P-value = 0.016) < ( = 0.1) thus reject the null hypothesis.

70

Conclusion: There is a significant difference between the experimental and control groups for map score. --Is there a difference between undergraduates and graduates for total score in the control group? Null: there is normality for undergraduates and graduates for total score in the control group. Alternative: there is not normality for undergraduates and graduates for total score in the control group. Assumption checking: Tests of Normality Group Kolmogorov-Smirnova Shapiro-Wilk Statistic df Sig. Statistic df Sig. Total Undergraduates .401 12 .000 .687 12 .001 Score Graduates .381 14 .000 .588 14 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

Most points are close to the line (except one), no significant curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for undergraduates.

71

Half the points are not close to the line and not evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

For undergraduates there are no outliers, which is conducive for normality. For control group there are3 outliers. Thus normality is not satisfied for graduates. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test for independent samples.

Test Statisticsb Total Score 80.000 185.000 -.242 .808 .860a

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)] a. Not corrected for ties. b. Grouping Variable: Group

Mann-Whitney test statistic = -.242

72

(Mann-Whitney test P-value = 0.808) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the control group for total score. --Is there a difference between undergraduates and graduates for text score in the control group? Null: there is normality for undergraduates and graduates for text score. Alternative: there is not normality for undergraduates and graduates for text score. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Text Undergraduates .401 12 .687 12 .000 .001 Score Graduates .388 14 .519 14 .000 .000 b. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

Most points are close to the line, no significant curvature, and evenly distributed along the line, thus assumption of normality is satisfied for undergraduates.

73

Most points are not close to the line (except one) and not evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

For experimental group there are no outliers, which is conducive for normality. For the control group there are 2 outliers. Thus normality is not satisfied for group control. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test for independent samples.

Test Statisticsb Text Score 80.000 158.000 -.252 .801 .860a

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)] a. Not corrected for ties. b. Grouping Variable: Group

Mann-Whitney test statistic = -0.252 (Mann-Whitney test P-value = 0.801) > ( = 0.1) thus fail to reject the null hypothesis. 74

Conclusion: There is no difference between undergraduates and graduates for text score in the control group. --Is there a difference between undergraduates and graduates for map score in the control group? Null: there is normality for undergraduates and graduates for map score in the control group. Alternative: there is not normality for undergraduates and graduates for map score in the control group. Assumption checking: Tests of Normalityb Kolmogorov-Smirnova Shapiro-Wilk Statistic df Sig. Statistic df Sig. .507 14 .000 .447 14 .000

Group

Map Graduates Score a. Lilliefors Significance Correction b. Map Score is constant when Group = Under-graduates. It has been omitted.

Since undergraduates all had the same map score, it was not possible to calculate normality using a test of normality or a normality quantile plot. Thus, it was necessary to use a Mann-Whitney Test. Test Statisticsb Map Score 72.000 177.000 -1.335 .182 .560a

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)] a. Not corrected for ties. b. Grouping Variable: Group

Mann-Whitney test statistic = -1.335

75

(Mann-Whitney test P-value = 0.182) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no difference between undergraduates and graduates for map score in the control group. --Is there a difference between undergraduates and graduates for total score in the experimental group? Null: there is normality for undergraduates and graduates for total score in the experimental group. Alternative: there is not normality for undergraduates and graduates for total score in the experimental group. Assumption checking: Tests of Normality Kolmogorov-Smirnova Statistic df Sig. .222 16 .034 .200*

Graduates .141 12 *. This is a lower bound of the true significance. a. Lilliefors Significance Correction

Total Score

Group Undergraduate s

Shapiro-Wilk Statistic df Sig. .870 .898 16 12 .027 .150

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are greater than (a = 0.10) thus normality assumption is satisfied. Independent Samples Test Levenes t-test for Equality of Means Test for Equality of Variances F Sig. t df Sig. (2Mean Std. Error tailed) Difference Difference Total Score Equal variances assumed Equal variances not assumed .876 .358 -.396 -.412 26 25.9 84 .695 .684 -3.90625 -3.90625 9.86310 9.48035

76

As per Levenes test: Levenes Test statistic = 0.876 (Levenes P-value = 0.358) > (a = 0.10), thus assumption of equal variances is satisfied. t-test statistic = -0.396 (t-test P-value = 0.695) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no difference between undergraduates and graduates for total score in the experimental group. --Is there a difference between undergraduates and graduates for text score in the experimental group? Null: there is normality for undergraduates and graduates in the experimental group for text score. Alternative: there is not normality for undergraduates and graduates in the experimental group for text score. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Undergraduates .174 16 .893 16 .200* .061 Text * Score Graduates .180 12 .889 12 .200 .114 *. This is a lower bound of the true significance. a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are greater than (a = 0.10) thus normality assumption is satisfied. Independent Samples Test Levenes t-test for Equality of Means Test for Equality of Variances 77

F Equal variances assumed Equal variances not assumed

Sig.

Df

Sig. (2Mean tailed) Difference .629 .620 -5.41667 -5.41667

Std. Error Difference 11.06839 10.79173

Text Score

.203

.656

-.489

26

-.502 25.589

As per Levenes test: Levenes Test statistic = 0.203 (Levenes P-value = 0.656) > (a = 0.10), thus assumption of equal variances is satisfied. T test statistic = -0.489 (T test P-value = 0.629) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no difference between undergraduates and graduates for text score in the experimental group. --Is there a difference between undergraduates and graduates for map score in the experimental group? Null: there is normality for undergraduates and graduates for map score in the experimental group. Alternative: there is not normality for undergraduates and graduates for map score in the experimental group. Assumption checking: Tests of Normality Kolmogorov-Smirnova Group Statistic Undergraduates .396 Map Score Graduates .330 a. Lilliefors Significance Correction df 16 12 Sig. Shapiro-Wilk df 16 12 Sig. .000 .002

Statistic .636 .000 .730 .001

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group. 78

Most points are close to the line (except one), slight curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for graduates.

Most points are close to the line (except one), slight curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for graduates.

For undergraduates there are 2 outliers, which is not conducive for normality. For graduates there is 1 outlier. This is acceptable for normality. Thus normality is satisfied for graduates. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test for independent samples.

79

Test Statisticsa Map Score 90.000 168.000 -.325 .745

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed .802b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -0.325 (Mann-Whitney test P-value = 0.745) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no difference between undergraduates and graduates in the experimental group for map score. --Is there a difference between the control and experimental groups for question 1? Null: there is normality for the control and experimental groups for question 1. Alternative: there is not normality for the control and experimental groups for question 1. Assumption checking: Tests of Normality Group Kolmogorov-Smirnova Shapiro-Wilk Statistic df Sig. Statistic df Sig. Question 1 Experimental .382 29 .000 .628 29 .000 Control .535 26 .000 .301 26 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

80

Most points are close to the line, no significant curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the experimental group.

Half the points are not close to the line, no significant curvature, and not evenly distributed along the line, thus assumption of normality is not satisfied for the control group.

For experimental group there are no outliers, which is conducive for normality. For control group there are 2 outliers. Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test for independent samples.

81

Test Statisticsa Question 1 Mann-Whitney U 250.000 Wilcoxon W 685.000 Z -2.837 Asymp. Sig. (2.005 tailed) a. Grouping Variable: Group Mann-Whitney test statistic = -2.837 (Mann-Whitney test P-value = 0.005) < ( = 0.1) thus reject the null hypothesis. Conclusion: There is a significant difference between the experimental and control groups for question 1. --Is there a difference between the control and experimental groups for question 2? Null: there is normality for the control and experimental groups for question 2. Alternative: there is not normality for the control and experimental groups for question 2. Assumption checking: Tests of Normality Group Kolmogorov-Smirnova Shapiro-Wilk Statistic df Sig. Statistic df Sig. Question 2 Experimental .501 29 .000 .460 29 .000 Control .539 26 .000 .198 26 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

82

Half the points are not close to the line, no significant curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for the experimental group.

Half the points are not close to the line, no significant curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for the control group.

There are 4 outliers for the experimental group and 1 outlier for the control group. Thus, the assumption of normality is not satisfied. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test for independent samples.

83

Test Statisticsa Question 2 Mann-Whitney U 326.500 Wilcoxon W 761.500 Z -1.576 Asymp. Sig. (2.115 tailed) a. Grouping Variable: Group Mann-Whitney test statistic = -1.576 (Mann-Whitney test P-value = 0.115) > (a = 0.10) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between experimental and control groups for question 2. --Is there a difference between the control and experimental groups for question 3? Null: there is normality for the control and experimental groups for question 3. Alternative: there is not normality for the control and experimental groups for question 3. Assumption checking: Tests of Normality Group Kolmogorov-Smirnova Shapiro-Wilk Statistic df Sig. Statistic df Sig. Question 3 Experimental .435 29 .000 .584 29 .000 Control .508 26 .000 .436 26 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

84

The points are close to the line, no significant curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the experimental group.

Half the points are not close to the line, no significant curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for the control group.

For experimental group there are no outliers, which is conducive for normality. For control group there are 4 outliers, which is not conducive for normality. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test for independent samples.

85

Test Statisticsa Question 3 Mann-Whitney U 318.000 Wilcoxon W 753.000 Z -1.351 Asymp. Sig. (2.177 tailed) a. Grouping Variable: Group Mann-Whitney test statistic = -1.351 (Mann-Whitney test P-value = 0.117) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between experimental and control groups for question 3. --Is there a difference between the control and experimental groups for question 4? Null: there is normality for the control and experimental groups for question 4. Alternative: there is not normality for the control and experimental groups for question 4. Assumption checking: Tests of Normality Group Kolmogorov-Smirnova Shapiro-Wilk Statistic df Sig. Statistic df Sig. Question 4 Experimenta .435 29 .000 .584 29 .000 l Control .535 26 .000 .301 26 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

86

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for experimental group.

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for control group.

For experimental group there are no outliers, which is conducive for normality. For control group there are 2 outliers. This is not conducive for normality. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test for independent samples.

87

Test Statisticsa Question 4 Mann-Whitney U 289.000 Wilcoxon W 724.000 Z -2.141 Asymp. Sig. (2.032 tailed) a. Grouping Variable: Group Mann-Whitney test statistic = -2.141 (Mann-Whitney test P-value = 0.032) < ( = 0.1) thus reject the null hypothesis. Conclusion: There is a significant difference between experimental and control groups for question 4. --Is there a difference between the control and experimental groups for question 5? Null: there is normality for the control and experimental groups for question 5. Alternative: there is not normality for the control and experimental groups for question 5. Assumption checking: Tests of Normality Group Kolmogorov-Smirnova Shapiro-Wilk Statistic df Sig. Statistic df Sig. Question 5 Experimental .501 29 .000 .460 29 .000 Control .539 26 .000 .198 26 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

88

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for the experimental group.

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for control group.

There are 4 outliers in the experimental group and 1 outlier for the control group. This is not conducive for normality. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test for independent samples.

89

Test Statisticsa Question 5 Mann-Whitney U 326.500 Wilcoxon W 761.500 Z -1.576 Asymp. Sig. (2.115 tailed) a. Grouping Variable: Group Mann-Whitney test statistic = -1.576 (Mann-Whitney test P-value = 0.115) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no significant difference between experimental and control groups for question 5. --Is there a difference between the control and experimental groups for question 6? Null: there is normality for the control and experimental groups for question 6. Alternative: there is not normality for the control and experimental groups for question 6. Assumption checking: Tests of Normality Group Kolmogorov-Smirnova Shapiro-Wilk Statistic df Sig. Statistic df Sig. Question 6 Experimental .382 29 .000 .628 29 .000 Control .492 26 .000 .484 26 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

90

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the experimental group.

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the control group.

For experimental group there are no outliers, which is conducive for normality. For control group there are 4 outliers. Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test for independent samples.

91

Test Statisticsa Question 6 Mann-Whitney U 293.500 Wilcoxon W 728.500 Z -1.758 Asymp. Sig. (2.079 tailed) a. Grouping Variable: Group Mann-Whitney test statistic = -1.758 (Mann-Whitney test P-value = 0.079) > ( = 0.10) thus reject the null hypothesis. Conclusion: There is a significant difference between experimental and control groups for question 6. --Is there a difference between the control and experimental groups for question 7? Null: there is normality for the control and experimental groups for question 7. Alternative: there is not normality for the control and experimental groups for question 7. Assumption checking: Tests of Normality Group Kolmogorov-Smirnova Shapiro-Wilk Statistic df Sig. Statistic df Sig. Question 7 Experimental .382 29 .000 .628 29 .000 Control .474 26 .000 .524 26 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

92

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the experimental group.

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for control group.

For the experimental group there are no outliers, which is conducive for normality. For the control group there are 4 outliers. Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

93

Test Statisticsa Question 7 Mann-Whitney U 308.000 Wilcoxon W 743.000 Z -1.431 Asymp. Sig. (2.152 tailed) a. Grouping Variable: Group Mann-Whitney test statistic = -1.431 (Mann-Whitney test P-value = 0.152) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no significant difference between experimental and control groups for question 7. --Is there a difference between the control and experimental groups for question 8? Null: there is normality for the control and experimental groups for question 8. Alternative: there is not normality for the control and experimental groups for question 8. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Question 8 Experimental .346 29 .638 29 .000 .000 Control .492 26 .484 26 .000 .000 b. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

94

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for experimental group.

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for control group.

For experimental group there are no outliers, which is conducive for normality. For control group there are 4 outliers Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

95

Test Statisticsa Question 8 Mann-Whitney U 254.500 Wilcoxon W 689.500 Z -2.478 Asymp. Sig. (2.013 tailed) b. Grouping Variable: Group Mann-Whitney test statistic = -2.478 (Mann-Whitney test P-value = 0.013) < ( = 0.1) thus reject the null hypothesis. Conclusion: There is a significant difference between the experimental and control groups for question 8. --Is there a difference between the control and experimental groups for question 9? Null: there is normality for the control and experimental groups for question 9. Alternative: there is not normality for the control and experimental groups for question 9. Assumption checking: Tests of Normalityb Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Question 9 Experimental .382 29 .628 29 .000 .000 a. Lilliefors Significance Correction b. Question 9 is constant when Group = Control. It has been omitted. As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, experimental group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for experimental group.

96

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for experimental group.

For control group, normality test could not be done as all the values were same so t-test for independent samples cant be used. Need to use non-parametric Mann-Whitney test for independent samples. Test Statisticsa Question 9 Mann-Whitney U 221.000 Wilcoxon W 656.000 Z -3.676 Asymp. Sig. (2.000 tailed) b. Grouping Variable: Group Mann-Whitney test statistic = -3.676 (Mann-Whitney test P-value = 0.000) < ( = 0.1) thus reject the null hypothesis. Conclusion: There is a significant difference between the experimental and control groups for question 9. --Is there a difference between the control and experimental groups for question 10? Null: there is normality for the control and experimental groups for question 10. Alternative: there is not normality for the control and experimental groups for question 10.

97

Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Question 10 Experimental .364 29 .635 29 .000 .000 Control .508 26 .436 26 .000 .000 b. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the experimental group.

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the control group.

98

For the experimental group there are no outliers, which is conducive for normality. For the control group there are 4 outliers. Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question 10 Mann-Whitney U 266.000 Wilcoxon W 701.000 Z -2.337 Asymp. Sig. (2.019 tailed) b. Grouping Variable: Group Mann-Whitney test statistic = -2.337 (Mann-Whitney test P-value = 0.019) < ( = 0.1) thus reject the null hypothesis. Conclusion: There is a significant difference between the experimental and control groups for question 10. --Is there a difference between the control and experimental groups for question 11a? Null: there is normality for the control and experimental groups for question 11a. Alternative: there is not normality for the control and experimental groups for question 11a.

99

Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Question 11a Experimental .435 29 .584 29 .000 .000 Control .539 26 .198 26 .000 .000 b. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the experimental group.

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for control group.

100

For experimental group there are no outliers, which is conducive for normality. For control group there is 1 outlier. This is acceptable for normality.

Test Statisticsa Question 11a Mann-Whitney U 274.500 Wilcoxon W 709.500 Z -2.586 Asymp. Sig. (2.010 tailed) b. Grouping Variable: Group Mann-Whitney test statistic = -2.586 (Mann-Whitney test P-value = 0.01) < ( = 0.1) thus reject the null hypothesis. Conclusion: There is a significant difference between experimental and control groups for question 11a. --Is there a difference between the control and experimental groups for question 11b? Null: there is normality for the control and experimental groups for question 11b. Alternative: there is not normality for the control and experimental groups for question 11b.

101

Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Question Experimenta .501 29 .460 29 .000 .000 11b l Control .535 26 .301 26 .000 .000 b. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

The points are not close to the line, no curvature, and evenly distributed along the line, thus assumption of normality is not satisfied for the experimental group.

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus assumption of normality is not satisfied for the control group.

102

For experimental group there are 3 outliers, which is not conducive for normality. For control group there are 2 outliers, which is not conducive for normality. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question 11b Mann-Whitney U 341.000 Wilcoxon W 776.000 Z -1.051 Asymp. Sig. (2.293 tailed) b. Grouping Variable: Group Mann-Whitney test statistic = -1.051 (Mann-Whitney test P-value = 0.293) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no difference between experimental and control groups for question 11b. --Is there a difference between the control and experimental groups for question 11c? Null: there is normality for the control and experimental groups for question 11c. Alternative: there is not normality for the control and experimental groups for question 11c.

103

Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Question Experimenta .527 29 .354 29 .000 .000 11c l Control .535 26 .301 26 .000 .000 b. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

The points are not close to the line, no curvature, and evenly distributed along the line, thus assumption of normality is not satisfied for experimental group.

The points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the experimental group.

104

For the experimental group there are 2 outliers, which is not conducive for normality. For control group there are 2 outliers. Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question 11c Mann-Whitney U 367.000 Wilcoxon W 802.000 Z -.339 Asymp. Sig. (2.735 tailed) b. Grouping Variable: Group Mann-Whitney test statistic = -0.339 (Mann-Whitney test P-value = 0.735) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no difference between the experimental and control groups for question 11c. --Is there a difference between the control and experimental groups for question 11d? Null: there is normality for the control and experimental groups for question 11d. Alternative: there is not normality for the control and experimental groups for question 11d.

105

Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Experimental .501 29 .460 29 .000 .000 Question 11d Control .535 26 .301 26 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for experimental group.

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for the control group.

106

For experimental group there are 3 outliers, which is not conducive for normality. For control group there are 2 outliers. Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question 11d Mann-Whitney U 341.000 Wilcoxon W 776.000 Z -1.051 Asymp. Sig. (2.293 tailed) a. Grouping Variable: Group Mann-Whitney test statistic = -1.051 (Mann-Whitney test P-value = 0.293) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no difference between experimental and control groups for question 11d. --Is there a difference between the control and experimental groups for question 11e? Null: there is normality for the control and experimental groups for question 11e. Alternative: there is not normality for the control and experimental groups for question 11e.

107

Assumption checking:

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the experimental group.

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus assumption of normality is not satisfied for the control group.

108

For experimental group there are 3 outliers, which is not conducive for normality. For control group there are 2 outliers. Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question 11e Mann-Whitney U 341.000 Wilcoxon W 776.000 Z -1.051 Asymp. Sig. (2.293 tailed) a. Grouping Variable: Group Mann-Whitney test statistic = -1.051 (Mann-Whitney test P-value = 0.293) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no difference between experimental and control groups for question 11e. --Is there a difference between the control and experimental groups for question 11f? Null: there is normality for the control and experimental groups for question 11f. Alternative: there is not normality for the control and experimental groups for question 11f.

109

Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic Df Sig. .515 29 .412 29 .000 .000 Question11 Experimental f Control .539 26 .198 26 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus assumption of normality is not satisfied for the control group.

110

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for the control group.

For experimental group there are 4 outliers, which is not conducive for normality. For control group there is 1 outlier. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question11 f Mann-Whitney U 339.500 Wilcoxon W 774.500 Z -1.269 Asymp. Sig. (2.204 tailed) a. Grouping Variable: Group Mann-Whitney test statistic = -1.269 (Mann-Whitney test P-value = 0.204) > ( = 0.1) thus fail to reject the null hypothesis.

111

Conclusion: There is no difference between experimental and control groups for question 11f. --Is there a difference between undergraduates and graduates in the control group for question 1? Null: there is normality for undergraduates and graduates in the control group for question 1. Alternative: there is not normality for undergraduates and graduates in the control group for question 1.

Assumption checking: Tests of Normalitya Kolmogorov-Smirnovb Statistic df Sig.

Group

Shapiro-Wilk Statistic df Sig. .000

Question 1 Graduates .510 14 .000 .428 14 a. Question 1 is constant when Group = Undergraduates. It has been omitted. b. Lilliefors Significance Correction

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, graduates P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

112

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus assumption of normality is not satisfied for graduates.

For undergraduates, normality test could not be done as all the values were same so t-test for independent samples cant be used. Need to use non-parametric Mann-Whitney test for indpendent samples. Test Statisticsa Question 1 72.000 177.000 -1.336 .181

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed .560b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -1.336 (Mann-Whitney test P-value = 0.181) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no significant difference between undergraduates and graduates in the control group for question 1. --Is there a difference between undergraduates and graduates in the control group for question 2?

113

Null: there is normality for undergraduates and graduates in the control group for question 2. Alternative: there is not normality for undergraduates and graduates in the control group for question 2. Assumption checking: Tests of Normalitya Kolmogorov-Smirnovb Shapiro-Wilk Statistic df Sig. Statistic df Sig. .534 14 .000 .297 14 .000

Group Question 2 Graduates

a. Question 2 is constant when Group = Undergraduates. It has been omitted. b. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, graduates P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

For undergraduates, normality test could not be done as all the values were same so t-test for independent samples cant be used. Need to use non-parametric Mann-Whitney test for indpendent samples. Test Statisticsa Question 2 78.000 183.000 114

Mann-Whitney U Wilcoxon W

Z -.926 Asymp. Sig. (2-tailed) .355 Exact Sig. [2*(1-tailed .781b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -.926 (Mann-Whitney test P-value = 0.355) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no significant difference between undergraduates and graduates in the control group for question 2. --Is there a difference between undergraduates and graduates in the control group for question 3? Null: there is normality for undergraduates and graduates in the control group for question 3. Alternative: there is not normality for undergraduates and graduates in the control group for question 3. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Undergraduates .499 12 .465 12 .000 .000 Question 3 Graduates .510 14 .428 14 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

115

Half the points are not close to the line and not evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

Half the points are not close to the line and not evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

For undergraduates there are 2 outliers, which is not conducive for normality. For control group there are 2 outliers. Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa 116

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed .940b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -.164 (Mann-Whitney test P-value = 0.869) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the control group for question 3. --Is there a difference between undergraduates and graduates in the control group for question 4? Null: there is normality for undergraduates and graduates in the control group for question 4. Alternative: there is not normality for undergraduates and graduates in the control group for question 4. Assumption checking: Tests of Normalitya Kolmogorov-Smirnovb Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Question 4 Graduates .510 14 .428 14 .000 .000 a. Question 4 is constant when Group = Undergraduates. It has been omitted. b. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, graduates P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Question 3 82.000 160.000 -.164 .869

117

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

For undergraduates, normality test could not be done as all the values were same so t-test for independent samples cant be used. Need to use non-parametric Mann-Whitney test for indpendent samples. Test Statisticsa Question 4 72.000 177.000 -1.336 .181

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed .560b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -1.336 (Mann-Whitney test P-value = 0.181) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no significant difference between undergraduates and graduates in the control group for question 4. --Is there a difference between undergraduates and graduates in the control group for question 5?

118

Null: there is normality for undergraduates and graduates in the control group for question 5. Alternative: there is not normality for undergraduates and graduates in the control group for question 5. Assumption checking: Tests of Normalitya Kolmogorov-Smirnovb Statistic df Sig.

Group

Shapiro-Wilk Statistic df Sig. .000

Question 5 Graduates .534 14 .000 .297 14 a. Question 5 is constant when Group = Undergraduates. It has been omitted. b. Lilliefors Significance Correction

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, graduates P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

For undergraduates, normality test could not be done as all the values were same so t-test for independent samples cant be used. Need to use non-parametric Mann-Whitney test for indpendent samples. Test Statisticsa Question 5 78.000 119

Mann-Whitney U

Wilcoxon W 183.000 Z -.926 Asymp. Sig. (2-tailed) .355 Exact Sig. [2*(1-tailed .781b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -0.926 (Mann-Whitney test P-value = 0.355) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no significant difference between undergraduates and graduates in the control group for question 5. --Is there a difference between undergraduates and graduates in the control group for question 6? Null: there is normality for undergraduates and graduates in the control group for question 6. Alternative: there is not normality for undergraduates and graduates in the control group for question 6. Assumption checking: Tests of Normality Kolmogorov-Smirnova Statistic df Sig. 12 14 .000 .000

Group

Shapiro-Wilk Statistic df Sig. .552 .428 12 14 .000 .000

Undergraduate .460 Question 6 s Graduates .510 a. Lilliefors Significance Correction

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

120

Half the points are not close to the line and not evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

Half the points are not close to the line and not evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

For undergraduates there are no outliers, which is conducive for normality. For graduates there are 2 outliers. Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

121

Test Statisticsa Question 6 75.000 153.000 -.678 .498

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed .667b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -.678 (Mann-Whitney test P-value = 0.498) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the control group for question 6. --Is there a difference between undergraduates and graduates in the control group for question 7? Null: there is normality for undergraduates and graduates in the control group for question 7. Alternative: there is not normality for undergraduates and graduates in the control group for question 7. Assumption checking: Tests of Normality Kolmogorov-Smirnova Statistic df Sig. 12 14 .000 .000 Shapiro-Wilk Statistic df Sig. .552 .516 12 14 .000 .000

Group

Undergraduate .460 Question 7 s Graduates .478 a. Lilliefors Significance Correction

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group. 122

Half the points are not close to the line and not evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

Half the points are not close to the line and not evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

For undergraduates there are no outliers, which is conducive for normality. For graduates there are 3 outliers. Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

123

Test Statisticsa Question 7 81.000 159.000 -.211 .833

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed .899b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -0.211 (Mann-Whitney test P-value = 0.899) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the control group for question 7. --Is there a difference between undergraduates and graduates in the control group for question 8? Null: there is normality for undergraduates and graduates in the control group for question 8. Alternative: there is not normality for undergraduates and graduates in the control group for question 8. Assumption checking: Tests of Normality Kolmogorov-Smirnova Group Statistic Undergraduates .460 Question 8 Graduates .510 a. Lilliefors Significance Correction df 12 14 Sig. Shapiro-Wilk df 12 14 Sig. .000 .000

Statistic .552 .000 .428 .000

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group. 124

Half the points are not close to the line and not evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

Half the points are not close to the line and not evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

125

For undergraduates there are no outliers, which is conducive for normality. For graduates there are 2 outliers. Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question 8 75.000 153.000 -.678 .498

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed .667b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -0.678 (Mann-Whitney test P-value = 0.498) > (a = 0.10) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the control group for question 8. --Is there a difference between undergraduates and graduates in the control group for question 9? Null: there is normality for undergraduates and graduates in the control group for question 9.

126

Alternative: there is not normality for undergraduates and graduates in the control group for question 9. Assumption checking: Normality cannot be calculated because all values for undergraduates and graduates are the same. Need to use non-parametric Mann-Whitney test for indpendent samples. Test Statisticsa Question 9 84.000 189.000 .000 1.000

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed 1.000b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = 0.000 (Mann-Whitney test P-value = 1.000) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no significant difference between undergraduates and graduates in the control group for question 9. --Is there a difference between undergraduates and graduates in the control group for question 10? Null: there is normality for undergraduates and graduates in the control group for question 10. Alternative: there is not normality for undergraduates and graduates in the control group for question 10. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Statistic df Sig. Statistic df Sig. .499 12 .465 12 .000 .000 127

Question

Group Undergraduates

10 Graduates .510 a. Lilliefors Significance Correction

14

.000

.428

14

.000

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

Half the points are not close to the line and not evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

Half the points are not close to the line and not evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

128

For undergraduates there are 2 outliers, which is not conducive for normality. For graduates there are 2 outliers. Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question 10 82.000 160.000 -.164 .869 .940b

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)]

a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -0.164 (Mann-Whitney test P-value = 0.869) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the control group for question 10. --Is there a difference between undergraduates and graduates in the control group for question 11a? Null: there is normality for undergraduates and graduates in the control group for question 11a. Alternative: there is not normality for undergraduates and graduates in the control group for question 11a. Assumption checking: 129

Group

Tests of Normalitya Kolmogorov-Smirnovb Statistic df Sig.

Shapiro-Wilk Statistic df Sig. .000

Question Graduates .534 14 .297 14 .000 11a a. Question 11a is constant when Group = Undergraduates. It has been omitted. b. Lilliefors Significance Correction

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for each group.

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

For undergraduates, normality test could not be done as all the values were same so t-test for independent samples cant be used. Need to use non-parametric Mann-Whitney test for indpendent samples.

130

For undergraduates there are no outliers, which is conducive for normality. For graduates there is 1 outlier, which is acceptable for normality.

Independent Samples Test Levenes t-test for Equality of Means Test for Equality of Variances F Sig. t df Sig. Mean Std. Error (2Difference Difference tailed) Question 11a Equal variances assumed Equal variances not assumed 4.000 .057 .923 1.000 24 13.00 0 .365 .336 .07143 .07143 .07738 .07143

As per Levenes test: Levenes Test statistic = 4.000 (Levenes P-value = 0.057) > (a = 0.10), thus assumption of equal variances is satisfied. t-test statistic = 0.923 (t-test P-value = 0.365) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no significant difference between undergraduates and graduates in the control group for question 11a. --Is there a difference between undergraduates and graduates in the control group for question 11b? 131

Null: there is normality for undergraduates and graduates in the control group for question 11b. Alternative: there is not normality for undergraduates and graduates in the control group for question 11b. Assumption checking: Tests of Normalitya Kolmogorov-Smirnovb Statistic df Sig. Shapiro-Wilk Statistic df Sig. .000

Group

Question Graduates .510 14 .428 14 .000 11b a. Question 11b is constant when Group = Undergraduates. It has been omitted. b. Lilliefors Significance Correction

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

For undergraduates, normality test could not be done as all the values were same so t-test for independent samples cant be used. Need to use non-parametric Mann-Whitney test for indpendent samples.

132

For undergraduates there are no outliers, which is conducive for normality. For graduates there are 2 outliers. Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question 11b 72.000 177.000 -1.336 .181 .560b

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)] a. Grouping Variable: Group b. Not corrected for ties.

Mann-Whitney test statistic = -1.336 (Mann-Whitney test P-value = 0.181) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the control group for question 11b. --Is there a difference between undergraduates and graduates in the control group for question 11c? Null: there is normality for undergraduates and graduates in the control group for question 11c.

133

Alternative: there is not normality for undergraduates and graduates in the control group for question 11c. Assumption checking: Tests of Normalitya Kolmogorov-Smirnovb Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Question Graduates .510 14 .428 14 .000 .000 11c a. Question 11c is constant when Group = Undergraduates. It has been omitted. b. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus assumption of normality is not satisfied for graduates.

For undergraduates, normality test could not be done as all the values were same so t-test for independent samples cant be used. Need to use non-parametric Mann-Whitney test for indpendent samples.

134

For undergraduates there are no outliers, which is conducive for normality. For graduates there are 2 outliers. Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question 11c 72.000 177.000 -1.336 .181 .560b

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)] a. Grouping Variable: Group b. Not corrected for ties.

Mann-Whitney test statistic = -1.336 (Mann-Whitney test P-value = 0.181) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the control group for question 11c. --Is there a difference between undergraduates and graduates in the control group for question 11d? Null: there is normality for undergraduates and graduates in the control group for question 11d.

135

Alternative: there is not normality for undergraduates and graduates in the control group for question 11d. Assumption checking: Tests of Normalitya Kolmogorov-Smirnovb Statistic df Sig. Shapiro-Wilk Statistic df Sig. .000

Group

Question Graduates .510 14 .428 14 .000 11d a. Question 11d is constant when Group = Undergraduates. It has been omitted. b. Lilliefors Significance Correction

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

For undergraduates, normality test could not be done as all the values were same so t-test for independent samples cant be used. Need to use non-parametric Mann-Whitney test for indpendent samples.

136

For undergraduates there are no outliers, which is conducive for normality. For graduates there are 2 outliers. Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question 11d 72.000 177.000 -1.336 .181 .560b

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)] a. Grouping Variable: Group b. Not corrected for ties.

Mann-Whitney test statistic = -1.336 (Mann-Whitney test P-value = 0.181) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the control group for question 11d. --Is there a difference between undergraduates and graduates in the control group for question 11e? Null: there is normality for undergraduates and graduates in the control group for question 11e.

137

Alternative: there is not normality for undergraduates and graduates in the control group for question 11e. Assumption checking: Tests of Normalitya Kolmogorov-Smirnovb Statistic df Sig. Shapiro-Wilk Statistic df Sig. .000

Group

Question Graduates .510 14 .428 14 .000 11e a. Question 11e is constant when Group = Undergraduates. It has been omitted. b. Lilliefors Significance Correction

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

For undergraduates, normality test could not be done as all the values were same so t-test for independent samples cant be used. Need to use non-parametric Mann-Whitney test for indpendent samples.

138

For undergraduates there are no outliers, which is conducive for normality. For graduates there are 2 outliers, which is not conducive for normality. Thus normality is not satisfied for the control group. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question 11e 72.000 177.000 -1.336 .181 .560b

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)] a. Grouping Variable: Group b. Not corrected for ties.

Mann-Whitney test statistic = -1.336 (Mann-Whitney test P-value = 0.181) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the control group for question 11e. --Is there a difference between undergraduates and graduates in the control group for question 11f? Null: there is normality for undergraduates and graduates in the control group for question 11f.

139

Alternative: there is not normality for undergraduates and graduates in the control group for question 11f. Assumption checking: Tests of Normalitya Kolmogorov-Smirnovb Statistic df Sig. Shapiro-Wilk Statistic df Sig. .000

Group

Question Graduates .534 14 .297 14 .000 11f a. Question 11f is constant when Group = Undergraduates. It has been omitted. b. Lilliefors Significance Correction

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

For undergraduates, normality test could not be done as all the values were same so t-test for independent samples cant be used. Need to use non-parametric Mann-Whitney test for indpendent samples.

140

For undergraduates there are no outliers. For graduates there is 1 outlier. This is conducive for normality.

Independent Samples Test Levenes t-test for Equality of Means Test for Equality of Variances F Sig. t df Sig. (2Mean Std. Error tailed) Difference Difference Question 11f Equal variances assumed Equal variances not assumed 4.000 .057 .923 1.000 24 13.000 .365 .336 .07143 .07143 .07738 .07143

As per Levenes test: Levenes Test statistic = 4.000 (Levenes P-value = 0.057) > (a = 0.10), thus assumption of equal variances is satisfied. t-test statistic = 0.923 (t-test P-value = 0.365) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no significant difference between undergraduates and graduates in the control group for question 11f. --141

Is there a difference between undergraduates and graduates in the experimental group for question 1? Null: there is normality for undergraduates and graduates in the experimental group for question 1. Alternative: there is not normality for undergraduates and graduates in the experimental group for question 1. Assumption checking: Tests of Normality Kolmogorov-Smirnova Group Statistic Undergraduates .334 Question 1 Graduates .460 a. Lilliefors Significance Correction df 16 12 Sig. Shapiro-Wilk df 16 12 Sig. .000 .000

Statistic .644 .000 .552 .000

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

The points are close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is satisfied for undergraduates.

142

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

For undergraduates and graduates there are no outliers, which is conducive for normality.

Test Statisticsa Question 1 72.000 208.000 -1.316 .188

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed .280b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -1.336 143

(Mann-Whitney test P-value = 0.181) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 1. --Is there a difference between undergraduates and graduates in the experimental group for question 2? Null: there is normality for undergraduates and graduates in the experimental group for question 2. Alternative: there is not normality for undergraduates and graduates in the experimental group for question 2. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Undergraduates .492 16 .484 16 .000 .000 Question 2 Graduates .530 12 .327 12 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

144

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus assumption of normality is not satisfied for graduates.

For undergraduates there are three outliers. This is not conducive for normality. For graduates there is one outlier, which is acceptable for normality. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question 2 86.000 222.000 -.765 .444

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed .664b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -0.765 145

(Mann-Whitney test P-value = 0.444) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 2. --Is there a difference between undergraduates and graduates in the experimental group for question 3? Null: there is normality for undergraduates and graduates in the experimental group for question 3. Alternative: there is not normality for undergraduates and graduates in the experimental group for question 3. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Undergraduates .431 16 .591 16 .000 .000 Question 3 Graduates .417 12 .608 12 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for undergraduates.

146

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for graduates.

For undergraduates and graduates there are no outliers, which is conducive for normality.

Test Statisticsa Question 3 94.000 172.000 -.115 .909

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed .945b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -0.115 147

(Mann-Whitney test P-value = 0.909) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 3. --Is there a difference between undergraduates and graduates in the experimental group for question 4? Null: there is normality for undergraduates and graduates in the experimental group for question 4. Alternative: there is not normality for undergraduates and graduates in the experimental group for question 4. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Undergraduates .462 16 .546 16 .000 .000 Question 4 Graduates .374 12 .640 12 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

148

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for graduates.

For undergraduates and graduates there are no outliers. This is conducive for normality.

Test Statisticsa Question 4 80.000 158.000 -.918 .359

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed .478b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -0.918 149

(Mann-Whitney test P-value = 0.359) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 4. --Is there a difference between undergraduates and graduates in the experimental group for question 5? Null: there is normality for undergraduates and graduates in the experimental group for question 5. Alternative: there is not normality for undergraduates and graduates in the experimental group for question 5. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Undergraduates .462 16 .546 16 .000 .000 Question 5 Graduates .530 12 .327 12 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

150

Half the points are not close to the line, no curvature, and not evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

For undergraduates there are no outliers. This is conducive for normality. For graduates there is one outlier, which is acceptable for normality.

Test Statisticsa Question 5 80.000 216.000 -1.119 .263

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed .478b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -1.119 151

(Mann-Whitney test P-value = 0.263) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 5. --Is there a difference between undergraduates and graduates in the experimental group for question 6? Null: there is normality for undergraduates and graduates in the experimental group for question 6. Alternative: there is not normality for undergraduates and graduates in the experimental group for question 6. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Statisti df Sig. Statistic df Sig. Group c Undergraduates .334 16 .644 16 .000 .000 Question 6 Graduates .417 12 .608 12 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for undergraduates.

152

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for undergraduates.

For undergraduates and graduates there are no outliers. This is conducive for normality.

Independent Samples Test Levenes Test t-test for Equality of Means for Equality of Variances F Sig. t df Sig. (2Mean Std. Error tailed) Difference Difference Equal variances assumed Question 6 Equal variances not assumed As per Levenes test: 153 1.857 .185 -.862 -.868 26 24.43 9 .397 .394 -.16667 -.16667 .19337 .19201

Levenes Test statistic =1.857 (Levenes P-value = 0.185) > (a = 0.10), thus assumption of equal variances is satisfied. t-test statistic = -0.862 (t-test P-value = 0.397) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 6. --Is there a difference between undergraduates and graduates in the experimental group for question 7? Null: there is normality for undergraduates and graduates in the experimental group for question 7. Alternative: there is not normality for undergraduates and graduates in the experimental group for question 7. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Undergraduates .334 16 .644 16 .000 .000 Question 7 Graduates .417 12 .608 12 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

154

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for undergraduates.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for graduates.

For undergraduates and graduates there are no outliers. This is conducive for normality.

155

Independent Samples Test Levenes t-test for Equality of Means Test for Equality of Variances F Sig. t df Sig. Mean Std. Error (2- Differenc Differenc tailed) e e Equal variances assumed Equal variances not assumed 1.857 .185 -.862 -.868 26 24.43 9 .397 .394 -.16667 -.16667 .19337 .19201

Question 7

As per Levenes test: Levenes Test statistic =1.857 (Levenes P-value = 0.185) > (a = 0.10), thus assumption of equal variances is satisfied. t-test statistic = -0.862 (t-test P-value = 0.397) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 7. --Is there a difference between undergraduates and graduates in the experimental group for question 8? Null: there is normality for undergraduates and graduates in the experimental group for question 8. Alternative: there is not normality for undergraduates and graduates in the experimental group for question 8. Assumption checking: Tests of Normality Kolmogorov-Smirnova Statistic df Sig. 156

Group

Shapiro-Wilk Statistic df Sig.

Undergraduates .334 Graduates .374 a. Lilliefors Significance Correction Question 8

16 12

.000 .000

.644 .640

16 12

.000 .000

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for undergraduates.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for graduates.

157

For undergraduates and graduates there are no outliers. This is conducive for normality.

Independent Samples Test Levenes t-test for Equality of Means Test for Equality of Variances F Sig. t df Sig. (2Mean Std. Error tailed) Difference Difference Equal variances assumed Question 8 Equal variances not assumed As per Levenes test: Levenes Test statistic =0.424 (Levenes P-value = 0.520) > (a = 0.10), thus assumption of equal variances is satisfied. t-test statistic = -0.423 (t-test P-value = 0.676) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 8. --Is there a difference between undergraduates and graduates in the experimental group for question 9? 158 .424 .520 .423 26 .676 .676 .08333 .08333 .19697 .19688

.423 23.887

Null: there is normality for undergraduates and graduates in the experimental group for question 9. Alternative: there is not normality for undergraduates and graduates in the experimental group for question 9. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Statistic df Sig. Statistic df Sig. .398 16 .621 16 .000 .000 .331 12 .650 12 .001 .000

Question 9

Group Undergraduates Graduates

a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for undergraduates.

159

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for graduates.

For undergraduates and graduates there are no outliers. This is conducive for normality.

Independent Samples Test Levenes t-test for Equality of Means Test for Equality of Variances F Equal variances assumed Question 9 Equal variances not assumed Sig. t df Sig. (2tailed) 26 .526 Mean Std. Error Difference Difference .12500 .19458

.743

.397

.642

.638

23.261

.530

.12500

.19584

160

As per Levenes test: Levenes Test statistic =0.743 (Levenes P-value = 0.397) > (a = 0.10), thus assumption of equal variances is satisfied. t-test statistic = -0.642 (t-test P-value = 0.526) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 9. --Is there a difference between undergraduates and graduates in the experimental group for question 10? Null: there is normality for undergraduates and graduates in the experimental group for question 10. Alternative: there is not normality for undergraduates and graduates in the experimental group for question 10. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Undergraduates .334 16 .644 16 .000 .000 Question 10 Graduates .374 12 .640 12 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

161

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for undergraduates.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for graduates.

For undergraduates and graduates there are no outliers. This is conducive for normality.

162

Independent Samples Test Levenes t-test for Equality of Means Test for Equality of Variances F Sig. t df Sig. (2Mean Std. Error tailed) Difference Difference Equal variances assumed Equal variances not assumed .424 .520 -.423 26 .676 -.08333 .19697

Question 10

-.423

23.88 7

.676

-.08333

.19688

As per Levenes test: Levenes Test statistic =0.424 (Levenes P-value = 0.520) > (a = 0.10), thus assumption of equal variances is satisfied. t-test statistic = -0.423 (t-test P-value = 0.676) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 10. --Is there a difference between undergraduates and graduates in the experimental group for question 11a? Null: there is normality for undergraduates and graduates in the experimental group for question 11a. Alternative: there is not normality for undergraduates and graduates in the experimental group for question 11a. Assumption checking: Tests of Normality 163

Kolmogorov-Smirnova Shapiro-Wilk Statistic df Sig. Statistic df Sig. Undergraduates .462 16 .546 16 .000 .000 Question 11a Graduates .374 12 .640 12 .000 .000 a. Lilliefors Significance Correction Group As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for undergraduates.

164

For undergraduates and graduates there are no outliers. This is conducive for normality.

Test Statisticsa Question 11a 80.000 158.000 -.918 .359 .478b

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)]

a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -0.918 (Mann-Whitney test P-value = 0.359) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 11a. --Is there a difference between undergraduates and graduates in the experimental group for question 11b? Null: there is normality for undergraduates and graduates in the experimental group for question 11b. Alternative: there is not normality for undergraduates and graduates in the experimental group for question 11b. 165

Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Undergraduates .518 16 .398 16 .000 .000 Question 11b Graduates .499 12 .465 12 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

166

For undergraduates and graduates there are both 2 outliers. This is not conducive for normality. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question 11b 92.000 170.000 -.306 .759 .873b

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)] a. Grouping Variable: Group b. Not corrected for ties.

Mann-Whitney test statistic = -0.306 (Mann-Whitney test P-value = 0.759) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 11b. --Is there a difference between undergraduates and graduates in the experimental group for question 11c? Null: there is normality for undergraduates and graduates in the experimental group for question 11c.

167

Alternative: there is not normality for undergraduates and graduates in the experimental group for question 11c. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Undergraduates .536 16 .273 16 .000 .000 Question 11c Graduates .530 12 .327 12 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

168

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

For undergraduates and graduates there are both 1 outlier. This is acceptable for normality.

Test Statisticsa Question 11c 94.000 172.000 -.208 .835 .945b

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)] a. Grouping Variable: Group b. Not corrected for ties.

Mann-Whitney test statistic = -0.208 169

(Mann-Whitney test P-value = 0.835) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 11c. --Is there a difference between undergraduates and graduates in the experimental group for question 11d? Null: there is normality for undergraduates and graduates in the experimental group for question 11d. Alternative: there is not normality for undergraduates and graduates in the experimental group for question 11d. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Undergraduates .492 16 .484 16 .000 .000 Question 11d Graduates .530 12 .327 12 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

170

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

For undergraduates there are 3 outliers. This is not conducive for normality. For graduates there is 1 outlier. This is acceptable for normality. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question 11d Mann-Whitney U 86.000 Wilcoxon W 222.000 Z -.765 Asymp. Sig. (2-tailed) .444 Exact Sig. [2*(1-tailed .664b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. 171

Mann-Whitney test statistic = -0.765 (Mann-Whitney test P-value = 0.444) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 11d. Is there a difference between undergraduates and graduates in the experimental group for question 11e? Null: there is normality for undergraduates and graduates in the experimental group for question 11e. Alternative: there is not normality for undergraduates and graduates in the experimental group for question 11e. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Undergraduates .492 16 .484 16 .000 .000 Question 11e Graduates .530 12 .327 12 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

172

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

For undergraduates there are 3 outliers. This is not conducive for normality. For graduates there is 1 outlier. This is acceptable for normality. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question 11e 86.000 222.000 -.765 .444 .664b

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)] a. Grouping Variable: Group b. Not corrected for ties.

173

Mann-Whitney test statistic = -0.765 (Mann-Whitney test P-value = 0.444) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 11e. Is there a difference between undergraduates and graduates in the experimental group for question 11f? Null: there is normality for undergraduates and graduates in the experimental group for question 11f. Alternative: there is not normality for undergraduates and graduates in the experimental group for question 11f. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Undergraduates .492 16 .000 .484 16 .000 Question 11f Graduates .530 12 .000 .327 12 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for undergraduates.

174

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for graduates.

For undergraduates there are 3 outliers. This is not conducive for normality. For graduates there is 1 outlier. This is acceptable for normality. Since normality is not satisfied, it is necessary to use a non-parametric test, Mann-Whitney test, for independent samples.

Test Statisticsa Question 11f 86.000 222.000 -.765 .444 .664b

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)] a. Grouping Variable: Group b. Not corrected for ties.

Mann-Whitney test statistic = -0.765 175

(Mann-Whitney test P-value = 0.444) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 11f. Post Assessment Is there a difference between the experimental and control groups for total score in the post assessment? Null: there is normality for the experimental and control groups for total score in the post assessment. Alternative: there is not normality for the experimental and control groups for total score in the post assessment. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic Df Sig. Statistic df Sig. Experimental .244 8 .896 8 .179 .267 Total Score Control .158 8 .963 8 .200* .835 *. This is a lower bound of the true significance. a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are greater than (a = 0.10) thus normality assumption is satisfied. Independent Samples Test Levenes Test t-test for Equality of Means for Equality of Variances F Sig. t df Sig. (2Mean Std. Error tailed) Difference Difference Total Score Equal variances assumed Equal variances not assumed .036 .853 -1.870 -1.870 14 13.831 .083 -28.75000 .083 -28.75000 15.37478 15.37478

As per Levenes test: Levenes Test statistic = 0.036 176

(Levenes P-value = 0.853) > (a = 0.10), thus assumption of equal variances is satisfied. T test statistic = -1.870 (T test P-value = 0.083) < ( = 0.1) thus reject the null hypothesis. Conclusion: There is a significant difference between the experimental and control groups for total score in the post assessment. --Is there a difference between the experimental and control groups for text score in the post assessment? Null: there is normality for the experimental and control groups for text score in the post assessment. Alternative: there is not normality for the experimental and control groups for text score in the post assessment. Assumption checking: Tests of Normality Kolmogorov-Smirnova Statistic df Sig. .150 .200* Shapiro-Wilk Statistic df Sig. .884 .841 8 8 .207 .078

Group Text Score

Experimental .250 8 Control .200 8 *. This is a lower bound of the true significance. a. Lilliefors Significance Correction

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are greater than (a = 0.10) thus normality assumption is satisfied. Independent Samples Test Levenes t-test for Equality of Means Test for Equality of Variances F Sig. t df Sig. (2Mean Std. Error tailed) Difference Difference Text Score Equal variances assumed .934 .350 -1.038 14 .317 -16.66750 16.05991

177

Equal variances not assumed As per Levenes test: Levenes Test statistic = 0.934

-1.038 13.292

.318 -16.66750

16.05991

(Levenes P-value = 0.350) > (a = 0.10), thus assumption of equal variances is satisfied. T test statistic = -1.038 (T test P-value = 0.317) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no difference between the experimental and control groups for text score in the post assessment. --Is there a difference between the experimental and control groups for map score in the post assessment? Null: there is normality for the experimental and control groups for map score in the post assessment. Alternative: there is not normality for the experimental and control groups for map score in the post assessment. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. * Experimental .201 8 .908 8 .200 .340 Map * Score Control .205 8 .939 8 .200 .597 *. This is a lower bound of the true significance. a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are greater than (a = 0.10) thus normality assumption is satisfied.

178

Independent Samples Test Levenes t-test for Equality of Means Test for Equality of Variances F Sig. T df Sig. (2Mean Std. Error tailed) Difference Difference Equal variances assumed Equal variances not assumed .307 .588 -1.605 14 .131 -24.21875 15.08844

Map Score

-1.605 13.826

.131 -24.21875

15.08844

As per Levenes test: Levenes Test statistic = 0.307 (Levenes P-value = 0.588) > (a = 0.10), thus assumption of equal variances is satisfied. T test statistic = -1.605 (T test P-value = 0.131) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no difference between the experimental and control groups for map score in the post assessment. --Is there a difference between the experimental and control groups for question 1 in the post assessment? Null: there is normality for the experimental and control groups for question 1 in the post assessment. Alternative: there is not normality for the experimental and control groups for question 1 in the post assessment.

179

Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Experimental .325 8 .665 8 .013 .001 Question 1 Control .513 8 .418 8 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for graduates.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the experimental group.

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the control group.

180

For the experimental group there are no outliers. This is conducive for normality. For the control group there is 1 outlier. This is acceptable for normality.

Test Statisticsa Question 1 20.000 56.000 -1.567 .117

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed .234b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -1.567 (Mann-Whitney test P-value = 0.117) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 1. --Is there a difference between the experimental and control groups for question 2 in the post assessment? Null: there is normality for the experimental and control groups for question 2 in the post assessment.

181

Alternative: there is not normality for the experimental and control groups for question 2 in the post assessment. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Experimental .455 8 .566 8 .000 .000 Question 2 Control .325 8 .665 8 .013 .001 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for both groups.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the experimental group.

182

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the control group.

For both groups there are no outliers. This is conducive for normality.

183

Independent Samples Test Levenes t-test for Equality of Means Test for Equality of Variances F Sig. t df Sig. (2Mean Std. Error tailed) Difference Difference Equal variances assumed Equal variances not assumed 2.333 .149 -1.000 14 .334 -.25000 .25000

Question 2

-1.000

13.720

.335

-.25000

.25000

As per Levenes test: Levenes Test statistic = 2.333 (Levenes P-value = 0.149) > (a = 0.10), thus assumption of equal variances is satisfied. T test statistic = -1.000 (T test P-value = 0.334) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no difference between the experimental and control groups for question 2 in the post assessment. --Is there a difference between the experimental and control groups for question 3 in the post assessment? Null: there is normality for the experimental and control groups for question 3 in the post assessment. Alternative: there is not normality for the experimental and control groups for question 3 in the post assessment.

184

Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Experimental .391 8 .641 8 .001 .000 Question 3 Control .391 8 .641 8 .001 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for both groups.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the experimental group.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the control group.

185

For both groups there are no outliers. This is conducive for normality.

Independent Samples Test Levenes Test t-test for Equality of Means for Equality of Variances F Sig. t df Sig. (2Mean Std. Error tailed) Difference Difference Equal variances assumed Question 3 Equal variances not assumed As per Levenes test: Levenes Test statistic = 0.000 (Levenes P-value = 1.000) > (a = 0.10), thus assumption of equal variances is satisfied. T test statistic = -0.966 (T test P-value = 0.350) > ( = 0.) thus fail to reject the null hypothesis. Conclusion: There is no difference between the experimental and control groups for question 3 in the post assessment. --.000 1.000 -.966 14 .350 -.25000 .25877

-.966 14.000

.350

-.25000

.25877

186

Is there a difference between the experimental and control groups for question 4 in the post assessment? Null: there is normality for the experimental and control groups for question 4 in the post assessment. Alternative: there is not normality for the experimental and control groups for question 4 in the post assessment. Assumption checking: Tests of Normality Kolmogorov-Smirnova Group Statistic Experimental .391 Question 4 Control .391 a. Lilliefors Significance Correction df 8 8 Sig. Shapiro-Wilk df 8 8 Sig. .000 .000

Statistic .641 .001 .641 .001

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for both groups.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the experimental group.

187

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the control group.

For both groups there are no outliers. This is conducive for normality.

Independent Samples Test Levenes Test t-test for Equality of Means for Equality of Variances F Sig. t df Sig. (2Mean Std. Error tailed) Difference Differenc e Equal variances assumed Question 4 Equal variances not assumed As per Levenes test: 188 .000 1.000 .000 .000 14 14.000 1.000 1.000 .00000 .00000 .25877 .25877

Levenes Test statistic = 0.000 (Levenes P-value = 1.000) > (a = 0.10), thus assumption of equal variances is satisfied. T test statistic = -0.000 (T test P-value = 1.000) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no difference between the experimental and control groups for question 4 in the post assessment. --Is there a difference between the experimental and control groups for question 5 in the post assessment? Null: there is normality for the experimental and control groups for question 5 in the post assessment. Alternative: there is not normality for the experimental and control groups for question 5 in the post assessment. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Experimental .455 8 .566 8 .000 .000 Question 5 Control .455 8 .566 8 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for both groups.

189

Half the points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the experimental group.

Half the points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the control group.

For both groups there are no outliers. This is conducive for normality.

190

Test Statisticsa Question 5 32.000 68.000 .000 1.000

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed 1.000b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = 0.000 (Mann-Whitney test P-value = 1.000) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is not a significant difference between undergraduates and graduates in the experimental group for question 5. --Is there a difference between the experimental and control groups for question 6 in the post assessment? Null: there is normality for the experimental and control groups for question 6 in the post assessment. Alternative: there is not normality for the experimental and control groups for question 6 in the post assessment. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Statisti df Sig. Statistic df Sig. Group c Experimental .391 8 .641 8 .001 .000 Question 6 Control .391 8 .641 8 .001 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for both groups. 191

Half the points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the experimental group.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the control group.

For both groups there are no outliers. This is conducive for normality.

192

Independent Samples Test Levenes Test t-test for Equality of Means for Equality of Variances F Sig. t df Sig. (2Mean Std. Error tailed) Differenc Differenc e e Equal variances assumed Equal variances not assumed .000 1.000 -.966 14 .350 -.25000 .25877

Question 6

-.966 14.000

.350

-.25000

.25877

As per Levenes test: Levenes Test statistic = 0.000 (Levenes P-value = 1.000) > (a = 0.10), thus assumption of equal variances is satisfied. T test statistic = -0.966 (T test P-value = 0.350) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no difference between the experimental and control groups for question 6 in the post assessment. --Is there a difference between the experimental and control groups for question 7 in the post assessment? Null: there is normality for the experimental and control groups for question 7 in the post assessment. Alternative: there is not normality for the experimental and control groups for question 7 in the post assessment.

193

Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Experimental .455 8 .566 8 .000 .000 Question 7 Control .513 8 .418 8 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for both groups.

Half the points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the experimental group.

Half the points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the control group.

194

For the experimental group there are no outliers. This is conducive for normality. For the control group there is 1 outlier. This is acceptable for normality.

Test Statisticsa Question 7 12.000 48.000 -2.440 .015

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed .038b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = 2.440 (Mann-Whitney test P-value = 0.015) < ( = 0.1) thus reject the null hypothesis. Conclusion: There is a significant difference between the control and experimental groups for question 7 in the post assessment. --Is there a difference between the experimental and control groups for question 8 in the post assessment? Null: there is normality for the experimental and control groups for question 8 in the post assessment.

195

Alternative: there is not normality for the experimental and control groups for question 8 in the post assessment. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Experimental .455 8 .566 8 .000 .000 Question 8 Control .455 8 .566 8 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for both groups.

Half the points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the experimental group.

196

Half the points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the control group.

For both groups there are no outliers. This is conducive for normality.

Test Statisticsa Question 8 16.000 52.000 -1.936 .053

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed .105b Sig.)] a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -1.936 197

(Mann-Whitney test P-value = 0.053) < ( = 0.1) thus reject the null hypothesis. Conclusion: There is a significant difference between the control and experimental groups for question 8 in the post assessment. --Is there a difference between the experimental and control groups for question 9 in the post assessment? Null: there is normality for the experimental and control groups for question 9 in the post assessment. Alternative: there is not normality for the experimental and control groups for question 9 in the post assessment. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Experimental .455 8 .000 .566 8 .000 Question 9 Control .455 8 .000 .566 8 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for both groups.

Half the points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the experimental group.

198

Half the points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the control group.

For both groups there are no outliers. This is conducive for normality.

Test Statisticsa Question 9 32.000 68.000 .000 1.000

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed 1.000b Sig.)] a. Grouping Variable: Group b. Not corrected for ties.

199

Mann-Whitney test statistic = 0.000 (Mann-Whitney test P-value = 1.000) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no significant difference between the control and experimental groups for question 9 in the post assessment. --Is there a difference between the experimental and control groups for question 10 in the post assessment? Null: there is normality for the experimental and control groups for question 10 in the post assessment. Alternative: there is not normality for the experimental and control groups for question 10 in the post assessment. Assumption checking: Tests of Normalitya Kolmogorov-Smirnovb Shapiro-Wilk Statistic df Sig. Statistic df Sig. .391 8 .001 .641 8 .000

Group Question 10 Control

a. Question 10 is constant when Group = Experimental. It has been omitted. b. Lilliefors Significance Correction Normality cannot be calculated for the experimental group because all values are the same. Need to use non-parametric Mann-Whitney test for independent samples. As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the control groups P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for the control group.

200

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the control group.

Normality cannot be calculated for the experimental group because all the values are the same.

For both groups there are no outliers. This is conducive for normality.

Test Statisticsa Question 10 12.000 48.000 -2.611 .009 .038b

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)]

201

a. Grouping Variable: Group b. Not corrected for ties. Mann-Whitney test statistic = -2.611 (Mann-Whitney test P-value = 0.009) < ( = 0.1) thus reject the null hypothesis. Conclusion: There is a significant difference between the experimental and control groups for question 10 in the post assessment. --Is there a difference between the experimental and control groups for question 11a in the post assessment? Null: there is normality for the experimental and control groups for question 11a in the post assessment. Alternative: there is not normality for the experimental and control groups for question 11a in the post assessment. Assumption checking: Tests of Normality Kolmogorov-Smirnova Group Statistic Experimental .513 Question 11a Control .325 a. Lilliefors Significance Correction df 8 8 Sig. Shapiro-Wilk df 8 8 Sig. .000 .001

Statistic .418 .000 .665 .013

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for both groups.

202

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the experimental group.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the control group.

For the experimental group there is 1 outlier. This is acceptable for normality. For the control group there are no outliers. This is conducive for normality.

203

Test Statisticsa Question 11a 20.000 56.000 -1.567 .117 .234b

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)] a. Grouping Variable: Group b. Not corrected for ties.

Mann-Whitney test statistic = -1.567 (Mann-Whitney test P-value = 0.117) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no significant difference between the experimental and control groups for question 11a in the post assessment. --Is there a difference between the experimental and control groups for question 11b in the post assessment? Null: there is normality for the experimental and control groups for question 11b in the post assessment. Alternative: there is not normality for the experimental and control groups for question 11b in the post assessment. Assumption checking: Tests of Normality Kolmogorov-Smirnova Statistic df Sig. 8 8 .000 .001 Shapiro-Wilk Statistic df Sig. .566 .641 8 8 .000 .000

Group Question 11b

Experimenta .455 l Control .391 a. Lilliefors Significance Correction

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for both groups. 204

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the experimental group.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the control group.

There are no outliers for either group. This is conducive for normality.

205

Independent Samples Test Levenes Test t-test for Equality of Means for Equality of Variances F Sig. t df Sig. (2Mean Std. Error tailed) Differenc Difference e Question 11b Equal variances assumed Equal variances not assumed 1.000 .334 -.509 14 .619 .619 -.12500 -.12500 .24550 .24550

-.509 13.829

As per Levenes test: Levenes Test statistic = 1.000 (Levenes P-value = 0.334) > (a = 0.10), thus assumption of equal variances is satisfied. T test statistic = -0.509 (T test P-value = 0.619) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no difference between the experimental and control group for question 11b in the post assessment. --Is there a difference between the experimental and control groups for question 11c in the post assessment? Null: there is normality for the experimental and control groups for question 11c in the post assessment. Alternative: there is not normality for the experimental and control groups for question 11c in the post assessment.

206

Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Experimental .455 8 .566 8 .000 .000 Question 11c Control .513 8 .418 8 .000 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for both groups.

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the experimental group.

Half the points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the control group.

207

There are 0 outliers for the experimental group and 1 outlier for the control group. This is acceptable for normality.

Test Statisticsa Question 11c 28.000 64.000 -.620 .535 .721b

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)] a. Grouping Variable: Group b. Not corrected for ties.

Mann-Whitney test statistic = -0.620 (Mann-Whitney test P-value = 0.535) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no significant difference between the experimental and control groups for question 11c in the post assessment. --Is there a difference between the experimental and control groups for question 11d in the post assessment? Null: there is normality for the experimental and control groups for question 11d in the post assessment.

208

Alternative: there is not normality for the experimental and control groups for question 11d in the post assessment. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Experimental .455 8 .566 8 .000 .000 Question 11d Control .391 8 .641 8 .001 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for both groups.

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the experimental group.

209

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the control group.

There are no outliers for either group. This is conducive for normality.

Test Statisticsa Question 11d 20.000 56.000 -1.464 .143 .234b

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)] a. Grouping Variable: Group b. Not corrected for ties.

Mann-Whitney test statistic = -1.464 210

(Mann-Whitney test P-value = 0.143) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no significant difference between the experimental and control groups for question 11d in the post assessment. --Is there a difference between the experimental and control groups for question 11e in the post assessment? Null: there is normality for the experimental and control groups for question 11e in the post assessment. Alternative: there is not normality for the experimental and control groups for question 11e in the post assessment. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Group Statistic df Sig. Statistic df Sig. Experimental .391 8 .001 .641 8 .000 Question 11e Control .391 8 .001 .641 8 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for both groups.

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the experimental group.

211

The points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the control group.

There are no outliers for either group. This is conducive for normality.

Independent Samples Test Levenes Test t-test for Equality of Means for Equality of Variances F Sig. t df Sig. Mean Std. Error (2- Difference Difference tailed) Question 11e Equal variances assumed .000 1.000 .000 14 1.000 .00000 .25877

212

Equal variances not assumed As per Levenes test: Levenes Test statistic = 0.000

.000 14.000

1.000

.00000

.25877

(Levenes P-value = 1.000) > (a = 0.10), thus assumption of equal variances is satisfied. T test statistic = -0.000 (T test P-value = 1.000) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no difference between the experimental and control group for question 11e in the post assessment. --Is there a difference between the experimental and control groups for question 11f in the post assessment? Null: there is normality for the experimental and control groups for question 11f in the post assessment. Alternative: there is not normality for the experimental and control groups for question 11f in the post assessment. Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Statisti df Sig. Statistic df Sig. c .455 8 .000 .566 8 .000 8 .000 .566 8 .000

Control .455 a. Lilliefors Significance Correction

Question 11f

Group Experimental

As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for both groups.

213

Half the points are not close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the experimental group.

Half the points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the control group.

There are no outliers for either group. This is conducive for normality.

214

Test Statisticsa Question 11f 32.000 68.000 .000 1.000 1.000b

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)] a. Grouping Variable: Group b. Not corrected for ties.

Mann-Whitney test statistic = 0.000 (Mann-Whitney test P-value = 1.000) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no significant difference between the experimental and control groups for question 11f in the post assessment. --Is there a difference between the experimental and control groups for interest level in the post assessment? Null: there is normality for the experimental and control groups for interest level in the post assessment. Alternative: there is not normality for the experimental and control groups for interest level in the post assessment.

215

Assumption checking: Tests of Normality Kolmogorov-Smirnova Shapiro-Wilk Statisti df Sig. Statistic df Sig. Group c Experimental .300 8 .000 .798 8 .027 Interest Level Control .513 8 .000 .418 8 .000 a. Lilliefors Significance Correction As per above table, according to Kolmogorv-Smirnov and Shapiro-Wilk test, both the group P-values are less than (a = 0.10) thus normality assumption is not satisfied. So need to check normal quantile plots for both groups.

Most of the points are close to the line, slight curvature, and evenly distributed along the line, thus the assumption of normality is satisfied for the experimental group.

Half the points are close to the line, no curvature, and evenly distributed along the line, thus the assumption of normality is not satisfied for the control group.

216

There are no outliers for the experimental group. This is conducive for normality. There is one outlier for the control group. This is acceptable for normality.

Test Statisticsa Interest Level 23.500 59.500 -1.046 .295 .382b

Mann-Whitney U Wilcoxon W Z Asymp. Sig. (2-tailed) Exact Sig. [2*(1-tailed Sig.)] a. Grouping Variable: Group b. Not corrected for ties.

Mann-Whitney test statistic = -1.046 (Mann-Whitney test P-value = 0.295) > ( = 0.1) thus fail to reject the null hypothesis. Conclusion: There is no significant difference between the experimental and control groups for interest level in the post assessment.

217

Annex H: Jericho Screenshots Below are several screenshots of the web-based geography game Jericho.

218

219

220