Running

head: EVALUATING THE EFFECTIVENESS OF A GRADUATE COURSE REDEVELOPMENT Evaluating the Effectiveness of a Graduate Course Redevelopment Kenneth Hirschmann Kim Prokosch University of Colorado at Denver

EVALUATING THE EFFECTIVENESS OF A GRADUATE COURSE REDEVELOPMENT Introduction and Problem Statement During the Fall 2011 semester, we redeveloped a graduate-level course on eLearning program management as part of our coursework for INTE 6750, Trends in ILT. The new format of the course is currently running, taught by an adjunct faculty member

and a TA. It is highly likely that we will be co-teaching the course in the Summer 2012 term, after our commencement. We will conduct research to measure the effectiveness of the redeveloped content, and will then make any necessary revisions to the course prior to its summer offering. The data we collect during this course will help us to analyze our course design in a thorough manner that will positively affect the future students of this course. By employing a pragmatic approach to our research, we will be able to understand the impact of one specific project in the larger context of the course: as Creswell (2009) states, Instead of focusing on methods, researchers emphasize the research problem and use all approaches available to understand the problem (p. 10). It is crucial that we employ techniques that will give us the most meaningful data possible, and a pragmatic approach will support and facilitate that process. The courses pedagogical approach is based primarily on M. David Merrills First Principles of Instruction with a heavy emphasis on the learner-centered approaches of Barbara McCombs, Donna Vakili and David Jonassen (Jonassen et al., 1995, McCombs & Vakili, 2005, Merrill, 2008). Using the data that we gather, we hope to evaluate the effectiveness of this approach and determine if we were successful in implementing the pedagogical design, or whether an alternative approach, such as Communities of Inquiry might better suit this course.

EVALUATING THE EFFECTIVENESS OF A GRADUATE COURSE REDEVELOPMENT As part of this pedagogical approach, one of the major projects that we introduced into the course is a case study project. The purpose of this project is to create a case study in project management through interviewing project management professionals, defining

organizational practices, and documenting past experiences. A comparison with a literature review and best practices will result in a section with suggested actions to improve the process. As this is a completely new project for the course, we feel it is important to answer the following question: How was motivation and engagement impacted by the inclusion of a personally meaningful case study project? Purpose and Intended Audience The purpose of this research is to evaluate the effectiveness of the learning environment to subsequently make improvements before the course runs again in the summer. The intended audience is divided into two distinct groups: our peers and instructor in INTE 6720, Research in Information and Learning Technologies, and the current instructor/TA and program director of the course. This research will be published in our portfolios as part of the graduation requirements for the ILT program. Research Questions To evaluate the effectiveness of the course, we will focus our research on the following questions. They will be broken down further in our data collection methods. 1. To what extent did the resources (readings and presentations) and activities (discussions and group assignments) in the course support the case study project? 2. To what extent did all course activities draw upon knowledge gained from other courses in the program and previous knowledge and/or experience of the subject matter?

EVALUATING THE EFFECTIVENESS OF A GRADUATE COURSE REDEVELOPMENT 3. Were the topics covered and activities personally meaningful and professionally relevant? 4. How did the requirements of the course compare to other courses in the program? In addition to these four questions we will be looking at the general satisfaction of specific resources, activities and the course in general as well as general open-ended feedback on the course design. Context of the Study The research is being conducted primarily to make improvements in the quality of

the learning environment in the course, with the ultimate goal being an improvement in the student experience. The beneficiaries of this research will be the students taking the course, and the program for which this course is a requirement. It is imperative that this course teaches the relevant competencies that are required within the program as there are some unique skills that will only be learned through this course. Due to the sensitive nature of performing research students and peers, we will address all political or privacy concerns with our research sponsor, our professor. Procedures Data Gathering Plans Table 1 lists the ways in which we will gather data in response to our questions. Table 1 Data Gathering Plans Question To what extent did the resources (readings and presentations) and activities (discussions and assignments) in the course support the case study project? To what extent did all course activities draw upon Data Collection Method(s) Student survey; instructor/TA survey; interviews; documentary data/course observations Student survey; instructor/TA

EVALUATING THE EFFECTIVENESS OF A GRADUATE COURSE REDEVELOPMENT knowledge gained from other courses in the program and previous knowledge and/or experience of the subject matter? Were the topics covered personally meaningful and professionally relevant? How did the requirements of the course compare to other courses in the program? General course and specific activity/resource feedback Table 1 survey; interviews; documentary data/course observations Student survey; instructor/TA survey; interviews; documentary data/course observations Student survey; instructor/TA survey; interviews; documentary data/course observations Student survey; instructor/TA survey; interviews

The combination of survey data and course observations will allow us to employ a

mixed-methods approach to this research project. Our surveys will yield us both quantitative and qualitative data, and our course observations will give us primarily qualitative data. Data Analysis Plans We will be collecting both quantitative and qualitative data during the course of our research. Our quantitative data will be analyzed and represented using various tools such as frequency counts, charts, and diagrams. We will give participants the opportunity to include additional qualitative data in comments fields, and we will analyze that information in an attempt to identify themes that may exist among participants responses. Some of our survey questions will be qualitative in nature, and the data we receive in response to interviews will be qualitative. We will analyze that data, again, with the intention of identifying trends that will give us actionable steps towards the future revision of the course. Schedule Table 2 shows our proposed schedule for the completion of this research project.

EVALUATING THE EFFECTIVENESS OF A GRADUATE COURSE REDEVELOPMENT Table 2 Project Schedule Timeline February 12 13, 2012 February 12 18, 2012 February 15 19, 2012 February 20 March 1, 2012 March 2 7, 2012 March 10, 2012 March 5 23, 2012 March 8 15, 2012 March 11 April 9, 2012 April 10, 2012 April 13 April 27, 2012 April 28, 2012 Table 2 Ethical Procedures We will ensure full adherence to ethical standards during our research. Implied informed consent forms (Appendix A) will be embedded in both the formative and summative evaluations. Participants will be asked to select either I agree or I do not Task Write formative evaluation questions; create survey Write and edit Action Research Proposal Receive responses to formative evaluation survey; start Literature Review Analyze data from survey responses Write summative evaluation questions; create survey Submit Literature Review Conduct individual interviews Receive responses to summative evaluation survey; edit Literature Review Analyze all data Submit Draft Data Analysis Write and edit Final Action Research Report Submit Final Action Research Report

agree after reviewing the informed consent information in each survey. If participants do not agree, they will not be able to complete the survey. Participation in our research will be voluntary, and participants may discontinue their involvement in our research at any point. Our formative and summative evaluations will be kept anonymous (unless participants voluntarily identify themselves). We will ask participants to identify themselves using a three-word passphrase: participants will enter this passphrase in the formative and summative evaluations. This will allow us to compare the responses of each individual from the first survey to their responses to the second survey. Participants will be advised of this approach, and entering the three-word passphrase is voluntary.

EVALUATING THE EFFECTIVENESS OF A GRADUATE COURSE REDEVELOPMENT We will conduct several individual interviews to gather more data, and those interviews will be recorded so we will be able to analyze that data. In the reporting of this

data, no personally identifying information will be used. The transcripts of these interviews will be made available, upon request, and the identity of the interviewees will still be kept confidential in that situation. Interviewees will be required to sign and submit a written consent form that will also outline the privacy policies of our research. Checks for Rigor To ensure the validity, credibility and dependability of our data, we will use the following methods in our data collection: A group of Critical Friends, made up of peers in our research class, as well as our professor, will help to ensure the validity of our procedures through careful review and feedback at each stage (Koshy, 2010). Triangulation and Diverse Case Analysis will help to ensure the credibility of our data by using a variety of sources (Stringer, 2007 and Creswell, 2009). Our sources of information include the surveying and interviewing of current course students, the instructor and TA of the course, and ourselves through non-participant observation and documentary data. Member Checking will be used during the interviewing process to allow participants an opportunity to confirm and clarify their statements (Stringer, 2007). An Inquiry Audit will be included in the final research report that details our survey, interview and documentary review procedures to help ensure dependability of our research (Stringer, 2007). Conclusion

EVALUATING THE EFFECTIVENESS OF A GRADUATE COURSE REDEVELOPMENT

As relatively new practitioners in the field of instructional design, the development of this course offered us an incredible challenge to test our new skills. The development process was not without its difficulties, but the overall product that resulted was something that we were proud of. Still, based on the fact that our level of experience in designing courses is at an early stage, we fully expect to find that we will need to make some changes. Fortunately, the students taking the course will have added insight as students of instructional design. We are confident that this research will provide a strong foundation to make some immediate and necessary changes to the design and implementation of the course. Through those refinements, we can be confident that its next offering will provide a more optimal learning environment for future students.

EVALUATING THE EFFECTIVENESS OF A GRADUATE COURSE REDEVELOPMENT References

Creswell, J. W. (2009). Research Design: Qualitative, quantitative, and mixed methods approaches (3rd ed.). Thousand Oaks, CA: Sage Publications, Inc. Jonassen, D., Davidson, M., Collins, M., Campbell, J., & Haag, B. (1995). Constructivism and computer-mediated communication in distance education. American Journal of Distance Education, 9(2), 7-26. Routledge. Retrieved from http://www.c3l.uni- oldenburg.de/cde/media/readings/jonassen95.pdf Koshy, V. (2010). Action research for Improving Educational Practice (2nd ed.). Thousand Oaks, CA: Sage Publications, Inc. McCombs, B.L. & Vakili, D. (2005). A Learner-Centered Framework for E-Learning. Teachers College Record Volume 107, Number 8, August 2005, pp. 15821600. Retrieved from http://edtech.boisestate.edu/elearn/advEDTECH597/McCombs(2005).pdf Merrill, M. D. (2008). Reflections on a four decade search for effective, efficient and engaging instruction. In M. W. Allen (Ed.), Michael Allen's 2008 e-Learning Annual (Vol. 1, pp. 141-167): Wiley Pfieffer Stringer, E. (2007). Action research (3rd ed.). Thousand Oaks, CA: Sage Publications, Inc.

EVALUATING THE EFFECTIVENESS OF A GRADUATE COURSE REDEVELOPMENT Appendix A: Implied Informed Consent Form

10

If you decide to participate in this study, you will be asked to submit two surveys. We anticipate that these surveys will take no longer than 45 minutes each to complete. We will also conduct individual interviews to obtain more in-depth data. Please email Kenny Hirschmann at kenneth.hirschmann@ucdenver.edu or Kim Prokosch at kim.prokosch@ucdenver.edu if you would like to be interviewed as part of this study. We will ask you to enter a three-word passphrase as you complete both of the surveys. The intention of this passphrase is for us to be able to compare your responses from the first survey to your responses in the second survey. Entering a three-word passphrase is optional. Answering and completing these online questionnaires indicate your willingness to participate in this study. Clicking next below indicates that you have read and understood the description of the study and you agree to participate. Thank you for your time.

EVALUATING THE EFFECTIVENESS OF A GRADUATE COURSE REDEVELOPMENT Appendix B: Formative Evaluation

11

EVALUATING THE EFFECTIVENESS OF A GRADUATE COURSE REDEVELOPMENT

12

EVALUATING THE EFFECTIVENESS OF A GRADUATE COURSE REDEVELOPMENT

13

EVALUATING THE EFFECTIVENESS OF A GRADUATE COURSE REDEVELOPMENT

14

EVALUATING THE EFFECTIVENESS OF A GRADUATE COURSE REDEVELOPMENT

15

EVALUATING THE EFFECTIVENESS OF A GRADUATE COURSE REDEVELOPMENT

16