EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

Evaluation Proposal for the Online Distance Learning Program

Meizhen Chen

December 05, 2012

EDF 5641

Dr. Linda Schrader

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

Table of Contents
Executive Summary.................................................................................................................................................................1 Overview ..............................................................................................................................................................................1 Program Background ...........................................................................................................................................................1 Purpose of Evaluation .........................................................................................................................................................1 Evaluation Approach and Question .....................................................................................................................................1 Data Collection Methods .....................................................................................................................................................1 Reporting in Evaluation .......................................................................................................................................................1 Evaluation Standards ...........................................................................................................................................................1 Program Description ...............................................................................................................................................................2 Background .........................................................................................................................................................................2 Initiation ..............................................................................................................................................................................2 Goal, Objectives and Expected Outcomes ...........................................................................................................................3 Stakeholders and Target Audience ......................................................................................................................................4 Service and Activities ...........................................................................................................................................................6 Resource and Budget ...........................................................................................................................................................7 Logic Model..........................................................................................................................................................................7 Evaluation................................................................................................................................................................................8 Rational ...............................................................................................................................................................................8 Evaluator Credibility ............................................................................................................................................................8 Purpose ...............................................................................................................................................................................9 Questions ...........................................................................................................................................................................10 Criteria and Standards .......................................................................................................................................................10 Approach ...........................................................................................................................................................................10 Evaluation Plan ..................................................................................................................................................................11 Communication Plan .............................................................................................................................................................17 Program Evaluation Standards .............................................................................................................................................18 Reflection ..............................................................................................................................................................................20 Reference ..............................................................................................................................................................................20 Appendix ...............................................................................................................................................................................21 Logic Model........................................................................................................................................................................23 Evaluation Plan ..................................................................................................................................................................29 Data Collection Plan ...........................................................................................................................................................35 Example of Instrumentation ..............................................................................................................................................37

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

I. Executive Summary
Overview This evaluation proposal is in response to the request of the Online Distance Learning Program (ODL) in Dedman School of Hospitality (DSH), examining to what extent the outcomes have been achieved. Program Background The goals of the ODL Program are: to reach and educate all on-campus students the basic knowledge and skills of hospitality management via internet, thus to help promote the hospitality industry in the state of Florida. The ODL Program currently consists of five elective online courses, which have been distributed through Blackboard since the 2011 spring semester. The funding for facilitating the program mainly comes from students tuitions. Purpose of Evaluation The purpose of the evaluation is to judge the quality of the online courses in the context of the current program, in terms of the course administration, instructional design, technology support, student achievement and satisfaction, faculty professional development and attitude, and logistical service management. The result of this evaluation is expected to be used by the stakeholders for their decision making, that is, whether continue, expand or terminate the funding for the new online courses development. Evaluation Approach and Questions Regarding the difference in the course content and assessments, this evaluation proposal will first focus on the quality of the most earliest distributed course online, Ales, Lagers & International Culture, which can provide sufficient source for data collection and analysis. Also, the quality of this course is expected to impact the stakeholders decision making in whether to invest funding on the new online course. Therefore, the nature of this evaluation is summative. The approach employed in this evaluation proposal is Stufflebeams CIPP (Context, Input, Process and Product) Model (Fitzpatrick, Sanders, Worthen, 2011, Pg 173-178), which is designed to provide information for program manager helping judge merit and decision making. Thus, this approach can be used to address the component Product of the program. In addition, with the constraint of the budget and time, the evaluation will be conducted by an internal evaluator. Data Collection Methods Relevant qualitative and quantitate data of the component Product will be collected within an entire semester period, through documents, students achievement data, surveys, interview and focus group. The entire process of data collection will have three parties of stakeholders participate into, including program administrators, subject experts and course enrolled students. Reporting in Evaluation The program administrators will be kept informed by the weekly data report. A final detailed written documentation with findings resulting from the data analysis, and recommendation for improvement will be presented to the program administrators, while the other participants will have a simplified report. Evaluation Standards The applicable evaluation standards for this evaluation proposal are: U2: Attention to Stakeholders and A2 Valid Information.
Page 1 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

II. Program Description

BackgroundDedman School of Hospitality The DSH is an independent, industry-specific school which is administered through the FSU College of Business. It has its own admission and degree requirements. Students can earn a Bachelor Degree of Science at HTM or PGM. The school offers its students with specific facilities to practice and hone their knowledge and skills in the real world, such as a teaching kitchen, a professionally managed city club and a placement center. The mission of the school is [To] provide the kind of education tomorrow's hospitality leaders will need.

Regarding Florida as a booming market of hospitality and tourism, the DSH also offers five elective face-to-face courses to on-campus students of all majors initially, which help broaden students career options. These courses are Introduction to Hospitality and Tourism, Ale Lager International Culture, International Wine & Culture, International Food and Culture and Managing Service Organizations (See Appendix 1 for a full list and description of the five elective courses).

Initiation of the Program

The elective face-to-face courses were always immediately full enrolled after they were offered to the on-campus students. On the other hand, the DSH only has four classrooms for conventional instruction. The biggest classroom accommodates no more than 50 students, while the smaller ones have a capacity of 10 to 20 students. These classrooms are mainly served for students majored in HTM or PGM. In addition, there are fourteen full-time faculties currently teaching 44 conventional courses in DSH, featuring one faculty from College of Business. The faculty number is relatively smaller compared with other schools in FSU.
Page 2 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

To address the shortage of classrooms and faculty, and to meet the increasing student enrollment for the elective courses, the DSH initiated an Online Distance Learning Program (ODL), converting those elective courses content from face-to-face format into web-based format. The first online course was delivered through Blackboard to students in the 2011spring semester.

Goal, Objectives and Expected Outcomes

The goals of the ODL program are to educate the basic hospitality management knowledge and skills for all FSU students via internet, and help promote the hospitality industry of the state of Florida. The objectives aligned with the ultimate goal are: (cited from the courses syllabus of 2012 fall semester) Help student master the fundamental concepts of the hospitality and tourism disciplines. Help student understand the management operations in the hospitality and tourism industry. Help student seize the developments and trends in the hospitality and tourism industry Help student identify the career opportunities and choices in the hospitality & tourism industry Help student demonstrate the leadership in management of the hospitality & tourism industry Increase students access to courses without faculty, time or location constraints; Maximize use of technology available for cost-effective instruction; Maximize use of specialized expertise of faculty and staff.

The short-term intended outcomes include: Meet the students' needs of flexibility learning schedule and quality of instruction Enhance students' learning efficiency via distance learning
Page 3 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

Enhance students knowledge and skills pertaining to hospitality and tourism management Enhance students interests, motivation and participation in learning hospitality and tourism management Enhance faculty's expertise in online instructional design

The long-term intended outcomes include: Reform students' study styles from face-to-face instruction to completely distance learning Promote the hospitality industry for the state of Florida by the excellent graduates from FSU. Increased readiness for future educational and career opportunities Enhance students autonomy Reform students' study styles from face-to-face instruction to online distance learning Increase desire to learn and practice hospitality and tourism management

Stakeholders and Target Audience (see Appendix 2) ODL Program administrators, including The Curriculum Committee ----Censor the proposed courses goal statement, content and assessment ----Examine whether the course qualifies with the basic course standard of FSU The Office of Distance Learning ---A separate office support online learning within the residential instruction environment. ---Provide permission of residential institutions request of developing online courses. ---Provide a central coordination between Blackboard and all FSU online courses. Blackboard Learning Management System ---Provide the readiness of technology infrastructure
Page 4 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

---Provide an online instruction and learning platform for participants. ---Provide technology support ---Provide user training Board of directors at DSH ---Program policy maker ---Provide vision and leadership for the program directions ---Initiate, operate and monitor the ODL Program ---Determine the continuation, expansion and termination of the funding ---Response to the request from the Office of Distance Learning Staff/Personnel ---Follow and response the directions of the program directors ---Prepare necessary resource for implementation of the program ---Schedule meeting among various stakeholders --Coordinate student enrollment and dropouts

Subject-Matter Experts Instructor, instructional designer, developer and facilitator ---Compliance with standards and principles of the program ---Plan, design, develop and distribute the course materials, tasks, assessments. ---Conduct and arrange instruction activities ---Provide feedbacks to students regarding to the course context Technician ---Provide technology support ---Maintain the readiness and accessibility of the technology infrastructure
Page 5 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

Target Audience Online courses recipients All FSU students at main campus, not include the remote-site learning students Mostly are undergraduate students Come from different majors with different ages, genders and various entry knowledge, skills, experience and learning styles

Service and Activities Provision of Support for faculty To balance facultys workload and enhance the course regulatory efficiency, all faculties are provided with one or two Teaching Assistants (TA) mainly hired from Instructional System Program from the College of Education. These TAs are current graduate students who are mastery of knowledge and skills in online instructional design and development. They are expected to assist faculty to convert the conventional course materials into web-based format, while utilizing appropriate multimedia to enhance the interaction between students and the course materials. Other TAs are graduate students hired from Business School, who are expected to contribute their business perspectives into the course design. All TAs have the privilege to help manage the online courses, from producing announcement, organizing activities, replying students question via emails to grading portion of assignments.

Provision of Student Service Blackboard offers an online platform for students operating learning activities, where they can read, download, post and submit the course materials. All the online courses are three-credit course, which comprise of 12 modules, 48 topics, 48 assignments, 12 quizzes, 2 examinations and 1 group project. The
Page 6 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

learning materials include PowerPoint, textbook content, literatures, graphics, audio and videos. Students are expected to spend 5 to 10 hours to accomplish the weekly learning tasks. All these online courses are asynchronous. That is, students can flexibly arrange their time or locations to complete the weekly course content. Faculty and TAs can provide response to students regarding to the course content questions. Blackboard also provides technology support when students encounter any technology issues.

Resources and Budget Hardware DSH purchased necessary equipment for supporting faculty and TAs to develop the online courses, such as laptops, computers, software and their copyrights, and printers. Others resource includes textbooks, paper, ink, pens, pencils. Software Specific personnel are created for coordinating the communication between the board of administrators and the course developers. The personnel host bi-weekly conference with the faculty and TAs, provide feedbacks and report the meeting minutes to the upper level. The funding comes from students tuitions. The budget is divided into parts, facultys compensation for their workload, TAs salaries, equipment purchases and other social activities.

Logic Model In order to have an efficient communication with the DSH administrator group about the whole picture of the program, this proposal formulates a logic model (see Appendix 3) , which is presented in a diagram form, including the programs input, activities, outputs, short-term outcomes, long-term outcomes, evaluative indicators, evaluative standards, sources and methods of collecting data. This logic
Page 7 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

model will be expected to help the DSH administrators further articulate each components of the program at their perspectives during the discussion with the evaluator.

III. Evaluation
Rational for Evaluation There are at least three reasons for conducting this evaluation, which are stated below. First, although the ODL Program has been facilitated for almost two years, it still lacks of a systematic and scientific evaluation system on the course quality in terms of its design phase, delivery phase and implement phase. The program currently only depends on the feedbacks from the student evaluation each semester. Furthermore, the utilization of these results is not clarified, and there is no follow-up evaluation to examine whether any changes might have occurred based on those results. Secondly, the program director who initiates the program expresses a strong interest in learning to what extent any outcomes have achieved, and where the program can improve. All these intended results from the evaluation will be benefit for the program director and other administrators to decide whether continue to sponsor the new online courses development. Last but not the least, this evaluation will only cover the earliest course, because it can be collected the most sufficient information. Furthermore, the course content and assessment are varied in five online courses. Hence, the evaluative approach and the process of the evaluation can serve as a reference for the rest of the courses. On the other hand, the process of evaluation also can contribute to the stakeholders acknowledge of the program evaluation, and empower them the active participation in the evaluation.

Evaluator Credibility

Page 8 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

This evaluation will be conducted by an internal evaluator, a graduate student major in Instructional System in the College of Education. She has been hired in the program since 2012 spring. She assisted three out of five online courses as a TA, as well as a content developer since being hired. On the hand, she has never assisted in the course to be evaluated in this proposal. Hence, she can maintain a relative objectiveness as an internal evaluator when conducting the evaluation. Regarding her profession and working experience, she is expected to be qualified and to attain the trust in the stakeholders.

Purpose of Evaluation To meets the compliance and outcomes requirements prescribed by the program administrator group, the evaluation will examine the programs quality at a holistic standpoint, meanwhile, the internal evaluator team will specifically look into the quality of the earliest course. The intended outcomes of this evaluation are expected to be used by the stakeholders to make decision about the continuation, expansion, termination and discontinuation of the funding on the current program.

Evaluation Questions The following evaluation questions are generated based on the communication with the program directors, which can help provide the direction of the following evaluation, and the foundation for the selection and implementation of the corresponding evaluative approaches. These questions also compliance with the programs goal and guide the evaluator to collect important information and to help stakeholders decision making. (See Appendix 4)

1. What is the quality of the online course Ales, Lagers & International Culture of the ODL Program?

Page 9 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

(1) Is the course instructional design consistent with the programs prescribed goals and objectives? (2) What is the students perception of the course quality? (3) To what extent the outcomes have been achieved from the course? 2. To what extent does the online course meet the program goals and objectives? (1) How is the consistency of the course effectiveness and the program goals and objectives? (2) Has student enrollment continued to increase? (3) How many students are hired in the hospitality industries? 3. Should the program continue to develop new online courses or not? (1) What is cost effect of the current program? (2) What strategies or recommendations for administrator decision making? These questions will be answered from data collection in terms of policy review, learning management system test, student achievement data and faculty performance, along with student, staff and faculty observation, surveys, interviews and focus groups.

Evaluation Criteria and Standards Fitzpatrick Sanders and Worthen ( Program Evaluation, page 332, 2011) emphasize that Criteria and the standards for those criteria [enable] an evaluator [to] make the judgment for the question requires. Therefore, this proposal identified a series criteria and corresponding standards from discussion with the stakeholders, which have been further determined and confirmed with the stakeholders as well. These criteria and standards are intended to answer the questions, and as the indicators for measuring the short-term outcomes and long-term outcomes. They can also be revised according to the change of the stakeholders expectations prior to implementing the data collection.

Page 10 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

The specific criteria and standards are presented in the programs logic model, which can be referred to the Appendix 3.

Evaluation Approach To answer the evaluative questions, and increase the chance of the results to be used for program administrators decision making, this proposal will employ a CIPP (Context, Input, Process and Product) Model from Decision-Oriented Approach in this summative evaluation. This model is designed to provide program manager with information of four components, which can enhance the accuracy of program managers decision making. Based on the summative nature of the proposal evaluation, the evaluation will closely look into the 4th component Product. This component cares about whether the program has achieved the goal, and what outcomes are. And these two questions are exact the interests that DSH stakeholders express.

Evaluation Plan The purpose of creating an evaluation plan is to ensure the process of the evaluation can be operated systematically and scientifically. Hence, the components of this plan includes the evaluative questions, and their corresponding qualitative data or/and quantitate data, data sources, data collecting methods, data collecting procedures, data analysis procedures and timeline (See Appendix 5, 6 and 7).

Timeline The proposal evaluation will be implemented thorough the spring semester of 2012, during which the program administrator group will be notified with the progress of the evaluative activities weekly. The relevant information, data, findings will be formulated as a weekly report and present to the

Page 11 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

stakeholders. Stakeholders are expected to provide feedback and articulate their perception and expectation, so that some necessary changes can be adopted or adjusted. Week 1-2 ---Initial interview with DSH program administrators Learning about programs organizational culture, operation, management and expectation ---Locate relevant documents and records, such as student enrollment/dropout record ---Observe the instructional activities of the present and previous online course ---Provide the instructor with the student pre-survey and post-survey instruments and have his/her input Week 3-4 ---Release students the corresponding survey via digital format ---Currently enrolled students are all required to complete the pre-survey for course credits ---Previously enrolled students will be invited to complete a post-survey ---Two parties of students will be asked whether volunteer to participate into focus group in the end of the surveys ---Observe the instructional activities of the present and previous online course ---Compare the course content, materials, activities, assessments between 2011 and 2012 Week 5-6 ---Gather and analyze both surveys results ---Produce an initial report based on the results ---Present and discuss the report with the DSH program administrators ---Produce and present an interim report with the instructors and have their input ---Students who volunteer to participate into focus group will be randomly selected and informed ---Observe the instructional activities of the present and previous online course
Page 12 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

---Compare the course content, materials, activities, assessments between 2011 and 2012

Week 7-10 ---Conduct focus group of previously enrolled students ---Conduct focus group of instructor and TAs ---Produce and present a discussion minute with the program administrators ---Prepare a post-survey for presently enrolled students ---Discuss the post-survey with the instructors and have their input ---Observe the instructional activities of the present and previous online course ---Compare the course content, materials, activities, assessments between 2011 and 2012 Week 11-14 ---Collect presently enrolled student achievement data ---Release post-survey to presently enrolled students ---Conduct focus group of presently enrolled students ---Conduct focus group of the instructor and TAs ---Produce a draft report based on the results ---Present and discuss the report with the DSH program administrators ---Review all data collected and records ---Produce a draft of the final report ---Discuss the draft of final report with program administrators ---Produce a complete and detailed report with data, findings and recommendation of the program improvement for program administrators ---Produce a simplified report along with charts, graphs, and tables for the instructors

Page 13 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

Methodology The evaluation questions in this proposal focus on the outcomes and impact of the program, thus a descriptive design is applicable for collecting the information to the questions. Regarding it as an internal evaluation, the first-hand raw data can be relatively easily to yield. Hence, the presently and previously enrolled students of the course, instructor, and other stakeholders will be the sources for collecting the relevant information. Once the source is identified, the appropriate methods of data collection will be selected from or combined with surveys or questionnaires, interviews, focus groups, observation, institutional records and documents.

Content Analysis In addition to the human dimension, sources such as the course content, course materials, course discussion board, assessment instruments, multimedia tools, equipment and program documents will be included. These types of descriptive data can provide the evaluator insights of the program from various aspects.

Sampling, Survey and Focus Group (see Appendix 8, 9, 10 Collection Plan, and Appendix 11 Example of Instrumentation. ) One of the advantages of the online course Ales, Lagers & International Culture is that it has a large amount of student enrollment. Since its an online course, it is also convenient to deliver an electronic survey to the population. The electronic survey will be required as the portion of the course tasks and credits. It will be built and sent through Blackboard Qualtrics in the middle of the semester. Students will have one week to complete and submit them. The survey incorporates standardized questions and open-ended questions, which can help evaluator to collect both quantitative and qualitative data. The items mainly focus on students perception of the course quality and their learning
Page 14 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

characteristics. One of the items in the student survey will ask whether they would like to participate in a focus group. If students decide to participate, they need to provide their contact information in an additional comment box, such as email and telephone number, so that the survey can remain anonymous. Once the number of students who indicate to participate into the focus group is collected and big enough, every 10 students will be purposely placed in three focus groups, according to their mid-term achievement data. Three groups are built according to high, medium and low achievement. This allows the evaluator to identify the course quality from three level achievements of students. If the population for focus group is not big enough, students will be randomly placed into three focus groups. A different survey will be sent out to the students who previously enrolled this course. They will be also invited to participate in the focus group as volunteers. Students will be notified that incentives are offered incentives once they participate. Since the number of instructors and TAs of this course is rather small, after they complete a survey, all of them will formulate as a focus group. The questions of the focus groups will be built in terms of the results of the survey, so that the evaluator can identify the specific problems of the course during the group discussion. With the permission of the participants, the discussion during the interview will be taped or written recorded. The stakeholders will be presented a summary of the findings after the minute fully analyzed.

Presenting Results Blackboard discussion board will be the place for evaluator to submit the period reports, such as course material analysis, instructional activity observation analysis, survey results and focus group findings. Some data is produced on a continuing basis, which needs the stakeholders input to guide the next phase. Stakeholders can access the discussion board and leave the comment or suggestion based on
Page 15 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

the report. The interactive communication between stakeholders and the evaluator can contribute in achieving the intended result of the evaluation, which can effectively address the stakeholders concerns. The final detailed report will be finalized based on the reports and comments. It particularly measures the outcomes of the online course Ales, Lagers & International Culture to provide a model to measure other online courses quality. In the meantime, it also examines the program operation and management which directly impact the course quality. The report will provide recommendation based on the findings. Hence, the proposed evaluation should help the stakeholders consider whether continue investing in the new online courses.

Required Resource and Budget Since it is an internal evaluation, most of the resource has been provided. So the required resource and its corresponding cost for conducting the evaluation include: ---Blackboard ($0, provided) ---Computer and office for evaluator working ($0, provided) ---Stationery ($0, provided) ---Statistical analysis software ($0, provided) ---Conference room for interview and focus group ($0, provided) ---Incentives for student interview and focus group ($ 100) ---Participation of program administrators, instructors and TAs (two program directors, two program assistants, one instructor and two TAs) ($3500) ---Evaluator payment ($1500)

Page 16 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

The maximum cost of the proposed evaluation is estimated to be $ 10000, which included the additional cost. If the necessary resource is cut, the length of expected evaluation will be extended, which will produce extra cost for paying the evaluator.

IV. Communication Plan

Fitzpatrick, Sanders and Worthen (Program Evaluation, pg454-458, 2011) suggest that the communication with all stakeholders should be an ongoing dialogue, the reports resulting from which should be tailored to the specific audience interests and needs. In the proposed evaluation process, three parties of stakeholders have different level of participation of the program, representing their different interests and needs. To enhance the chance of using the result, this proposal will present the specific group of stakeholders with different format of report, to help address their concerns.

Program Administrator Communication

Prior to a weekly meeting with the stakeholders, a weekly based report resulting from the evaluation progress will be offered to the stakeholders. The evaluator will submit the reports via Blackboard discussion board, which provides a platform to record/document the interaction between the stakeholders and the evaluators. Based on the present week reports, stakeholders and the evaluator can handle some surprising findings more effectively. A final report will be written in Word document and email to stakeholders. It summarizes all the data and findings, which includes the reflection of the evaluator based on the weekly discussion with the stakeholders, and the recommendation for stakeholder making decision of the program. The written report also will be transformed into a PowerPoint presentation, which is expected to help stakeholders grasp the essential information effectively.
Page 17 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

Instructor and TAs Communication

The instructor and TAs will be informed to complete an online survey through Blackboard Qualtrics via email in week 9 to 10. The results of the survey will be presented to them in the other discussion board on Blackboard in week 11, where they also can leave the comments and suggestion. This discussion board will be not seen by stakeholders, but the evaluator will summarize the information and provide them with stakeholders. The purpose of this discussion board is to engage the instructor and TAs to freely talk their concerns and needs. Moreover, the interview will be conducted in week 12, where the instructor and TAs can further elaborate their concerns. The final report will be tailor to mainly provide the data and findings, which result from the surveys of the students and instructors.

Students Communication
As stated as previously, the survey will be required to complete by student as a course task. Student will earn a portion of credit from the course once they submit the survey by the designated time (by week 7). Since the target audience is on-campus students, so it should be relatively easier to contact the students to participate the focus group. Hence, an invitation of participating in the focus group will be attached in the last item of the survey, where students can type their contact information with confidential format. The result of survey may not provide with students, which is determined by the stakeholders. Evaluator will take use of the survey result to generate the questions for facilitating the focus groups.

V. Program Evaluation Standards

Page 18 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

U2 Attention to Stakeholders Evaluations should devote attention to the full range of individuals and groups invested in the program and affected by its evaluation.

This standard selected is intended to identify the voices from the three parties of stakeholders, especially the program administrator group and faculty. Since the first online courses has been facilitated for two years, the previously course enrolled students perceptions are also important, which can help the evaluator identify what outcomes have been continually achieved. Only through the sufficient communication, the evaluator can have a comprehensive picture about each partys expectations. Once find out the conflicts, the evaluator can provide appropriate recommendations or intervention for program directors. For instance, what is the perception of the faculty to be required to develop the online instruction? Does the program policy haves any specific compensation for the faculty who works on the ODL Program? Are they motivate to do this additional workload? What kind of supports do they need to develop a quality online instruction? Is the online instructional training helpful for them? The information for these questions will be mainly collected from the anonymous survey.

P6 Conflicts of Interests: Evaluations should openly and honestly identify and address real or perceived conflicts of interests that may compromise the evaluation.

Regarding the role as an internal evaluator in the program, its essential for the evaluator to remain an independent and objective perspective when communicating with different types of stakeholders. Since the program has been implemented without a systematic evaluation for two years, some unexpected consequence or surprise outcomes might be identified through this proposed evaluation. Hence, the evaluator should be able to justify the data and findings in terms of the theories, methodology and tools

Page 19 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

that being applied and used, to help all stakeholders understand how the results of the evaluation come into being.

VI. Reflection
Although this assignment is going to be wrapped up soon, it is still a long way for me to master the essence of program evaluation. During this long journey, it was actually full of rewards to learn every bit of evaluation, either from Dr. Schraders intrinsic praise or from the peers open critique. The following bulleted points are the knowledge and skills I have acquired from this course, most of which were my misconceptions and have been clarified by my professional instructor and wonderful classmates. 1. Difference between purpose of program and evaluation. The purpose of the program usually serves for a specific group or groups benefits, while the purpose of evaluation is to examine whether or how well the program works for these stakeholders. 2. Difference between resource and activities in Logic Model Activities emphasize what participants are doing can result the intended outcomes by using the resource, so they usually do not include the activities of preparing the resource, such as purchasing the equipment. 3. The relationship between the Logic Model and CIPP Model The Logic Model can be applied to analyze the components of a program, from input, activities, output and outcomes. By using this model, an evaluator can clarify what the program currently exactly possesses. The CIPP Model is an evaluation approach, which can help an evaluator to decide which component of the program should be evaluated and how to be evaluated.

Page 20 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

Last but not the least, my role into this class has been dramatically changed since being required to conduct this project. Only closely associating with the stakeholders, till then I can better comprehend those fact knowledge stated in the book. The deeper I was immersed into the program, the more I could understand and apply. So the accountability was not just for attained a number, but a truly understanding what program evaluation is. Thanks to Dr. Schrader provides us such a good project to learn from doing, and your meaningful feedbacks in and off the class.

Reference
Fitzpatrick, J.L., Sanders, J.R., & Worthen, B.R. (2011). Program evaluation: alternative approaches and practical guidelines. Upper Saddle River, NJ: Pearson. Holsapple, C., & Lee-Post, A. (2009). How to Design, Develop, and Deliver Successful E-Learning Initiatives. Student Satisfaction and Learning Outcomes in E-Learning: An Introduction to Empirical Research, 9, 195-229. Can main campus students take online courses? | Office of Distance Learning. (n.d.). ODL | Office of Distance Learning. Retrieved November 2012, from http://distance.fsu.edu/students/can-main-campus-students-take-online-courses

Page 21 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

Appendix 1

Courses of Online Distance Learning Program in Dedman School of Hospitality

HFT1000 - Introduction to Hospitality and Tourism (3) This course offers an introductory review of the segments, disciplines, career opportunities, and current issues in the hospitality industry. Tuition and Fees: The cost of this course is equal to undergraduate fees for main campus students plus a $70 per credit hour distance learning fee. HFT2061 - Ale Lager International Culture (3) An introduction to wines of the world. This course meets the University's multicultural course requirements. Tuition and Fees: The cost of this course is equal to undergraduate fees for main campus students plus a $70 per credit hour distance learning fee. HFT2062 - International Wine & Culture (3) This course provides an introduction to wines of the world with a focus upon the importance to global cultures. This course meets the University's multicultural course requirements. Tuition and Fees: The cost of this course is equal to undergraduate fees for main campus students plus a $70 per credit hour distance learning fee. HFT2890 - International Food and Culture (3) This course explores the world's cuisines with a focus on the history of culinary arts, indigenous ingredients, and customs, as well as various cooking methods and terminology from around the world. This course meets the University's multicultural course requirements. Tuition and Fees: The cost of this course is equal to undergraduate fees for main campus students plus a $70 per credit hour distance learning fee. HFT3240 - Managing Service Organizations (3) An in-depth examination of the concept of service and the linkages of marketing, operations, quality, and human resources. Tuition and Fees: The cost of this course is equal to undergraduate fees for main campus students plus a $70 per credit hour distance learning fee.

Reference:
Can main campus students take online courses? | Office of Distance Learning. (n.d.). ODL | Office of Distance Learning. Retrieved November 2012, from http://distance.fsu.edu/students/can-main-campus-students-take-online-courses

Page 22 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen Appendix2:

2012

Checklist of Potential Stakeholders and Audiences

To Make Policy To Make Operational Decisions To Provide Input to Evaluation

Entity to be evaluated: Online Distance Learning Program of Dedman School of Hospitality at Florida State University Individuals, groups, or agencies needing the evaluations findings Category Audience member For Interest Only

To React

Developer of the program Funder of the program Person/Agency who identified the local need Boards/agencies who approved delivery of the program at local level Local funder Other providers of resources (facilities, supplies, inkind contributions) Top managers of agencies delivering the program Program managers Program deliverers Sponsor of the evaluation Direct clients of the program Indirect beneficiaries of the program (parents, children, spouses, employers) Potential adopters of the program

ODL program personnel ODL program personnel ODL program personnel FSU Office of Distance Learning, Curriculum Committee N/a n/a ODL program personnel ODL program personnel ODL program personnel; instructors ODL program personnel OTD personnel; instructors; FSU on-campus students Florida hospitality industries FSU on-campus students not enrolled in program
Page 23 of 40
X X X

x x x x

x x x x x

X x X X x X
X

X
X

EDF 5461: Final Evaluation ProposalMeizhen Chen

Individuals, groups, or agencies needing the evaluations findings Category Audience member To Make Policy

2012 To Provide Input to Evaluation

To Make Operational Decisions

To React

For Interest Only

Agencies who manage other programs for this client group Groups excluded from the program Groups perceiving negative side effects of the program or the evaluation Groups losing power as a result of use of the program Groups suffering from lost opportunities as a result of the program Public/community members Others

N/A

N/A N/A

N/A

FSU on-campus students who are unaware of ODL Program N/A N/A

Page 24 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen Appendix 3:

2012

LOGIC MODEL for_Online Distance Learning Program at Dedman School of Hospitality ___
Project Summary / Project Purpose: To measure the quality of the online course delivered by Hospitality School, and identify its limitations for improvement

Inputs Program Administrator Curriculum Committee at FSU Office of Distance Learning at FSU

Activities Census feasibility of the curriculum Provide distance learning policy guidance Coordinate with Bb to implement OPL Program Provide learning platform Provide users workshops Provide technology front-desk support Facilitate ODL Program Policy Lead, operate and manage the program operation Compliance with request from administrators Regulate faculty and student service

Outputs The numbers of the courses is approved The numbers of guidelines The numbers of course allowed to distribute via Bb The numbers of course on Bb The numbers of workshops The schedule and categories of technology support The numbers of recruitment The numbers of conferences The structure of the program model The numbers of conferences The numbers of relevant documents The schedule and categories of front desk support

Short-Term Outcomes

Long-Term Outcomes

Evaluation Indicators /Standards Sources/Methods S1 Sources Faculty Course Content Blackboard Help Methods Document Review Observation Survey Interview Focus group L1 Sources Program document Local industries Alumni Methods Document Review Survey L2 Sources Program document Local industries Alumni Methods Document Review Survey

L1. Increased readiness for future educational and career opportunities S1. Enhance faculty's expertise in online instructional design. L2. Promote the hospitality industry in the state of Florida S1. 85% of DSH faculty trained to teach online distance learning courses L1. 100% student job placement L2. 29% DSH increased partnerships with hospitality industries in the state of Florida

Blackboard

DSH ODL Program Administration Group

DSH ODL Program Coordinator

Program Policy

Page 25 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

Short-Term Outcomes

2012

Inputs

Activities Design Activities Compliance with program objectives Define the scope objectives, assessment methods of the course Conduct learners analysis Development Activities Develop course modules, study guidelines & syllabus Develop assessment scale Select appropriate media and tools Evaluation Activities Review course materials accuracy Examine accessibility of course website Examine availability of course tools Delivery Activities Obtain access to the course platform via Bb Upload course materials on Bb Implement Activities Implement instructional strategies

Outputs

Long-Term Outcomes

Evaluation Indicators /Standards Sources/Methods S2, S3, S4. 85% of students satisfaction of the course quality S3. 50% of students acquire A, 48% of students attain B, 2% of students attain B below S5. 10% continual increased enrollment within two years. S6. 90% of majors of students enroll the program. L3. 85% of students submissions are on time. L4 20% of students online courses amount is more than the conventional classes L5. 30% of students enroll more than two courses of the program S2,S3,S4 Sources Students Students achievement data Methods Survey Interview Focus group

Faculty and TAs

S2. Meet the students' needs for flexible and quality of instruction S3. Enhance students knowledge and skills pertaining hospitality and tourism management S4. Enhance students online learning techniques S5. Enhance student enrollment S6. Reach a larger diversity of students

L3. Enhance students autonomy L4. Reform students' study styles from faceto-face instruction to online distance learning L5. Increase desire to learn and practice hospitality and tourism management

The number of online courses developed, including the numbers of modules, units, topics, syllabus, lesson plans, handouts, assignments, quizzes, exams and projects.

S5, S6 Sources Documents Methods Document Review Survey L3 Sources Documents Students Instructors Methods Document Review Observation Survey Interview Focus group L4 L5 Sources Documents Students Methods Document Review Survey

Page 26 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

Short-Term Outcomes S1. Enhance faculty's expertise in online instructional design.

2012

Inputs DSH Staff

Activities Provide faculty service Provide student front desk service

Outputs The number of weekly meetings The schedule and categories of assistantship

Long-Term Outcomes

Evaluation Indicators /Standards Sources/Methods S1 S1. 85% of DSH Sources faculty trained to teach Faculty online distance learning Course Content courses Blackboard Help Methods Document Review Observation Survey Interview Focus group S3. 85% of students satisfaction of the course quality S3. 50% of students acquire A, 48% of students attain B, 2% of students attain B below L3. 85% of students submissions are on time. S3 Sources Students Students achievement data Methods Survey Interview Focus group

On-Campus Students

Enroll online courses Study course content Participate in course activities Complete evaluation

The number of students of enrollment/dropouts The attendance of students in online courses Students achievement grades The numbers of valid evaluation

S3. Enhance students knowledge and skills pertaining hospitality and tourism management

L3.Enhance students autonomy

Resource/Equipment Computers Laptops Printers Software/Copyrights Textbooks/Copyrights Library research and literature Local industry or business community

Involved into all activities stated above

Involved into all activities stated above

Page 27 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen Appendix 4:

2012

Divergent Evaluation Questions Matrix

Original List of Evaluation Questions: 1. What is the quality of the online course Ales, Lagers & International Culture of the ODL Program? 2. Is the course instructional design consistent with the programs prescribed goals and objectives?
3. What is the students perception of the course quality? 4. What is the facultys perception of the workload of this program? 5. What is the facultys perception of the workshops pertaining to online instructional training?

6. To what extent the outcomes have been achieved from the course? 7. To what extent does the online course meet the program goals and objectives? 8. How is the consistency of the course effectiveness and the program goals and objectives? 9. Does student enrollment of this course continue to increase? 10. How many FSU students are hired in the hospitality industries in Florida? 11. Should the program continue to develop new online courses or not? 12. What strategies or recommendations for administrator decision making? 13. How effectives is the technology supporting for the faculty and students in this program?
Questions Be of interest to key Audiences? Reduce present uncertainty? Yield important information? Be of continuing interest? Be critical to the studys scope and comprehensiveness? Have an impact on the course of events? Be answerable in terms of a.) financial &human resources b.) time and c.) available methods & technology 1
X X X X X X X

2
X X X X X X X

3
X X X X X X X

4
X X X X X X X

5
X X X X X X X

6
X X X X X X X

7
X X X X X X X

8
X X X X X X X

9
X X X X X X X

10
X

11
X X X X X

12
X X X X X X

13
X X X X X

X X

Page 28 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen Appendix5:

2012

Evaluation Plan:_Online Distance Learning Program at Dedman School of Hospitality ___

Information Required Measures of Activity The number of courses are produced The number of syllabus, lesson plans, lecture slides, handouts, study guides, assignments, quiz, and projects. The number of faculty and TAs The number of students served The number of potential students are turned away The number of training workshops The frequency of troubleshooting support Measures of Efficiency The level of instructors subject matter knowledge, skills of engagement and facilitation The instructors perception: readiness and preparedness to using technology The number of students completed the course successfully The average students workload The number of students enrolled in additional courses The course cost Measures of Outcomes Students perceptions: --motivation, -- readiness and preparedness to delivery method --community sense --interaction Training Measures of Program Aims Measures of Policy Course Standard Evaluation Procedures Measures of Technology Access Design Descriptive Design Qualitative data will be gathered in the form of survey, interviews, focus group, and document review. Quantitative data will be gathered in the form of student achievement data, enrollment, dropouts and survey. Information Source Administrative records Blackboard course content Method for Collecting Information Document Review Observation Survey ---current online course enrolled students, faculty ---previous students. Interview Focus group

Evaluation Question Main Question: 1. What is the quality of the online course Ales, Lagers & International Culture of the ODL Program?

Page 29 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

Sampling Survey to all students and all faculty and TAs (Student= 600; faculty=1; TAs=2) Interview with student sample of 30, and all faculty, TAs and administrative group Focus group with 30 students, 1 faculty, and 2 TAs , 2 administrators

Information Collection Procedures Document Review Survey distributed digitally before the students completed the course Interviews arranged after survey Focus group conducted after interview

Schedule Week1-2 ---Collecting Information Week3-4 ---Survey, observation Week5-6 --- Survey data analysis --- Produce report --- Interview, observation Week7-10 ---Focus group ---Observation Week11-14 ---Focus group data analysis ---Produce final report

Analysis Procedure Descriptive stats and chi square for survey Use results for interview Summarize major themes of interviews Use results to plan focus groups Use taped transcript of focus groups for analysis Integrate all results to describe trends and solutions

Reporting Procedures Interpretation Procedures Have at least 85% of students indicated that they satisfied with the course quality? Does the faculty indicate they feel confidence to teach and manage class online? Audience(s) Funding sources Program administrators Program delivers Program facilitators Program users Program participants Content Help Answer question Is the course instructional design consistent with the programs prescribed goals and objectives? How well does the technology service support the student and faculty? What is the students perception of the course quality? What is the facultys perception of the online course instruction? Format Technical report to funding sources and program administrators Meeting s with funding sources to discuss results Meetings with administrators and deliverers to discuss their interpretation of results and possible changes Meeting with faculty to report results and receive their input Press release Schedule Week 5-6, Week 11-14 ---Meeting to discuss results Week11-14 ---Release report Week 14 ---Press release

Page 30 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen Appendix6:

2012

Evaluation Plan:_Online Distance Learning Program at Dedman School of Hospitality ___

Information Required FSU online distance learning program missions statement DSH ODL Program goal statement DSH ODL Program staff action plans DSH ODL Program history Findings of the first evaluation questions Design Descriptive Design Information Source Method for Collecting Information Document Review Observation Interview with stakeholders Focus group of staffs, and faculty

Evaluation Question Main Question: 2. To what extent does the online course meet the program goals and objectives?

FSU Office of Distance Learning Qualitative data will be FSU Curriculum gathered in the form of Committee interviews, focus group, and DSH ODL Program document review. administrators Finds of the first evaluation questions Schedule Week1-2 ---Interview with program directors --- Review documents, records Week5-6 --- Interview with the course instructor and TAs Week7-10 ---Focus group

Sampling Independently interview with the course instructor 1, TAs 2 and administrative group 3 of the program Focus group with 1 instructor, and 2 TAs , 2 administrators

Information Collection Procedures Document Reviews Generate questions for different interviews Interviews arranged with different stakeholders individually. Generate questions for Focus group Focus group of all stakeholders conducted after interview

Analysis Procedure Use results of first evaluation question for interview Use results to plan focus groups Use taped transcript of focus groups for analysis Integrate all results to describe trends and solutions

Reporting Procedures Interpretation Procedures Has the program achieved all objectives? Is it 100% student job placement? Audience(s) Funding sources Program administrators Program participants Content Help Answer question How is the consistency of the course effectiveness and the program goals and objectives? Has student enrollment continued to increase? How many students are hired in the hospitality industries? Format Meeting s with funding sources to discuss results Meetings with administrators and deliverers to discuss their interpretation of results and possible changes Meeting with faculty to report results and receive their input Schedule Week 5-6, Week 11-14 ---Meeting to discuss results Week11-14 ---Release report Week 14 ---Press release

Page 31 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen Appendix7:

2012

Evaluation Plan:_Online Distance Learning Program at Dedman School of Hospitality ___

Information Required DSH ODL Program history Findings of the first and second evaluation questions Design Descriptive Design Information Source Method for Collecting Information Document Review Interview with stakeholders Focus group of staffs, and faculty

Evaluation Question Main Question: 3. Should the program continue to develop new online courses or not?

FSU Office of Distance Learning Qualitative data will be FSU Curriculum gathered in the form of Committee interviews, focus group, and DSH ODL Program document review. administrators Finds of the 1st and 2nd evaluation questions Schedule Week11-14 ---Interview with all instructors ---Interview with all TAs ---Interview with all program administrators ---Focus group

Sampling Interview with all instructor 14, TAs 20 and administrative group 3 of the program Focus group with 5 instructor, and 3 TAs , 3 administrators

Information Collection Procedures Document Reviews Generate questions for different interviews Interviews arranged with different stakeholders individually. Generate questions for Focus group Focus group of all stakeholders conducted after interview

Analysis Procedure Use results of 1st and 2nd evaluation question for interview Use results to plan focus groups Use taped transcript of focus groups for analysis Integrate all results to describe trends and solutions

Reporting Procedures Interpretation Procedures What course should the program develop into webbased format? Can sponsor continue invest 20% of the funding into new online course ? Audience(s) Funding sources Program administrators Program participants Content Help Answer question What is cost effect of the current program? What strategies or recommendations for administrator decision making? Format Meeting s with funding sources to discuss results Meetings with administrators and deliverers to discuss their interpretation of results and possible changes Meeting with faculty to report results and receive their input Schedule Week 11-14 ---Meeting to discuss results Week14 ---Release report Week 14 ---Press release

Page 32 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

Appendix8:

Data Collection Plan: _Online Distance Learning Program at Dedman School of Hospitality

___
Question 1: What is the quality of the online course Ales, Lagers & International Culture of the ODL Program? Data 1. Measures of Activity The number of syllabus, lesson plans, lecture slides, handouts, study guides, assignments, quiz, and projects. The number of students enrolled and dropouts The number of training workshops The frequency of troubleshooting support 2. Measures of Efficiency Instructors subject matter competency of teaching online The instructors perception: readiness and preparedness to using technology (interview, focus group) The number of students completed the course scored above B The course cost Sampling Method 1. Purposive sampling. Quantitative or Qualitative 1. Quantitative Analysis

Quantitative data will be used to identify whether current course materials meet the standard of the FSU online distance learning program. The evaluator will used survey and interview to examine the instructor and TAs readiness and competence of teaching online, which will be used to compare with students satisfactory rates.

2. Purposive sampling (only 1 instructor and 2 TAs are in one course)

2. Qualitative and Quantitative

Page 33 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen 3. Measures of Outcomes Students perceptions: -motivation, readiness and preparedness to delivery method, community sense and interaction Faculty and TAs Training, professional development 4. Measures of Program Aims 5. Measures of Policy Course Standard Evaluation Procedures 6.Measures of Technology Access 3. Purposive sampling, to ensure that different levels of achievement of students to be included.

2012

3. Qualitative and Quantitative

Purposive sampling

Qualitative data will be resulted from survey, observation and focus group, by which to identify whether what outcomes the program have achieved.

4. Purposive sampling 5. Purposive sampling

4. Qualitative 5. Qualitative

Qualitative data will be used to identify whether the programs vision, management and resource meet the needs of the participants

6. Purposive sampling

6. Qualitative and Quantitative

Page 34 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

Appendix9:

Data Collection Plan: _Online Distance Learning Program at Dedman School of Hospitality

___
Question 2: To what extent does the online course meet the program goals and objectives? Data 1. Independently interview with the course instructor 1, TAs 2 and administrative group 3 of the program 2. Focus group with 1 instructor, and 2 TAs , 2 administrators 3. Document Review FSU online distance learning program missions statement DSH ODL Program goal statement DSH ODL Program staff action plans DSH ODL Program history Findings of the first evaluation questions Sampling Method 1. Purposive sampling. Quantitative or Qualitative 1. Quantitative Analysis Quantitative data will be used to identify the perceptions of all stakeholders, to find out whether there are existing or potential conflicts. The evaluator will used survey results to generate questions in the focus group, which correlate to the documents and the findings resulting from the first evaluation questions.

2. Purposive sampling 3. Purposive sampling

2. Qualitative 3. Qualitative

Page 35 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

Appendix10:

Data Collection Plan: _Online Distance Learning Program at Dedman School of Hospitality

___
Question 3: Should the program continue to develop new online courses or not? ? Data 1. Interview with all instructor 14, TAs 20 and administrative group 3 of the program 2. Focus group with 5 instructor, and 3 TAs , 3 administrators 3. DSH ODL Program history Sampling Method 1. Purposive sampling. Quantitative or Qualitative 1. Qualitative Analysis Quantitative data will be used to help stakeholders for decision making whether continue or terminate the program. The evaluator will summarize all the data and findings to offer the stakeholders with recommendation.

2. Purposive sampling

2. Qualitative

3. Purposive sampling

3. Qualitative

4. Findings of the first and second evaluation questions

4. Quantitative and Qualitative

Page 36 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen Appendix11:

2012

Example of Instrumentation

for: Online Distance Learning Program at Dedman School of Hospitality

Question1: What is the quality of the online course Ales, Lagers & International Culture of the ODL Program? Survey
Online Distance Learning Program at Dedman School of Hospitality

Evaluation of Quality of the Online Course Ales, Lagers & International Culture
Purpose of this survey
Your answers to the following questions will help identify to what extent this online course have achieved its objectives. It is an anonymous survey, the data collected from which will be confidential. Your responses are very important and will directly impact the final conclusion.

Student information 1. What is your current GPA? <2.0 2.0-2.5 2.6-3.0 3.0-3.5 3.6-4.0 2. Have you participated in an web-based-course before? Yes No 3. Have you used Blackboard before? Yes No 4. How do you rate your knowledge of using the internet? Negative 1 2 3 4 Positive 5 Learning Evaluation 1. The following have been valuable to you in your learning experience Strongly Degree Powerpoint Slides Not Used 1 2 3 Audio to accompany the slides Not Used 1 2 3 Script to accompany the slides Not Used 1 2 3 Discussion board questions Not Used 1 2 3 Case Studies Not Used 1 2 3 Group Project Not Used 1 2 3 Strongly Agree 5 5 5 5 5 5

4 4 4 4 4 4

Page 37 of 40

EDF 5461: Final Evaluation ProposalMeizhen Chen

2012

2. Please evaluate the course material section of the course Strongly Degree Materials are well organized 1 2 Materials are effectively 1 2 presented Materials are the right length 1 2 Materials are clearly written 1 2

3 3 3 3

4 4 4 4

Strongly Agree 5 5 5 5

3. Compared to the traditional classroom format, the web-based delivery of the course better enable you to: Strongly Strongly Degree Agree Be actively involved in the 1 2 3 4 5 learning process Address my questions and 1 2 3 4 5 concerns Stimulate my interest in the 1 2 3 4 5 subject Control when and where to 1 2 3 4 5 learn Learn the material in less 1 2 3 4 5 time Complete the assignments in 1 2 3 4 5 less time 4. 5. 6. 7. 8. What do you like the most about the web-based format of the course? What do you like the least about the web-based format of the course? What elements of the subject have you found most difficult to master on the web? How could the instructor make these subjects more easily understandable on the web? Do you have any other comments, questions or feedback?

Most of the Items cited from :

Holsapple, C., & Lee-Post, A. (2009). How to Design, Develop, and Deliver Successful E-Learning Initiatives. Student Satisfaction and Learning Outcomes in E-Learning: An Introduction to Empirical Research, 9, 195-229.

Page 38 of 40