Helen Miller Preliminary Formative & Summative Evaluation Plan EDTECH 512 Fall 2010

Evaluation Criteria Effectiveness: mastery of goals and success of WBI Efficiency: Delivered in a timely or cost-saving manner Appeal: Gain and maintain learner attention and interest Ease of use

Explanation Do the students have solid understanding Typography Investigate time required in using the Web delivery Review video, content, text and tutorials for interest. Review use of technology for ease of use.

Data Sources Quizzes & tests Discussions Projects Document time it took Instructor for preperation, facilitation, and feedback, Student opinions about coursework and content. Expert reviews of course. Instructor and student opinions regarding ease of use.

Formative Evaluation Evaluation Criteria & Categories EFFECTIVENESS Goals Sample Questions Are the goals accurate? Are the goals & objectives clear and achievable? Are the goals and content appropriate for Web Based Delivery Is the content relevant to the course objectives? Do discussions work well for critiques of learning activities? Do activities promote learning? Do the technology applications function properly? Do students have good access to Data Sources Expert Reviews (SME, ID) Test scores Grades End-user reviews surveys

Content

Expert Reviews (SME, Instructor, ID) End-user reviews End-user surveys Discussions Expert Reviews (Instructor, Technical, ID) Observation

Technology

 Message Design EFFICIENCY Goals

computers and internet? Is the website appropriate for the learners? Does the course reflect good use of typography? Are graphics used to promote learning? Do they detract from learning? Are instructions and expectations clear? Are icons used appropriately to aid learner in the use of the website? Are goals and objectives stated clearly? Do the activities aid in achieving the goals? Is the content relevant? Is the content dated? Do the technology applications work effectively? Is the organization of the coursework clear? Are students and instructor able to work asynchronously? Are the activities in line with the goals and objectives of the course? Is the content interesting and engaging? Do students and instructor have access to course materials when needed? Are online tutorials effective? Is there tech support available? Is the course free of grammatical and typographical errors? Is the course easy to navigate?

Surveys Expert Reviews (SME, ID, instructor) End-user reviews surveys

Expert Reviews (SME, ID) End-user survey Expert Reviews (SME, Instructor, ID) End-user survey Expert Reviews (SME, technical, ID) End-user survey Observation Expert Reviews (SME, ID) Survey End-user reviews

Content Technology

Message Design

APPEAL Goals Content

Expert Reviews (SME, ID) End-user survey Expert Reviews (SME, ID, Instructor) End-user survey Expert Reviews (SME, technical, ID) Observation End-user survey & reviews

Technology

Message Design

Do supporting graphics aid in learning? Does the design of website set a good example of design for budding designers? Is typography used creatively and appropriately?

Expert Reviews (SME, instructor, technical, ID) End-user surveys

Who are the stakeholders? Primary Stakeholders: Instructor: Helen Miller. She is responsible for the successful implementation of the course and for its content. Designer: Helen Miller. She is responsible for the design of the course and its goals and objectives. Students: Students participation will be integral in the implementation and success of the course. Secondary Stakeholders: Tech Support Team: They will be called on to assist students, instructor and designer for technical assistance when needed. Graphic Design Department Head: They will observe to see if the course is in line with program objectives. What is being evaluated? Course Design Materials to be evaluated Goals and objectives Assessment items Organization of course Instructional strategies Instructional materials Relevance of instructional content Availability of content Appropriateness of content Learning activities Navigation of course Ease of use Availability (is the site down a lot?)

Course Content

Mode of Delivery (Moodle)

Who are the evaluators? Evaluator/Designer: Helen Miller , Currently a candidate for a Masters in Education Technology as well as a Graduate Certificate in Online Course

Design. Is designing this course and has six years professional experience working as a graphic designer. Evaluator/ Instructor: Department Head, Has M.F.A. and 10+ years experience teaching Graphic Design and Art. Is responsible for setting the curriculum standard in the department. Expert Reviewer/Technical Support: IT Department Head. Experienced and seasoned professional in information technology. Has 10-15 years experience facilitating web-based applications, oversees all technical support for online campus. Expert Reviewer/Content: Dr. Diane Hall is a professor of Web-Base End-User Reviewer/Students: Students will be surveyed at the completion of course. They will be asked review the course design, the instructor, the technology and their overall learning experience.

Evaluation Methods and Schedule Formative data will be collected through the entire design, development and implementation process. This data collected will be used to improve the web-based instruction of the typography course. After the initial evaluation by SME, ID, and technical, the evaluation will consist of three one-to-one tryouts of students and instructors. These tryouts will give the ID different points of view as to what is and isnt working for instructors and students. Once major changes have been made the course will go into a field trial where five to ten students will participate in the course. At the conclusion of the field trial all major goal, technical, content and design issues will worked out and the course will be implemented. One-to-one and Field trial participants/volunteers will be current design students who will be recruited by the use of a letter distributed by the art department. They will sign off on a consent form to use the collected data from the study. Ongoing Decisions The formative evaluation will occur on a continuous basis as data is collected. The ID and SME will review surveys, notes, checklists, etc. and meet with department head on a regular basis to make decisions regarding what should be changed, improved or removed from course. Participants course projects and test scores will be thoroughly reviewed and compared with that of on-campus students during one-to-one and field trials. Preliminary Summative Evaluation Evaluation Criteria Effectiveness Main Questions How did the final product Data Sources Performance activities

 Efficiency Appeal

meet the course objectives? Do learners believe the WBI was worth attending? How long did learner participate in WBI per module? Will learners continue to take online design courses through the school?

(tests, and projects) End-user surveys and opinions Learner login and logout times. Stakeholder surveys.

The summative evaluation will occur at the end of the initial implementation. The evaluation will look at how the course fulfills the learning objectives and standards. This will be determined by looking at class project and test scores. Evaluating enduser and stakeholder surveys will help to assess the efficiency and appeal of the course.