EDTECH522- Online Course Evaluation Project Jeff Toorongian, Summer 2012 Standards Criteria Course: Introduction to Astronomy Score/(Comments)

Course Overview and Introduction 1.1- Instructions 3/3 (Video and make clear how to get syllabus) started and where to find various course components 1.2- Students are introduced to the purpose and structure of the course 1.3- Etiquette expectations (sometimes called netiquette) for online discussions, email, and other forms of communication are stated clearly 1.4- Course and/or institutional policies with which the student is expected to comply are clearly stated, or a link to current policies is provided 1.5- Prerequisite knowledge in the discipline and/ or any required competencies are clearly stated 3/3 (On video in first lecture titled policies) Course: Finite Math Online Score/(Comments) 3/3 (Getting started page)

2/3 (Some information in the news area, but difficult to scroll through to find it)

1/2 (Some info related to discussions, but no netiquette discussion)

1/2 (Info for email, but no netiquette info.)

2/2 (Syllabus and video)

2/2 (Policies outlined at several locations)

1/1 (In Policies video)

0/1 (Could not find this info.)

1.6- Minimum technical skills expected of the student are clearly stated 1.7- The selfintroduction by the instructor is appropriate and available online 1.8- Students are asked to introduce themselves to the class

1 /1 (Skills outlined in video)

1/1 (Computer skills and requirements reviewed)

1/1 (Video in lecture #1)

1/1 (Web page intro)

0 /1 (Could not find this requirement) Total: 12/14

0/1 (Could not find this requirement) Total: 10/12

Learning Objectives

2.1- The course learning objectives describe outcomes that are measurable 2.2- The module/unit learning objectives describe outcomes that are measurable and consistent with the course-level objectives 2.3- All learning objectives are stated clearly and written from the students perspective 2.4- Instructions to students on how to meet the learning objectives are adequate and stated clearly

0/3 (not present)

0/3 (not present)

0/3 (not present)

0/3 (not present)

0/3 (not present)

0/3 (not present)

1/3 (only indirectly stated in assignment descriptions)

1/3 (tips and suggestions on Webassign homework and tests are provided, but no objectives)

2.5- The learning objectives are appropriately designed for the level of the course Assessment and Measurement 3.1- The types of assessments selected measure the stated learning objectives and are consistent with course activities and resources

0/3 (objectives not stated directly) Total: 1/15

0/3 (objectives not stated directly) Total: 1/15

2/3 (inconclusive, since no specific learning objectives provided. Assessment are comprehensive and seem to align with topics. Would need a SME to confirm these findings)

2/3 (homework is delivered via web-based WebAssign account. Not able to access these assignments or tests to confirm alignment.

3.2- The course grading policy is stated clearly 3.3- Specific and descriptive criteria are provided for the evaluation of students work and participation and are tied to the course grading policy 3.4- The assessment instruments selected are sequenced, varied, and appropriate to the student work being assessed 3.5- Students have multiple opportunities to measure their own learning progress

3/3 (detailed grading policy provided)

3/3 (detailed grading policy provided)

3/3 (detailed information on course assignments and grading policy is provided)

3/3 (detailed information on course assignments and grading policy is provided)

2/2 (mix of homework (via WebAssign), discussions, presentation, and exams)

2/2 (homework (via WebAssign) and tests)

2/2 (in addition to homework and exams, there are opportunities for group discussion and an individual

1/2 (assessments seem limited to homework and tests)

presentation) Total: 11/13 Total: 12/13 Instructional Materials 4.1- The instructional materials contribute to the achievement of the stated course and module/unit learning objectives 2/3 (Objectives not stated, but materials appear comprehensive) 2/3 (Objectives not stated, but materials appear comprehensive)

3/3 (Good description of 4.2- The purpose of how learning materials instructional materials are to be used is and how the materials provided) are to be used for learning activities are clearly explained 4.3- All resources and materials used in the course are appropriately cited 4.4- The instructional materials are current 1/2 (Not all content showed citations in PowerPoint slides)

3/3 (Good description of how learning materials are to be used is provided)

2/2 (Content primarily the work of instructor, who indicates copyright info on videos) 2/2 (Copyright info on video lectures is 2006, but since this is general math content it can be considered current) 1/1 (Primarily info from the instructor via videos, homework and tests. No text noted for this course)

2/2 (Text is reasonably current, as are other resources)

4.5- The instructional materials present a variety of perspectives on the course content

1/1 (The instructor content, textbook readings, other supplemental links and resources, and student presentations)

4.6- The distinction between required and optional materials is clearly explained

1/1 (Good description or resources and their use) Total: 10/12

1/1 (Good description or resources and their use)

Total: 11/12

Learner Interaction and Engagement

5.1- The learning activities promote the achievement of the stated learning objectives 5.2- Learning activities provide opportunities for interaction that support active learning

2/3 (Difficult to confirm since no specific objectives were provided)

2/3 (Difficult to confirm since no specific objectives were provided)

3/3 (A good mix of online lectures, WebAssign homework, student presentations, and discussions)

1/3 (This appears to be very much a individual course with few opportunities for student to really engage each other with the content)

5.3- The instructors plan for classroom response time and feedback on assignments is clearly stated 5.4- The requirements for student interaction are clearly articulated

3/3 (Well described policy for instructor response time and feedback in syllabus and first lecture on course policies) 2/2 (Detailed description provided for student expectations) Total: 10/11

3/3 (Comprehensive web-based description of instructor feedback and communication policies)

2/2 (Limited student interaction is required, but policies are stated)

Total: 8/11 3/3 (No specific learning objective are provided. Course tools and media are comprehensive and provide good content coverage, but would need a SME to confirm)

Course Technology

6.1- The tools and media support the course learning objectives

3/3 (No specific learning objective are provided. Course tools and media are comprehensive and provide good content coverage, but would need a SME to confirm) 3/3 (Opportunities for student engagement with peers via discussion and with content via topic presentations)

6.2- Course tools and media support student engagement and guide the student to become an active learner

2/3 (Course is primarily video lecture based with WebAssign homework. No active learning activities evident)

6.3- Navigation throughout the online components of the course is logical, consistent, and efficient

3/3 (Most navigation is provided within the LMS. Good organization of materials in modules.

1/3 (Course is entirely web-based from what appears to be custom html pages. Some confusion with links in top banner. News tool uses Tumblr, and is a confusing display of new items that are difficult to navigate through. 2/2 (Access to course content appears to be easily accomplished. Does require Quicktime plug-in to view video lectures)

6.4- Students can readily access the technologies required in the course

2/2 (Access to course content appears to be easily accomplished. Does require Silverlight plug-in to view video lectures)

6.5- The course technologies are current

1/1 (Modern Canvas LMS, Silverlight-based video, and WebAssign)

Total: 12/12 Learner Support 7.1- The course instructions articulate or link to a clear description of the technical support offered and how to access it 7.2- Course instructions articulate or link to the institutions accessibility policies and services 7.3- Course instructions 3/3 (Adequate information provided on tech support options for learners)

1/1 (Not cutting edge technology. WebAssign is current, but basic web pages. Quicktime-based video is current) Total: 9/12 3/3 (Adequate information provided on tech support options for learners)

2/3 (No link provided, but information is reviewed)

3/3 (Link and discussion of policy and services available is provided)

1/2 (No specific services are discussed,

2/2 (Support resources and instructor-based

articulate or link to an explanation of how the institutions academic support services and resources can help students succeed in the course and how students can access the services 7.4- Course instructions articulate or link to an explanation of how the institutions student support services can help students succeed and how students can access the services Accessibility 8.1- The course employs accessible technologies and provides guidance on how to obtain accommodation

but instructor-based support is mentioned)

support are both reviewed)

0/1 (No link or mention evident)

0/1 (No link or mention evident)

Total: 6/9 Total: 8/9

2/3 (Modern LMS supports web-assistive technologies via modern browsers. No captioning for video lectures)

2/3 (Basic web-page format would support assistive browser supports. No captioning available for Quicktimebased videos. 0/2 (No alternatives specifically presented)

8.2- The course contains equivalent alternatives to auditory and visual content

0/2 (No alternatives specifically presented)

8.3- The course 2/2 (Clean design design facilitates present within LMS) readability and minimizes distractions

1/2 (Web design has some confusion navigation and multicolumn approach is difficult to read)

8.4- The course design

0/2 (No specific

0/2 (No specific assistive technologies were noted)

accommodates the use of assistive technologies

assistive technologies were noted) Total: 3/9 Total: 4/9 Course Total 67/95 Course Total: 61/95

Reflections of Evaluation Project: This course evaluation project was a valuable exercise. I chose to evaluate an online course from my home institution called PH1600, Introductory Astronomy, and the Finite Math Online course that was provided in the project description. I used the Quality Matters Rubric Standards from the 2011-2013 edition. Eight broad standards categories were provided to evaluate the courses against. I knew going in that a course evaluation could be time consuming and that was confirmed in this project. To provide a thorough and fair evaluation I needed to engage the course design and navigate to all areas that I was able to access. I also made sure I was able to view the video lecture components that comprised the heart of each course. I quickly discovered that each course did not provide clear, distinct, learning objectives. This quickly made the evaluation process difficult because the Quality Matters Rubric draws upon learning objectives as the basis for many evaluation criteria. Without these I relied upon information in the syllabus or course descriptions to help assess criteria. Through this process it became clear to me that one of the important areas needing evaluation in an online course is the syllabus or course description. Just as in face to face courses, this document is extremely important (perhaps even more so in the online environment) for providing the information students need to form a clear understanding of course policies, requirements, and assessments. Though not formally described as the syllabus in the case of the Finite Math course, the course description, or syllabus, was the document I referred to frequently to provide the information needed to evaluate course criteria against. An evaluation of course content (videos and assessments in this project) was also critical, since these materials were the foundation of each course. I tried to review segments of several videos scattered across the curriculum for each course. I checked the videos for technical issues, such a playback, required web resources, and technical quality. I assessed other course assignments against the learner engagement standards of the rubric. While the Introductory Astronomy course did have learner engagement activities included, such as a modern astronomy presentation and discussion posts, I found the Finite Math course to lack engagement and collaboration opportunities for students. The Quality Matters rubric provided me an opportunity to conduct a thorough review of each online course and alerted me to aspects of a course that I did not previously consider. Although I knew the importance of clearly stated learning objectives as the basis of good course design, I did not appreciate the importance of presenting these objectives directly to the learner. Since neither of my courses provided these objectives, it made a valid evaluation difficult to conduct. I found I had to fill in the blanks and make assumptions with the other course materials that were available. I also realized, even more strongly than before, the importance of building in opportunities for student engagement and collaboration on meaningful learning activities.

One of the dangers of an online course is establishing an environment where the students are islands in an online sea, with no connections or opportunities to share ideas, talents, feedback or support. Further course designs that I conduct will include a course rubric as a reference at the initial planning stages. Its not necessary that each rubric criteria is checked off, and it doesnt mean that a course is necessarily deficient if some standards are not fully realized. The rubric is a valuable tool, however, for both instructional designers and instructors to evaluate their course design work and measure it against accepted standards in the field.