CHIRON implications for UK HE

Paul Bacsich This public update of an earlier short heads-up paper looks at the CHIRON methodology from EDEN and partners and analyses its relevance to UK HE activities in benchmarking e-learning. CHIRON was the first of the European benchmarking e-learning methodologies (other than the ESMU work with which OBHE is closely in touch) to hit our shores. More have now arrived, in particular E-xcellence from EADTU. The structure below has been used for other reports in this vein.

1.

What is CHIRON?

The following is taken from http://www.eden-online.org/eden.php?menuId=87&contentId=302: CHIRON Referring Innovative Technologies and Solutions for Ubiquitous Learning (Leonardo da Vinci Programme Runtime: 2004-2006) The project aim is to develop reference material presenting and analysing research outcomes, experiments and best practice solutions for new forms of e-learning, based on integration of broadband web-, digital TV- and mobile technologies for ubiquitous applications in the sector of non-formal and informal life-long learning. Ubiquitous learning gives new opportunities for non-formal and informal vocational training of different users groups due to: richer access to vocational training facilities anywhere, anytime, in accordance with increasing occupational and geographical mobility of European citizens; further development of tendency to ensure personalised training on demand; improvement of Internet-based structures for vocational e-training, facilitating also the combinations of formal and nonformal learning to serve the rapidly changing competence needs in the Knowledge society. There is a project web site at http://semioweb.msh-paris.fr/chiron/.

2.

Who are the UK HEIs beholden to EDEN?

The UK members of EDEN are noted in the left-hand column below. The right-hand column gives information on the HEIs interest in benchmarking. There are only two members in Phase 1. At first sight, this seems a much less potent contingent than for E-xcellence (see companion paper).
HEI City University Dundee University Henley Management College Sheffield Hallam University The Open University Phase 1 (was to be) Known to be working on benchmarking e-learning with University of Sydney. Also has partnership with Manchester who are in Benchmarking Pilot. Member of Scottish Benchmarking Club Phase 1 Pilot Interest in benchmarking Made a presentation not unrelated to benchmarking at European WebCT conference Member of Scottish Benchmarking Club

University of the Arts University of Glasgow The University of Hull University of Leicester University of London (External Programme) Paul Bacsich 1 file CHIRON-rel-2

CHIRON implications for UK HE

3.

The CHIRON scheme

This is described in detail in the Appendix. Its provenance is best described as motley. Several criteria come verbatim (including US spellings) from Quality on the Line. See the questionnaire at http://chiron.eduweb.hu/T6.2/index.html.

4.

Conclusions

Since the first draft of this report was written in June 2006, little has been heard of CHIRON even at European events. The claque for it in the UK is likely to be small and unlikely to grow since it has a very mixed provenance, far too many (sub-)criteria and a strong orientation to quality tick-boxes. However, given the current alleged interest in moving benchmarking e-learning down from institution to department and perhaps even course level, with an increased focus on pedagogy, the CHIRON scheme is of value to consider, even if the main conclusion is that such a change in benchmarking focus is rather harder than it might seem.

Appendix
The CHIRON scheme is used in a rather unusual way in their online questionnaire but underneath is a conventional scheme with criteria and good practice indicators. There are 11 criteria. (The code X for Chi is used for CHIRON.) There are a frighteningly large number 216 of indicators. In the list that follows, the indicators in italics come from Quality on the Line.1 The code in parentheses after such an indicator is the Quality on the Line reference.

X01

Goals and Objectives of the course (12 indicators)

(See http://chiron.eduweb.hu/T6.2/cr_1.htm. The other criteria have similar URLs.) 1. Goals and objectives cover course content and are related to the program of study. 2. They are relevant to the subject matter and to the real world in which the content may be applied. 3. Objectives specify learning outcomes related to knowledge, skills, competencies, behaviours, and/or attitudes. 4. Appropriate action verbs are used in goals and objectives. The accomplishment of objectives should be measurable; therefore, vague words such as understood and realize are not used. 5. Objectives state clearly and concisely what must be done. 6. Different levels of outcomes are stated, including those that call for critical thinking and problem-solving skills. 7. Goals and objectives provide a discernable sequence of expectations. 8. The target group and prerequisites are clearly defined 9. The learning goals are clearly defined and easily measured 10. The course study time is adequate to the course content 11. The content has a proper structure and contains clear interrelations 12. The course structure is presented in a clear and logical manner, is flexible and modular and its organization contributes to achieving the course goals View: All of these seem reasonable but are not much to do with benchmarking.
1

See http://www.ihep.com/Pubs/PDF/Quality.pdf. 2 Release 2, 22 December 2006

Paul Bacsich

CHIRON implications for UK HE

X02

Institutional Support (14 indicators)

1. A documented technology plan that includes electronic security measures (i.e., password protection, encryption, back-up systems) is in place and operational to ensure both quality standards and the integrity and validity of information. (Q01) 2. The reliability of the technology delivery system is as failsafe as possible. (Q02) 3. A centralized system provides support for building and maintaining the distance education infrastructure. (Q03) 4. How Administrative and Management aspects are managed by the course customer. How the educational activities are designed and managed. [This is a question, not a sub-criterion.] 5. Costs can be classified into: fixed costs (technological infrastructure, educational material, administrative management), and variable costs (depending on the number learners: tutor fees, individual learner costs, etc.). 6. Pedagogical effectiveness of the system related to learning objectives, easy and friendly navigation, easy and friendly use of the system functions either for teachers or for learners, environment customization (personal home pages, etc). Compatibility with international standards for education (AICC/SCORM, IMS), in order to allow: 7. management of educational content (design and update e-learning interactive courses by defining rules and course schedules); 8. interaction/animation (by activating synchronous and asynchronous activities with the tutor: forums, chat, virtual classes, etc..); 9. management of collaboration activities (applications sharing, virtual blackboard, virtual classroom, etc); 10. management of messaging activities (private mail, public mail, mailing list, etc.); 11. Management of Learner's profile; 12. Tracking of Learner's activities; 13. Scheduling and Management of e-learning activities. 14. Privacy of personal data assured. View: All of these after number 3 seem reasonable but are not much to do with benchmarking and several have a work in progress feel.

X03

Course Development (50 indicators)

1. Guidelines regarding minimum standards are used for course development, design, and delivery, while learning outcomesnot the availability of existing technologydetermine the technology being used to deliver course content. (Q04) [Notice the US m-rules here: ] 2. Instructional materials are reviewed periodically to ensure they meet program standards. (Q05) 3. Courses are designed to require students to engage themselves in analysis, synthesis, and evaluation as part of their course and program requirements. 4. If the course is intended for an international audience, the level of reading proficiency (such as a TOEFL score) necessary to complete the course successfully in the language in which it is written is specified. 5. A brief description of the course includes a goal and learning objectives and/or outcomes. 6. A program map and/or rationale indicates how this course is related to other courses in the program and the credit value of each course.
Paul Bacsich 3 Release 2, 22 December 2006

CHIRON implications for UK HE

7. Prerequisites and/or corequisites are identified. 8. A list of required and recommended resources includes all textbooks, courseware, and online resources necessary to complete the course. If learners must access online databases, instructions are provided for locating and accessing these resources, including password information. If the materials are located in a library, learners are told whether the items are on reserve and, if so, for how long. 9. Special requirements are identified, such as recommended modem speed or Internet bandwidth, hardware (computing speed and storage capacity), software (including an e-mail program adequate for handling assignments and other attachments), and plug-ins. Online sources for these items are provided. 10. The estimated time required to complete the module and/or course is stated. If feasible, a timeline is provided that outlines dates by which specific activities must be completed. 11. Guidelines for participating in online discussions, also referred to as netiquette, are provided, as well as suggestions for handling incoming e-mail, e-mail attachments, viruses, and e-mail filters. 12. Learners are informed about group-work activities, the guidelines for forming groups, grading criteria for group participation and assignments, and their responsibilities as group members. 13. The introduction to the course takes into account the learners' backgrounds, ability levels, and expectations, including their personal learning goals and objectives, or specifies the attributes of the learners for whom the course is designed. 14. The availability of technical support is stated, and links to online technical information are provided. The hours during which technical support is available are clearly identified, including the time zone. 15. The availability of the instructor is specified, as is the turnaround time for responses to learners' questions. 16. Learners are directed to a source for answers to frequently asked questions pertaining to online learning. This source may provide information covering many of the items listed above, as well as items related to plagiarism, virus protection, and firewalls. 17. Learners are informed about their right to privacy and the conditions under which their names or online submissions may be shared with others. 18. Learners are directed to a Web site that helps them determine whether online education is appropriate for them. 19. The developers and reviewers of the course are listed. Brief biographies may be provided to assure students of the developers' knowledge and expertise. 20. A copyright statement or disclaimer clearly identifies the owner(s) of the course and the source(s) of the material students are about to use. 21. Illustrations, photographs, animations, and other forms of multimedia are used to present facts and reinforce concepts. Note that if multimedia material is built into a course, learners may need advanced computer hardware and software, as well as a high-speed Internet connection. If receiving multimedia on a CD-ROM or DVD is an option, this must be stated, as well as any additional costs involved. 22. Learners can select activities that are relevant for them, as the activities have been designed with their ages, cultural backgrounds, and experiential needs in mind. 23. Learners can proceed at a pace that is appropriate for them and can repeat sections as often as they need to.
Paul Bacsich 4 Release 2, 22 December 2006

CHIRON implications for UK HE

24. Activities engage and motivate the learners. Learners must frequently respond to questions, select options, provide information, or contact others. 25. Activities and materials are presented sequentially in order of difficulty. 26. Learners are encouraged to interact with others and benefit from their experience and professional expertise. 27. Learners are linked to resources beyond the course material. 28. Activities are sequenced logically, as in a hierarchical or chronological manner. 29. Frequent opportunities are provided for practice and knowledge transfer. 30. Activities are realistic and appropriate and can be performed with the resources and time available to the learners. 31. Summaries are provided throughout the material, particularly at the end of topics, lessons, and modules. 32. The layout is appropriate for the content and intended audience. 33. Navigational icons or cues are used consistently. 34. The typeface is appropriate for the content and common to all programs and computers, such as Times Roman. 35. Bold-face type is used sparingly, to highlight important terms, for example. 36. Capital letters and underlining are not used for emphasis. Underlining is used only for hyperlinks. 37. Key words are highlighted, especially when they are first used. 38. Headings and subheadings are used to organize content. 39. The format is uncluttered and includes white space. 40. Ragged right margins are used or letters are kerned (evenly spaced on the line). 41. The contrast between text and the background material makes the text legible. 42. Colour is used effectively. 43. Graphic elements such as diagrams, tables, and photographs illustrate or clarify information presented in the text. 44. Illustrations can be seen easily on a computer screen, and JPEG files are used to accommodate different download speeds. 45. Text explaining a graphic is aligned with the non-textual material. 46. To assist those who may use screen readers, a description of each illustration is available as an image tag. 47. Frames are used effectively to provide a consistent format. 48. The material is displayed attractively. 49. The selected pedagogical model depends on the educational methodology and the learning objectives. 50. The educational multimedia material indexed and loaded by the Learning Management System should be shared and reused. View: All of these after number 2 are not much to do with benchmarking but there are some useful design and layout guidelines.

Paul Bacsich

Release 2, 22 December 2006

CHIRON implications for UK HE

X04

Course Structure (12 indicators)

1. Before starting an online program, students are advised about the program to determine (1) if they possess the self-motivation and commitment to learn at a distance and (2) if they have access to the minimal technology required by the course design. (Q10) 2. Students are provided with supplemental course information that outlines course objectives, concepts, and ideas, and learning outcomes for each course are summarized in a clearly written, straightforward statement. (Q11) 3. Students have access to sufficient library resources that may include a virtual library accessible through the World Wide Web. (Q12) 4. Faculty and students agree upon expectations regarding times for student assignment completion and faculty response. (Q13) 5. The table of contents gives an accurate indication of how the material is arranged. 6. The organization or sequencing of the content is appropriate for the subject matter and the intended audience. 7. Units of instruction or topics are divided into subunits or subtopics. 8. Subtopics are related to main topics. 9. The organization of components is consistent throughout the course. For example, each module may have the following sections: introduction, objectives, pretest, directions, explanatory text including learning activities such as case studies, suggested answers for learning activities and links to additional information, module summary, self-test, self-test answers, references, additional readings, and module assignment. 10. Required course elements are clearly delineated from supplementary elements. 11. References to other parts of the material are correct. 12. Links to other parts of the course or external sources of information are accurate. View: All of these after number 4 seem reasonable guidelines but are not much to do with benchmarking.

X05

Course Content (25 indicators)

The content is: 1. Directly related to learning objectives 2. Complete, providing all the content or learning experiences needed to achieve the learning objectives 3. Appropriate to the learners' characteristics (ability and maturity level) and experiences 4. Comparable and at least equal in rigour to similar on-campus courses 5. Broken into small, incremental learning steps 6. Presented in a logical sequence 7. Related to other material the learners may have studied or experiences they may have had 8. Illustrated by examples and/or case studies when new information is presented 9. Linked to other sources, with reading assignments clearly specified 10. Learning materials are appropriate for the learners and the subject matter.
Paul Bacsich 6 Release 2, 22 December 2006

CHIRON implications for UK HE

11. Lists of learning resources are divided into required and optional categories. 12. Various learning resources are used to ensure compatibility with learners' different interests, abilities, and learning styles. 13. Multimedia (such as audio and video) clips are included only if the learners have access to the appropriate hardware and software and these items are specified at the beginning of the course. 14. The different media types (text, sound, graphics, etc.) are properly chosen and used, t he format is specified, and a direct link to a required plug-in is provided. 15. Learning resources reflecting different points of view are provided when appropriate, they are accurate, current, and related to the course content. 16. A bibliography or reference list includes a variety of material. 17. Links are provided to material within and external to the course. 18. The learning materials are interesting and motivating 19. The style and the presentation correspond to the apperception and attitude of the learners 20. The learning materials are interactive and reusable 21. The level of complexity of the materials is suitable for the learners 22. The learning materials do not contain race, sexual, age or other injuries [insults?] 23. The learning content is precise, contemporary and free of grammar and meaning mistakes 24. No copyrights are broken 25. The learning content is distributed in similar units which use the same template. View: All of these seem reasonable guidelines but are not much to do with benchmarking.

X06

Teaching/Learning (19 indicators)

1. Student interaction with faculty and other students is an essential characteristic and is facilitated through a variety of ways, including voice-mail and/or e-mail. (Q07) 2. Feedback to student assignments and questions is constructive and provided in a timely manner. (Q08) 3. Students are instructed in the proper methods of effective research, including assessment of the validity of resources. (Q09) 4. The level of interactivity of the learners with the educational content and the platform services. [This is a question not a subcriterion.] 5. The chosen learning strategy is adequate to the content and goals of the course 6. The learning process instructions are easily accessible and clear 7. The learning activities are interesting and stimulate the learning process as a whole 8. The learning activities are practical 9. The learning activities give the opportunities for individual learning 10. The level of complexity of the learning activities is proper 11. The learning activities require adequate study time 12. The homework and self-assessment are interesting and motivating 13. The knowledge assessment methods correspond to the course content
Paul Bacsich 7 Release 2, 22 December 2006

CHIRON implications for UK HE

14. The assessment criteria are clearly defined 15. The knowledge assessment tools are practical enough 16. Feedback is ensured 17. Effective self-assessment abilities are provided 18. Effective communication is provided to the learners and tutors 19. Each one of the learners gets personalized help during the learning process View: All of these after 4 seem reasonable guidelines but are not much to do with benchmarking.

X07

Student Support (18 indicators)

1. Students receive information about programs, including admission requirements, tuition and fees, books and supplies, technical and proctoring requirements, and student support services. (Q14) 2. Students are provided with hands-on training and information to aid them in securing material through electronic databases, interlibrary loans, government archives, news services, and other sources. (Q15) 3. Throughout the duration of the course/program, students have access to technical assistance, including detailed instructions regarding the electronic media used, practice sessions prior to the beginning of the course, and convenient access to technical support staff. (Q16) 4. Questions directed to student service personnel are answered accurately and quickly, with a structured system in place to address student complaints. (Q17) 5. Instructions or directions are clear and concise. 6. Learners are told which activities must be performed synchronously and which may be performed asynchronously. 7. Learners are told whether learning activities are sequential or whether they can be completed in any order. 8. Learners are informed about their own responsibilities in online learning. 9. Expectations are clearly specified for participation in collaborative or team-based learning activities. 10. Procedures for grouping learners for team-based learning activities are specified. 11. Deadlines are specified, and the consequences of missing deadlines are clearly stated. 12. A variety of instructional or learning activities are used to promote interactivity. These may include online discussions, online conferencing, collaborative assignments, and listserv participation. 13. Activities develop appropriate cognitive, affective, and psychomotor skills. 14. Activities encourage critical thinking, creativity, and problem-solving. 15. The number of activities is sufficient to support learning. 16. Constructive, relevant, and frequent feedback is provided to promote clarification, elaboration, and transfer. 17. Create a learning community among all actors involved in the learning process: teacher and learners; learners and learners; tutor and learners. 18. Helpdesk service always available to solve any problems with the use of platform services.

Paul Bacsich

Release 2, 22 December 2006

CHIRON implications for UK HE

View: With the exception of 18 and perhaps 9, most of these after 4 seem reasonable guidelines but are not much to do with benchmarking.

X08

Faculty Support (4 indicators)

1. Technical assistance in course development is available to faculty, who are encouraged to use it. (Q18) 2. Faculty members are assisted in the transition from classroom teaching to online instruction and are assessed during the process. (Q19) 3. Instructor training and assistance, including peer mentoring, continues through the progression of the online course. (Q20) 4. Faculty members are provided with written resources to deal with issues arising from student use of electronically-accessed data. (Q21) View: All of these are benchmarks even if some are quite old-fashioned.

X09

Evaluation and Assessment (24 indicators)

1. The program's educational effectiveness and teaching/learning process is assessed through an evaluation process that uses several methods and applies specific standards. (Q22) 2. Data on enrolment, costs, and successful/ innovative uses of technology are used to evaluate program effectiveness. (Q23) 3. Intended learning outcomes are reviewed regularly to ensure clarity, utility, and appropriateness. (Q24) 4. Learners are given clear expectations and criteria for credit assignments. Examples of assignments that meet the criteria may be included for students to review. 5. The number of assignments and their due dates are reasonable. 6. Evaluation and grading procedures are clear and explicit. 7. Appropriate links to institutional policies on grading and evaluation are provided. 8. The relationships between course learning outcomes, evaluation strategies, and course assignments are evident to the learner. 9. The relationship between individual assignments and the final course grade is clearly specified. 10. Detailed step-by-step instructions are provided for each evaluative exercise. 11. Guidelines for submitting assignments are provided. 12. Students are informed about the criteria that will be used to evaluate their participation in online activities such as discussion groups. 13. A variety of feasible and content-relevant assignments or evaluative exercises are provided. 14. The evaluative exercises are relevant to the learners and the career or profession they may pursue. 15. Learners are able to track and evaluate their own progress. Self-tests are similar to the final evaluation instruments. 16. Criteria and procedures for peer review and evaluation are clearly specified if these elements are included in the course. 17. Learners are informed about the consequences of plagiarism and the failure to properly cite copyrighted material.
Paul Bacsich 9 Release 2, 22 December 2006

CHIRON implications for UK HE

18. Learners are told when they can expect to receive feedback from the instructor. 19. Evaluation procedures are congruent with the objectives and reflect any priorities that have been established for the objectives. 20. Assessment of learners competences and satisfaction. [This is a statement.] 21. Assessment of the Tutors and of the Course. [This is a statement.] 22. Assessment of the transfer of competences in other contexts. [This is a statement.] 23. The learner's assessment should be designed since the beginning of the course and should be scheduled more times during the course. The assessment tests should clearly assess the knowledge and the competences. 24. Definition of standards to certify quality of e-learning [This is a statement.] View: Only the first three are benchmarks.

X10

Accessibility (26 indicators)

1. The function of each icon or button is explained and/or is naturally evident to the learners. 2. A detailed table of contents includes objectives, learning outcomes, or topics. 3. Every section of the course or module begins with a preview. 4. Every page is linked to the previous page, the start of the module, the beginning of the course, and to e-mail so that learners may contact instructors and other learners for clarification and discussion. 5. Links within the course are provided to other parts of the course. 6. Page headers or footers identify where the learner is in the course. 7. An index lists key words or topics. 8. A glossary defines unusual or technical terms used in the course and may provide links to sources of supplementary information. 9. Consideration is given to learners who may have visual or auditory challenges. 10. Space and Time Accessibility of the system. [This appears to be a heading but is reproduced here exactly as it is in the web page.] 11. The interface is intuitive and user-friendly 12. The site organization is logical and understandable 13. The site provides convenient help system 14. Easy search mechanism 15. Quick information loading 16. The graphical interface corresponds to the content and navigation 17. Convenient navigation 18. The text is sharp and easy to read 19. Compatible with different browsers and web content standards 20. Suitable for different resolution, colour schemes and connection speed 21. The site contain administrative module 22. High level of security

Paul Bacsich

10

Release 2, 22 December 2006

CHIRON implications for UK HE

23. Reliability 24. Flexible mechanism for extending and improvement 25. Suitable personalization level 26. No copyrights are broken View: These are useful guidelines but not benchmarks.

X11

Language (12 indicators)

1. The writing style is clear and direct; spelling and grammar are consistent and accurate. 2. Clear directions are given. 3. The tone of the writing is supportive and encouraging. 4. A conversational tone employs the second person: you, not the learner. 5. Verbs are in the active, not passive, voice; for example, Maslow developed the theory, not the theory was developed by Maslow. 6. The sentences are short and the paragraphs are brief; familiar and/or common words and terms are used, consistently. 7. Numbers are used to identify sequential steps in a task or process. 8. Bullets are used to list items that are not prioritized or sequential. 9. Abbreviations and symbols are defined. 10. Bullets, dashes, and numbers are used consistently. 11. Instructions are stated simply and are easy to understand. 12. The writing conveys no explicit or implicit bias relative to age, culture or ethnicity, race, gender, or sexual preference. However, biased opinions may be included as examples if they are relevant to the course content. View: These are useful guidelines but not benchmarks.

Paul Bacsich

11

Release 2, 22 December 2006