EENOVATE – A New Agenda for Universities in Candidate Countries

a Minerva project

Research Methodology (proposal)

Draft 23 June 2005

This document provides the methodological basis to be use by EENOVATE project team for an action-
research approach in evaluating models for client-focused University elearning services. It also
presents the elearning concept map developed by the RO1, RO2, UK1, ES1, BG1, IT1.

The methodological framework of the evaluation research on the quality of academic elearning
services is based on:
• the theories and practice of evaluation research, and especially those applicable to distance
education;
• EENOVATE project design and the development process;
• the context in which the project is implemented and could be replicated.
An empowerment evaluation approach was adopted, in order to allow an active implication of the
project actors and stakeholders within the evaluation research design and implementation phases.
The context reflects the political, social, economical, and technological media within which the
research and the whole project are developed. Various contexts magnify or diminish the research
findings and their value. In this perspective, we consider important to assign concepts as innovation,
change and evolution a significance that is relative to local/ regional framework – what is considered
innovative in a particular context could be a current practice in another one.

Designing the present research evaluation plan, the following ideas were taken into account:
− − The actors involved in the project share the same professional backgrounds (also working in
academic elearning environments) and are used to working in transnational projects, though
coming from different cultures. Still, because of the context the higher education institutions are
functioning in, the evaluation research tools will be tailored to each partner’s peculiar needs and
experiences. There were established two “areas” to be surveyed: two countries from the EU
member states (Western Europe) and two countries from candidate countries (Eastern Europe).
There will be developed a common analysis model of elearning programmes, but with two sets of
research instruments. Common basis for data gathering would help in building an integrated up-to-
date model for client focused university elearning services.
− − Evaluation research will refer to the academic elearning practices, focusing on a variety of
activities, interactions, procedures, products, and performances. Thus, it will not only be a
“summative” exercise at the end of an elearning programme of higher education institutions, but it
will also involve “in-the-making” dynamics, work styles, and reactions (– partly due to the
involvement of the researchers in academic elearning programmes).
− − Evaluation is an on going process, made of “formal” procedures as well as of “informal”
interactions, and unpredictable contingencies. It would be restrictive to consider the present
research evaluation plan complete and non-changeable. On the contrary, it contains the main
trends to be followed, leaving a margin of manoeuvre in order to have a hold on unmanageable
complexity.
Component Focus on…
Purpose of the evaluation What is the main purpose of, and who are the main audiences for
research the evaluation research?
The overall purpose is the development and validation of an up-to-date model for client-
focused academic elearning services.
Scope of the evaluation What is the “object” of the evaluation research? What should be
research evaluated?
There will be evaluated the [quality of] elearning programmes in 10 universities from
Candidate Countries (Romania and Bulgaria) and 6 universities from EU member states (UK
and Spain.
Target group What is the target group?

Teachers/ tutors, students, elearning programme coordinators.

Stakeholders What main stakeholders are involved or implicated in the project

and its evaluation?
Policy makers, professionals in higher education institutions, elearning practitioners,
educational software developers, researchers, students.
Type of research What kind of research is to be developed?

Evaluation research; action research; correlation research, ex post facto research.

Research methods, techniques, What kind of research methods, techniques and instruments will be
instruments used?
Survey. Semi-structured interview, questionnaire, focus group. Self-administered
questionnaire, operator-led questionnaire, interview guides, focus group guides.
Utilisation of the evaluation How will the evaluation main findings be used and disseminated
results and what activities will be carried out to facilitate use?
The final report will be geared to the information needs of policy makers, professionals in
higher education institutions, elearning practitioners, educational software developers,
researchers. The report, together with a synthesis, will be disseminated directly in universities
through national education networks.
Work share Who will carry out the evaluation research and with what
resources?
Project partners.

Agreement on working definition of key concepts

BG:

E – learning is:

- education via the Internet, network, or standalone computer;

- component of distributed learning that includes digital content;

- experienced through a technology interface, and is Internet-enabled;

- essentially the network-enabled transfer of skills and knowledge;

- it refers to using electronic applications and processes to learn;

- its applications and processes include Web-based learning, computer-based learning, virtual
classrooms and digital collaboration;

- content is delivered via the Internet, intranet/extranet, audio or videotape, satellite TV,
interactive TV and CD-ROM;

- it encompasses wireless and mobile learning as well /mobile phones, smart phones, PDA
devices, mobile computers/. Some sources define Knowledge Management as a form of e-
learning.

- it encompasses all learning undertaken, whether formal or informal, through electronic

delivery;

- it is often used in distance learning programs. But it includes any technologically mediated
learning using computers whether from a distance or in face to face classroom setting
(computer assisted learning).

- it focus on putting the student in control.

- there are two quite different forms of teaching by means of a computer: pre-programmed
computer-based learning (CBL) and computer-mediated communications (CMC). With CBL
the student works through pre-designed material, interacting by answering questions
embedded within the materials and choosing options or routes through the learning material.
Within these CBL programs different kinds of media can be used such as audio and video.
(Bates, 1995) Bearing this definition in mind we can determine CMC as a part of e – learning.

RO:

< electronic learning

(1) Largo senso, elearning (or e-learning) stand for any educational situation significantly
using ICT means. The term, taken from Anglo-Saxon literature, was extended from the
primary, etymological sense of learning through electronic medium, covering nowadays the
entire domain of educational interventions using modern ICT. Defined like this, more as e-
education, the semantic area of elearning concept is interfering and variably overlapping a
multitude of terms intercepting the variety of didactic experiences which could benefit of
technological support: computer assisted instruction, computer mediated communication,
computer mediated learning, digital/ mobile/ online learning/ education, learning through
multimedia etc.

The concept of educational software covers a large area of electronic materials/ applications
developed to simplify/ support educational process: maps, dictionaries, encyclopaedias,
didactic movies, presentations in various formats, e-books, tests, tutorials, simulations,
didactic games, elearning platforms, elearning solutions etc. The computer and electronic/
multimedia materials are used to support teaching, learning, evaluation or as a mean of
communication (to deliver/ receive individual learning tasks etc.).
(2) In a more restricted sense, elearning represents a type of distance education, as a planed
teaching/ learning experience organised by an institution, which delivers learning materials in
a sequential and systematised order to be achieved by students in their own rhythm. The
delivery is done through the new communication and information technologies, especially
through the Internet (online learning/ education) that represents both the distribution medium
and the communication channel between the actors involved. Being used now in higher
education and for continuous training, the online education systems imitate and adapt
traditional (face-to-face) education components: planning, specific content and methodology,
interaction, support and evaluation.

The extensions brought by technological medium, insufficiently explored and used, refer to
(claim for):

(a) a student centred approach, through personalised learning path – different ways to
compose learning objects taking into account beneficiaries needs -, through individualisation
of training – non-linear structure of the content, student autonomy and independence,
asynchronous seminars.

(b) distributed resources – by using/ integrating/ accessing electronic libraries and multimedia
materials, by involving top specialists in students discussions.

(c ) interchangeable roles (Symmetric Knowledge Advancement – Scardamalia, 1995) –

continuous restructure of learning teams

Principles of quality in elearning

(Chickering& Ehrmann, 1987)

1. Good distance learning encourages and maximizes contacts between learners and instructors.
Distance learning does not minimize or obstruct contact between learners and teachers, in spite of the
fact that face-to-face meetings may never take place and the number of "contact hours" is reduced or
eliminated. Communication between students and instructor is important for enhancing learning as
well as for student motivation and involvement. Technology that increases access through
asynchronous or synchronous means is a necessary component of effective distance learning.
2. Good distance learning develops relationships and promotes collaboration among students.
Learning at its best is as much a collaborative activity as an independent one. Peers can be invaluable
in the learning process. Sharing ideas, sharing resources, and collaborating on projects or problems
promote higher level thinking skills and communication. Distance educators should promote
collaborative learning.
3. Good distance learning incorporates active learning.
Effective learning involves more than listening and reading, and effective teaching is more than
talking and encouraging memorization of text. Active learning should involve application and problem
solving, research, and simulation.
4. Good distance learning gives rich and rapid feedback to students.
Distance education students should be able to regularly assess their own learning as well as get
feedback from others. Feedback can be as simple as a webpage link to answers for a problem set or an
e-mail message or as complex as using "hidden text" options on special software to give a critical
analysis of written work.
5. Good distance learning stresses time-on-task.
The distance-learning environment should be rich with reading, activity, and interaction. Busy work,
time wasted on locating materials, and time wasted figuring out the glitches in the system minimize
learning and maximize frustration.
6. Good distance learning sets high standards for student performance.
Distance learning should be challenging -- indeed, the opportunities for interaction, collaboration, and
applied activity can make distance learning more challenging than many of the conventional "chalk-
and-talk" classrooms. The expectations in terms of projects, products, and performances should be
clearly communicated from the onset. Clearly established criteria set the standards for judging
excellence.
7. Good distance learning respects individual differences and allows students opportunities for
learning that acknowledge those differences.
Distance education should not incorporate a lock-step, "one size fits all" approach. Instead, distance
learning should be organized such that students find their own best ways for approaching problems,
completing tasks, and using learning materials.

EENOVATE project general assumptions

The general philosophy of the EENOVATE project relies in the assumption that differences in
elearning programmes can be identified and analysed. Therefore, a proper operationalisation of the
elearning concepts concealed by various theories and practices could lead to a model pivoting the
institutional policy transfer. Three general assumptions stand for milestones in the evaluation research
grid:

1. Quality in elearning programmes is supported by an integrated institutional approach comprised of

concern for and continuous amelioration of general policies, management, curriculum, support and
technical facilities, staff development, evaluation.
2. Quality in elearning programmes is revealed by the extent to which staff commitment is reflected in
instructional activities and complementing services.
3. Quality in elearning programmes is measurable through institutional outputs, student performance,
staff and students satisfaction,

Each of these assumptions is addressed by different items in the research instruments: an

institutional evaluation grid (INST), a interview guide for elearning programmes managers (MAN), a
interview guide for teachers/ tutors (TUT), a questionnaire for students (STUD) and a focus-group
guide (FOCUS). Some specific issues the project team focuses on are to be employed in each research
instrument (as a question also for teachers/ tutors/ administrative staff and students) – for a
triangulation of data collected.

PROJECT MAIN DIMENSIONS (as outlined in the project proposal):

Design, implementation and data gathering

Analysis of main findings

Analysis of conditions for transfer

Elaboration of project report and setting a new agenda for CC

Platform creation and development

Information and dissemination

Validation of project results/conclusions

Monitoring and evaluation

RESEARCH WORKING PLAN

STAGE 1: Data gathered on current situation in higher education elearning programs (CC and EUC)

1.1. Desk research in CC and EUC (outcome: national reports and synthesis report)
1.2. Development of an Elearning Research Matrix including:
- research plan
- agreed working definitions
- principles of quality elearning program
- fundamental assumptions of the project
- overview of elearning concept map, including categories such as: institutional policies,
curriculum and instruction, technical facilities, staff development, evaluation.

1.3. Refining the research methodology and design of research instruments

1.4. Field research in CC and EUC (outcome: report on CC representative universities needs and
report on EUC practices on identified needs)

STAGE 2: Data analysis; identification of CC needs and EUC practices

STAGE 3:Identification of transferable policies from EU countries and analysis of conditions of

transfer

STAGE 4: Validation of outcomes at institutional level (based on focus groups with stakeholders in
elearning at national level in both CC and EU)

STAGE 5: Elaboration of recommendations for further action in CC countries (policy level)

DETAIL OF STAGE 1.4.

CC
STEP 1 Institutional evaluation grid
(5 universities in RO,
5 universities in BG)
STEP 2 Interviews with managers,
teachers/tutors and students
(5 universities in RO,
5 universities in BG)
EU countries
Institutional evaluation grid
(… universities in UK,
…universities in ES)
Interviews with managers,
teachers/tutors and
students(?)
(… universities in UK,
…universities in ES)
Elearning concept map
Legend:
CC- Candidate Countries INST– Institutional Evaluation Grid
EUC- European Union Countries MAN – Interview Guide for Elearning Programmes Managers
TUT – Interview Guide for Teachers/ Tutors
STUD – Questionnaire for Students
FOCUS – Focus-Group Guide

Please, note that only some of the items below will be taken into account and reflected in questionnaires or interviews to be developed!

A. Elearning-related institutional policies

University’ distance education policy

Corresponding item no. in the

questionnaire/ interview/ focus group
guides
CC EUC
1. Program plans and budget include integration of distance delivery with the institution’s overall policy
framework.
2. Program plans include written policies for all aspects of the distance education program.
3. Program plans and budget include financial and administrative commitment to the continuation of a program
for a period sufficient to enable students to complete a degree/certificate.
4. Distance learning program is supported by an institutional commitment to instructional quality and
effectiveness.
5. Distance education program completion takes the form of credits or credentials that are recognized by
professional accreditation bodies and employers.

Draft 23.06.2005 7
Management

6. Distance programme development and maintenance procedures (establishing responsibilities, developing and
completing checklists, submitting proposals, approval procedures, scheduling) are clearly stated.
7. Appropriate and necessary personnel include customer-oriented management that helps with information,
course/program advising, application and registration procedures.
8. Management of student records consists in documentation of student achievement in each course and at
completion of a program.
9. Learner involvement in decision-making is possible.
10. Program plans include an adequate budget to achieve stated program goals.
11. There are stated clear quality standards to be achieved and/or maintained.

Marketing

12. Product/service information for potential distance-learning students is comprehensive and complete.
13. Study programmes are promoted via Internet.
14. Study programmes are promoted via traditional means.
15. Distance learning programmes are presented on distinct leaflets. N/A N/A
16. Access to elearning services is facile. Enrolment services are free of cultural, racial, class and gender bias.

B. Curriculum and Instruction

Materials, programs and courses

1. The course design has to fit some institutional standards that regulate the length, the percentage of theory,
application, evaluation, the course description etc.
2. Intended learning outcomes are clearly stated. INST 1.1.
3. Intended learning outcomes are shaped with input from learners.
4. Learning outcomes are matched to instructional and learning activities.
5. Intended learning outcomes are appropriate to the rigor and breadth of the degree or certificate awarded.
6. Teaching / learning materials are prepared by qualified content experts working with qualified design experts

Draft 23.06.2005 8
 [identified].
7. Teaching / learning materials are readily available and learner friendly – able to be used by the average
student.
8.
9. Teaching / learning materials are free of cultural, racial, class and gender bias. INST 1.1.;
10. Courses accommodate multiple learning styles.
11.
12. A complete learning package includes information about most of the following: course description, course/
project objectives, information about the instructor[s], learning/ lecture notes and additional learning
resources, course activities and assignments, quizzes and examinations, answers to questions/ quizzes, a
portfolio of acquired learning.
13. Course maximizes the opportunities for regularized and ongoing interaction between teacher and students,
among students, and between students and learning environment.
14. The course provides opportunities for students to engage in activities and tasks that enhance comprehension,
understanding and knowledge.
15. Syllabi are reviewed regularly to ensure clarity, utility, and appropriateness. INST.1.1.

Content

16. The course content is approved/ validated by a (internal/ external) commission. INST 1.1.
17. Curriculum content is relevant. STUD x; TUT
19;
18. Curriculum content is culturally sensitive and free of bias.
19. The courses’ content is updated regularly (at least once at every 3 years). STUD x; TUT TUT 19;
19;
20. Curriculum content is updated consistently.
21. Multimedia use is suitable to the topic and to the constraints of delivery.
22. Course content is provided in segments that are technologically deliverable.
23. Course is consistent in delivery and information format (i.e. consistent use of file format, course schedule, STUD x; TUT
navigation etc.). 19;
24. Plans are developed to have materials delivered to or available for students at remote sites in a timely manner.

Distance learning methodology

Draft 23.06.2005 9
 25. The distance learning environment is rich with activity. Students are active learners in presenting, organizing, INST 1.2.
applying and constructing information, ideas and knowledge.
26. The distance learning environment is rich with interaction. Programs provide opportunities for timely and INST 1.2.
appropriate interaction between students and faculty, and among students.
27. Teaching staff/ counsellors/ tutors respect diverse ways of learning.
28. Distance education students can follow personalised learning routes.
29. Courses and seminars involve application and problem solving, research, and simulation.
30. Collaborative learning (student-student) is requested by specific tasks during the courses/ online seminars.
31. Instructor-led discussion forums are used. Student-led/ instructor monitored forums are used.
32. Specific ground-rules for discussion are established and communicated.
33. Scheduling and timetabling is flexible and responsive to learners.
34. Frequency and timing of participation expectations are stated and described.
35. If required, on campus extended campus, or synchronous session (live chat, labs, reviews etc.) are identifies
and scheduled.

Support

36. Distance students are given advance information about succeeding in a distance learning environment.
37. Contacts and communication between learners and the (teaching/ counselling/ technical) staff are encouraged.
(contact data are available on the institution website)
38. Faculty and student opportunities for interaction are frequent and planned.
39. Any required course-specific tutorial/ learning support resources and services are available online or at
students’ locations.
40. An orientation program/service is available for the enrolling students. STUD 3 STUD 3
41. Specialised personnel offer advice and counselling within the distance education program.
42. During the learning programme, on every subject, one tutor is assigned to small groups (5-20) of students.
43. Teaching staff/ tutors provide prompt feedback - in less than 24 hours - to (individual) students.
44. Feedback to student assignments and questions is constructive.
45. Specialised personnel deal with students’ technical difficulties.
46. Alternate backup plans and responsibilities in the event of technical difficulties and technology failure are
listed.
47. Explanations of how students will receive their first orientation to the course are given.
48. (Online) meetings are scheduled to accommodate working students.
49. Career counselling and/ or appropriate placement is ensured for the distance education programme graduated

Draft 23.06.2005 10
 students.

Library and learning resources

50. Teaching staff/ Tutors provide links to further resources.

51. The distance learning environment is rich with readings.
52. Online learning support resources are easily and accessible via distance delivery.
53. If other materials (books, journals, software, and electronic services/ resources) are required, they are listed
with description of how they may be obtained.

Assessment

54. Methods and plans for student assessment are described.

55. Grading criteria are listed (percentages, weighted scores, points, letter, grades etc.)
56. Evaluation of learning is against criteria that are transparent, relevant, realistic, reliable, and valid.
57. Student evaluation combines two or more forms of assessment (continuous evaluation, final exam, project-
based evaluation/ student portofolio).
58. Distance education students have the opportunity to regularly assess their own learning.
59. Distance education students get feedback from their colleagues.
60. Learning assessment is frequent.
61. Feedback procedures on assessments are described (posted grades in the gradebook, or how critiques of
assignments are returned).
62. Assignment submission instruction or guidelines (acceptable formats/ file/ software formats are provided and
described).

C. Technical facilities
1. Teaching / learning materials are relatively easy to use and free from technical hitches.
2. The elearning system provides appropriate interface design.
3. Technologies are appropriately used to make students feel comfortable.
4. Technologies are appropriately used to accommodate and promote individualization.
5. Technologies are appropriately used to create opportunities for students to do meaningful work.
6. Technologies are appropriately used to increase proficiency at accessing, evaluating and communicating

Draft 23.06.2005 11
 information.
7. Technologies are appropriately used to improve students’ abilities to solve complex problems.
8. Technologies are appropriately used to nurture artistic expression.
9. Technologies are appropriately used to enable active engagement in the construction of knowledge.
10. Technologies are appropriately used to drill students on basic concepts to reach mastery.
11. Communications facilities, processes and practices are able to encourage contact between students and
faculty.
12. Communications facilities, processes and practices are able to provide flexible opportunities for interactions
and problem-solving.
13. Communications facilities, processes and practices are able to develop reciprocity and cooperation among
students.
14. Communications facilities, processes and practices are able to provide the opportunity to “hear” other
students’ questions.

D. Staff
1. Appropriate and necessary personnel include instructors / teachers / professors with qualifications in the
subject area.
2. Distance education personnel are qualified for teaching in elearning environments.
3. Human resources management practices include a requirement for ongoing professional development in
content areas.
4. Human resources management practices include availability of technical skills development and support.
5. Human resources management practices include regular evaluation of competence.
6. Staff is assisted in the transition from classroom teaching to online instruction. TUT 22;
7. Instructor training and assistance, including peer mentoring, continues through the progression of the online
course.

E. Evaluation/ monitoring
1. The institution's faculty assumes responsibility for and exercises oversight over distance education, ensuring
both the rigor of programs and the quality of instruction
2. Evidence of program success is support through routine review and evaluation of course content and

Draft 23.06.2005 12
 objectives, learning materials, instructional design.
3. Evidence of program success is support through routine review and evaluation of teaching staff.
4. Evidence of program success is support through routine review and evaluation of learning and student
achievement. The system provides a link to evaluation form/process for assessing student satisfaction with
services provided by the distance learning programme.
5. Evidence of program success is support through routine review and evaluation of policies and management
practices.
6. Evidence of program success is support through routine review and evaluation of students’ satisfaction.
7. Distance learning program is evaluated periodically and includes: course content, course design (academic
rigor, pedagogical and technical suitability), course presentation, curriculum alignment, student performance,
student participation, students’ interactions, teacher interaction with students, evaluation tools etc.

Draft 23.06.2005 13