Towards a Multi-criterion Web-Based Tool for Evaluation of an eLearning System

Nikolaos F. Matsatsinis, Karagounakis Agelos, Pavlos Delias

Technical University of Crete Decision Support Systems Laboratory University Campus, 73100, Chania, Greece

{nikos,merlin, mirouvor}@ergasya.tuc.gr

Abstract. By the growing pervasiveness of computers and e-learning systems in the educational reality and as the e-learning alternatives are fairly enhanced, an evaluation need arise. This need comes not as a valorization process but as a forthright effort to ameliorate elearning systems. There are many factors, other than learning, impacting behavior, and many factors, other than behavior, impacting performance. In this study1 we map out a web- based evaluation modus which implements a multiple criteria methodology. This technique allots elearning systems an appraisal on their performance while it exhibits an overall view of their users behavior.

Keywords: Multiple Criteria Analysis, Decision Support Systems, Education

1. Introduction
The constant growth of the Web influences over and above education and has a large effect on how courses are designed and implemented. The e-learning perspective has intensely covered the way educational organizations think of teaching. The increasing online access to programs, courses, and course information is exciting. Initial research exploring the potential of online learning has provided some overall insights ([5] [7] [14] [15]). The Making the Virtual Classroom a Reality (MVCR) online program at the University of Illinois alone had admitted over 1000 individuals from various states and foreign countries by December 2002 [30]. Some of the top institutions in the United States (e.g., MIT, Indiana University, Pennsylvania State University) are offering entire degree programs on line, ranging from business to education, criminal justice to nursing [33]. Yet, the movement towards online learning is not always assuring successful implementation, thus it is not always confirming the expecting results. E-learning is also one of the areas that attract the most research and development funding. If this investment is to be

This research has been carried out with financial support from the European Community and the Ministry of National Education and Religious Affairs of Greece EPEAEK II Project.

maximized, it is imperative that we generate robust models for the systematic evaluation of elearning and produce tools which are flexible in use but consistent in results [16]. Moreover, there has been concern among educators that quality assurance procedures for the design and delivery of Internet-based courses have been ignored, as colleges and universities rush to offer an array of distance-delivered programs that will allow them to maintain and grow enrollments [27] [28], therefore they need tools and instruments which will increase their ability to make more analytic and interpretive evaluations of e-learning, using a greater range of methodologies. In the next section, we try to figure out how researchers commune with evaluation need in elearning systems while in section three we present a multiple criteria evaluation method [25]. In section four we present the implementation of that method, while conclusions are set out in the last paragraph.

2. Evaluation of e-learning systems

In every evaluation process there is an interrogative issue that concerns the whole series of actions. This is about the evaluation purpose. Therefore, we wonder if the purpose of evaluation is about justification and valorization or about improvement. To this question, like many cases, there is no single answer. Evaluation is about both. Anyone who would like to design or implement an e-learning platform should care about its valorization and its upswing as well. Moreover, that interrogative spirit concerns the evaluations scope. Should we compare elearning with traditional learning methods or we should just judge the efficiency of a stand-alone e-learning platform? Is evaluation about a contest among e-learning platforms? These very distinct perspectives lead Hughes et al. to classify e-learning evaluation into the following seven categories: Case studies of specific e- training programs, Comparisons with traditional learning, Tools and instruments for evaluation of e-learning, Return on Investment (ROI) reports, Benchmarking models, Product evaluation and Performance evaluation. The systems approach to program evaluation is highly compatible with strategic planning, which has the goal of strengthening the management decision-making process by having it recognize and address internal and external key factors that affect the organization. Since programs delivered at a distance consist of multiple components, e.g., the e-learning software, academic and technical support, presentation of content, and interaction, evaluators must recognize that all components of the program must work together in an efficient manner if the entire system is to be effective. Consequently, it is important to evaluate distance education programs by how they work as a whole rather than by evaluating individual components without regard to overall program effectiveness [2]. Let us perambulate in bibliography to investigate common features of the proposed evaluation models. Yi-Shun Wang [35] carried into effect a survey intending to end up with an evaluation tool. In his survey, he detected five different instruments [4][6][11][23][24] that evaluate teaching activities, merely mentioning here Instructor and Course Evaluation System, Student Description of Teaching Questionnaire and Students Evaluations of Educational Quality Instrument. Moreover he issued a questionnaire in order to obtain some sample data. That questionnaire [35] included a 24-item list that constitutes the total of the satisfactions factors plus a pair of global satisfaction questions (global satisfaction and success of the e-learning system). The satisfactions dimensions that he used referred to user information satisfaction, end-user computing satisfaction, customer satisfaction, and student satisfaction [9] [10]. The 24-item instrument (with the two global items excluded) was refined through analyzing the pooled data. Yi-Shun Wang computed the coefficient alpha and the item-to-total correlations of the sample data. In addition, to further

examine the factor structure of the 24-item instrument, he used principal components factor analysis, resulting this way to a final total of 17-item tool. Finally, Yi-Shun Wang proposes as an ultimate evaluation measure and a better way of assessing individual satisfaction the comparison between individual satisfaction levels and norms (the total distribution of the satisfaction levels rated by other people). While the previous case implements some statistics tools to evaluate e-learning systems, a groupdecision approach is followed by Gwo-Jen Hwang et al. [17]. Actually, Gwo-Jen Hwang et al. have developed Educational Web Site Evaluator (EWSE) which is a system that handles methodologies such as multiple criteria decision-making [20], fuzzy logic, Grey system theory [8] and Analytic hierarchy process (AHP) [29]. The model of EWSE takes place through four phases. In the first phase, the system collects the criteria that domain experts suggest and it select the most appreciated ones with the grey system method. In the second phase the EWSE adjusts the weight of each evaluator, judging by the evaluators ratings accordance. It is in phase three, that the system implements fuzzy theory and memberships functions in order to rate the educational web-sites. While fuzzy theory is used to assess alternatives ratings to every criterion, criteria weights are arbitrarily decided. Finally, during the last step, grades of all the evaluation items are integrated and the final decision is presented to the users. A technical approach to evaluate an elearning platform is presented by the UCF research team [26] who have successfully designed and executed a mail-based survey system for evaluating the university's distributed learning initiative. Building on their work, the authors designed a Web-based survey system featuring three components: data collection, data validation, and data analysis. According to the above approaches of e-learning systems evaluation plus according to the standardization that ISO and IEEE [18] [34] have established about software quality, an efficient way to measure these systems performance is a multiple criteria approach. Each criterion is qualified with its own weight and the total grade of each alternative (in our case the term alternative signifies an e-learning system) is a weighted sum of alternatives ratings in each criterion. Furthermore, although the criteria are being decided by experts, the evaluation is procreated by the end-users. The number of those criteria is pretty large (above fifteen) so there is a need to be presented to the users in a homogeneous way hereby there is an additional need for grouping those criteria into definite and unambiguous groups. Nonetheless, in the field of elearning evaluation systems there is deficiency in estimating criteria weights, resulting to an assumption but also to a challenge that we try to face in this study.

3. The proposed methodology 3.1 Main principles and notations

Foremost, we should retain that the evaluation process is fully depended on the users judgements and that the more satisfied the users are, the higher the system should be evaluated. Considering users global satisfaction as a multivariate function, we just have to compute an additive formula to aggregate every dimensional satisfaction. What often happens in quantitative methods and data analysis techniques [13] is a two-level or three-level regression analysis between overall satisfaction and satisfaction originated from criteria and sub-criteria. In this model there are still the quotidian questions about the overall satisfaction and the dimensional satisfaction arrived from each sub-criterion. However, we interpenetrate these questions interpolating a criteria level. While the questions of the global and the partial satisfactions are being answered by a numerical scale 0-10 (where 10 means absolutely satisfied), criterias satisfaction is measured by an ordinal scale. This happens because criteria refer to a more abstract topic (such as those described in the

next section) and a vocable term would better describe users beliefs, while every sub- criterion corresponds to a very specific and quite measurable characteristic. Let us define the variable used by the model:
Table 1 Definition of model's variables. Y n ni M Xi Xij Xi* bi bij ai Users Overall Satisfaction Number of the criteria Number of sub-criteria referring to the ith criterion Number of the users Value of the satisfaction level of the ith criterion Users satisfaction for the j sub criterion of the ith criterion. Value function of the ith criterion Weight of the ith criterion Weight of sub-criterion j of the ith criterion. Users satisfaction level on the ith criterion

It should be noted that the method follows the principles of ordinal regression analysis under constraints, using linear programming techniques [19] [31] [32] and may be considered as an extension of the MUSA satisfaction evaluation model [12]. Based on the previous framework, the proposed model infers a set of value functions Xi* respecting the assumed value hierarchy of user satisfaction. By this way, the model can handle qualitative (ordinal) as well as quantitative data. The main objective of the model is to achieve the maximum consistency between the overall satisfaction judgements Y, the value functions Xi* and the sub-criteria satisfaction judgements Xij. It should be emphasized that wile Y and Xij are ratio variables measured in a 0-10 scale, Xi is a set of ordinal variables.

3.2 Model development

The assessment of user satisfaction is based on a simple ordinal regression model and is introducing a double-error variable for each level interaction, the ordinal regression equation becomes as follows:

b X +
*
+ q

n tqi 1

i =1 k =1
* i

=Y

10

With

b =1
i =1 i

and bij = 1
j =1

ni

(1) (2)

X =
j =1

ni

bij X ij

10

+ q + q i i

i = 1, 2,..., n

Where Y and Xij are multiplied be 10 in order to have Xi* normalised in the interval [0,100] and the variables are the error variables. Equations (1) and (2) hold for each user who has expressed a set of satisfaction judgements. For this reason, four pairs of error variables should be assessed for each user separately. In order to eliminate the monotonicity constraints at Xi* in a mathematical programming formulation and to overcome the program of non-linearity in equation (1), we introduce the following transformations:

wik = bi ( X i*k +1 X i*k ),

*k So that bi X i =
*1 While X i = 0

i = 1,..., n k = 1, 2,..., ai
k = 2,..., ai i = 1, 2,..., n

(3) (4)

w ,
t =1 it

k 1

& bij ' bi

X iai = 1,
ni

i = 1,..., n

Moreover let bi*bij = bij . That leads to the following transformation:

bij = 1
j =1 j =1

ni

ni

= 1 bij ' = bi ,
j =1

i, bi 0

(5)

According to the aforementioned definitions and assumptions, the users satisfaction evaluation problem can be formulated as a linear program in which the goal is the minimisation of the sum of errors, under the constraints: (i) ordinal regression equation (1) and (2) for each user, (ii) normalisation constraints for wik. Therefore, the final linear problem is modelled as follows: [min] Subject to: 1st-2nd level

(
M
+

q =1

+ q

+1

+ n (
M n
+

qi

q =1 i =1

qi

(6)

w +
+

n tqi 1

2 -3 level

i =1 k =1 nd rd

ik

=Y
ni

10

, q=1,, M

(7)

wik q+i + qi =
k =1
n

tqi 1

bij ' X ij

j =1

10

, i=1,, n , q=1,,M

(8) (9) (10) (11)

w
i =1 k =1

qi =1

ik

=1

bij ' = wik , i=1,,n

j =1

ni

ni 1

Wik, bij,

k =1 q+,

qi+ q-, qi- 0 i, j, k, q,

The second part of the objective function is weighted out by the factor 1/n just to limit the relative importance of the error variables that refer to the sub-criteria. The stability analysis is considered as a post-optimality analysis problem, considering that the proposed method is based on a linear programming modelling. The post-optimal solutions space is defined by the polyhedron {F F*+ , all the constraints of Linear Problem (6)-(11)}, where F* is the optimal value for the objective function of LP (6)-(11), and is a small percentage of F*. According to the aforementioned remarks, during the post optimality analysis stage of the method, a number of linear programs equal to the number of sub criteria are formulated and solved. Each linear program maximises the weight of a criterion and has the following form:

[max] F ' = bij for i = 1,,n j=1,..,ni Subject to:

(12)

F F* + (13) All the constraints of the LP (6-11) The average of the optimal solutions given by the n LPs (12) may be considered as the final solution of the problem.

3.3 Criteria Selection

Although the proposed methodology could be used including any criteria and sub-criteria ensemble that represent an hierarchy form, in this study we suggest an indicative set of fifteen sub-criteria that refer to three main criteria. As mentioned in section 2, the evaluations dimension should spread across the fields of users satisfaction, teaching activities effectiveness as well as of the software quality domain. Therefore, when deciding about the evaluation criteria, we ought to consider that the criteria should scatter the above topics, while their number should remain small. Further properties that the criteria should fulfill are proposed by Keeney [21] and mainly refer to their structure that should be essential, controllable, complete, measurable, operational, decomposable, nonredundant, concise and understandable. For instance, in the proposed methodology, all the questions that users are being asked are satisfaction-oriented in such a way that the satisfactions conception inheres in every simple question. Conversely, the effectiveness of teaching activities and the quality of the software can not be evaluated this way; hence we should implement a criterion that would be more teaching-aware and another one that would particularly concern the softwares quality topic. Thereupon, three main criteria were decided to altogether evaluate the e-learning systems. First of all, there is an interface criterion which awakes users aestheticism. It mainly concerns the presentation layer of the system and it employs six sub-criteria [10] [17] [22] [35]. In order to delineate the teaching activities dimension, the contents criterion is implemented. Its six subcriteria venture to depict the ability of the system to achieve its original objective, and that is the learning idea [1] [4] [6] [11] [13] [17] [24]. Finally, a software entity is perpetually judged by its performance. Hereby the functionality criterion emerges. We attempt to remain consistent to the software quality ideal by inspecting over four sub-criteria [3] [18] [34]. Approaching the interface concept, the proposed methodology contemplates on the web designs style. The question that arises in this point is about users design expectation. This is a question just for the optical effect and not for the ergonomics aspects of the design. These aspects refer to the user-friendliness sub-criterion that surveys the guidance that the system provides the user with. The Personalization sub-criterion checks over the users ability to customize the e-learning systems working environment. Moreover, the quality of the digitations such us paper notes, diagrams, videos etc. is scrutinized through the multimedia sub-criterion. This question merely refers to the quality of the digitations and should not expand over any other view (hereby reminding that for that purpose we examine the contents criterion). Finally, we examine if the communicative functions of the system really enhance it. Let us define the communicative functions as the chat rooms, the forums, the FAQs, the e-mailing potential etc. Things are quite more straightforward in the contents criterion: structure of contents subcriterion reveals users satisfaction about data organization while the up-to-date part of the system is being check up by a corresponding sub-criterion. In addition it is scanned the way that the

system itself evaluates the students learning progress with another distinct sub-criterion. Moreover, two separate sub-criteria are used to examine not just only the stuffiness of the contents but also the potential of extra learning-source. Finally, users are asked if they are satisfied with the format of the contents, that is if data are readable and downloadable. The third criterion concerns the functionality of the system as a software object. We should keep in mind that e-learning systems are web-based systems therefore we survey the following dimensions: the response time that must be kept in a minimum; the security level that should not only protect the system from hack-assaults but as well it should assure the fairness of the learning process; a way to measure the users expectations about the reliability of the system and finally the ability of the system to co-operates with different operating systems. An analytic description of the selected sub-criteria could be found in [25].

4. Implementing the evaluation system

The implementation of the methodology we presented at the last section demands an interface to provide access to users in order to let them evaluate the system and to monitor the results, a logic module to perform the calculations needed and a database to store the information. These demands lead us to a three-tier architecture depicted in figure 1. Users access the system through a web interface where they can submit their assumption on the e-learning platform and they can observe the results of the evaluation process. Middle tier is responsible to solve the linear problem described in section 3, to perform all the calculations needed and to update the database. Database itself is a distinct tier.

Figure 1 Three tier architecture of the evaluation system.

The sequence of these actions is illustrated in figure 2. The interactive evaluation process can be supported by such a system as it dynamically adjusts to any modification of the data. We shall mention that the solver module recalculates the multiple criteria solution as soon as a small amount of new users submits its opinion. This happens to increase the performance rate of the system as calculating the new solution every time a new user submits his opinion costs in performance units (time, CPU) plus, it is not significantly modify the solution. We may remind that the proposed methodology seeks to unify users conceptions under a single set of utility

functions. Anyway, we ponder that the system should perform the calculations steps periodically.

The results that are published consist of the following: 1. Overall Satisfaction Index : this average index shows, in the range 0-100%, the level of global satisfaction of the users; it may be considered as the basis average performance indicator for the e-learning system and is calculated as the average of the Y answers 2. Criteria value functions : These curves shows the real value (0-100) that users give for each level of the criteria ordinal satisfaction scale; the form of these curves indicates how demanding users are for each criterion 3. Criteria & sub-criteria satisfaction indices: These indices demonstrate the level of partial satisfaction of the users to every criterion and sub-criterion, similarly to the overall satisfaction index 4. Weights of criteria & sub-criteria: they show the relative importance within a set of criteria. 5. Demanding indices: these indices concern just the criteria (not the sub ones!) and are normalized in the interval [-100%, 100%] and they are calculated based on the set of estimated added-value curves; these indices show users demanding level (globally and per criterion) and they may be considered as an indicator for the extent of the systems improvement efforts. 6. Action diagrams: these performance/importance diagrams are developed through the combinations of Figure 2 Sequence UML Diagram. criteria weights and satisfaction indices. They are similar to SWOT analysis and they can represent the strong and the weak points of the business organization, indicating which satisfaction dimensions should be improved.

The UML diagram of the system is altogether illustrated in figure 3

Figure 3 UML Diagram of the evaluation system.

5. Conclusions
Evaluation is an essential component of program improvement and renewal and long-term success. Furthermore, when the evaluation occurrence concerns educational activities, evaluation results turn to become more sensitive and can be extrapolated to improve the distance education context. Hence, there should be a growing practice of building evaluation into an e-learning process through the use of on-line tools that assess students perception and performance based on the belief that this can save time as well as money. This notion should be examined from the perspective of the pedagogical assumptions underpinning it and the robustness and usefulness of the data generated in this way. The method we presented in this paper using the analytic synthetic approach described in section three and implemented by the web architecture described in section four battles with these assumptions and provides e-learning decision makers with reliable data. We consider the proposed tool to merge existing research and to join e-learning researchers towards this vision.

References
[1] Aldridge, S. and Rowley, J. (1998). Measuring customer satisfaction in higher education, Quality Assurance in Education 6 (4), pp. 197204. [2] Alfred P. Rovai, (2003). A practical framework for evaluating online distance education programs. Internet and Higher Education 6 (2003) 109124 [3] Blin, M.J. and Tsoukis, A. (2001). Contribution of Multi-criteria Methodology to Software Quality Evaluations. Software Quality Journal, vol.9, pp. 113 132. [4] Cashin, W.E. and Downey, R.G.(1992). Using global student rating items for summative evaluation. Journal of Educational Psychology 84 (4), pp. 563572. [5] Cereijo, M.V.P.,Young, J.,&Wilhelm, R.W. (1999). Factors facilitating learner participation in asynchronousWeb-based courses. Journal of Computing in Teacher Education, 18(1), 3239. [6] Cohen, P.A.(1981). Student ratings of instruction and student achievement. Review of Educational Research 51 (3), pp. 281309. [7] Conrad, D.L. (2002). Engagement, excitement, anxiety, and fear: Learners experiences of starting an online course. American Journal of Distance Education, 16(4), 205226. [8] Deng, J. L. (1989). Introduction to Grey System Theory. The Journal of Grey System, vol.1, pp. 124 1989. [9] Doll, W.J. and Torkzadeh, G. (1991). The measurement of end-user computing satisfaction: theoretical and methodological issues. MIS Quarterly 15 (1), pp. 510. [10] Doll, W.J., Xia, W. and Torkzadeh, G.(1994). A confirmatory factor analysis of the end-user computing satisfaction instrument. MIS Quarterly 18 (4), pp. 453461. [11] Feldman, K.A. (1989). The association between student ratings of specific instructional dimensions and student achievement: refining and extending the synthesis of data from multisection validity studies. Research in Higher Education 30 (6), pp. 583645 [12] Grigoroudis, E. and Siskos, Y. (2002). Preference disaggregation for measuring and analyzing customer satisfaction: The MUSA method, European Journal of Operational Research, 143 (1), pp. 148-170 [13] Guolla, M.(1999). Assessing the teaching quality to student satisfaction relationship: applied customer satisfaction research in the classroom. Journal of Marketing Theory and Practice 7 (3), pp. 8797. [14] Hartley, K., & Bendixen, L.D. (2001). Educational research in the Internet age: Examining the role of individual characteristics. Educational Researcher, 30(9), pp. 2226. [15] Hill, J.R. (2002). Overcoming obstacles and creating connections: Community building in Web-based learning environments. Journal of Computing in Higher Education, 14(1), 6786. [16] Hughes, J., Attwell, G., (2002(.A Framework for the Evaluation of E-Learning, Stirling, Scotland, November 2002, from http://www.theknownet.com/ict_smes_seminars/papers/Hughes_Attwell.html [17] Hwang,, G.J., Huang, C.K.T. and Tseng, C.R.J. (2003). A group-decision approach for evaluating educational web sites. Computers & Education, vol. 42 , pp. 6586. [18] International Organization for Standardization, (1991) ISO 9126: Information Technology - Software product evaluation - Quality characteristics and guidelines for their use [19] Jacquet-Lagreze, E. and Siskos, J. (1982). Assessing a set of additive utility functions for multicriteria decision-making: The UTA method. European Journal of Operational Research 10 (2), pp. 151164. [20] Keeney, R. L., & Raiffa, H. (1976). Decisions with multiple objectives: preferences and value tradeoffs. New York: John Wiley and Sons. [21] Keeney, R. L., (1992). Value focused thinking : A path to creative decision making, Harvard University Press, London. [22] Long, P.D., Tricker, T., Rangecroft, M. and Gilroy, P. (1999). Measuring the satisfaction gap: education in the market-place. Total Quality Management 10 (45), pp. S772S778. [23] Marsh, H.W. (1987).Students evaluations of university teaching: research findings, methodological issues, and directions for future research, International Journal of Educational Psychology 11 (3), pp. 253388. [24] Marsh, H.W. (1991). Multidimensional students evaluations of teaching effectiveness: a test of alternative higher-order structures, Journal of Educational Psychology 83 (2), pp. 285296. [25] Matsatsinis, N., Grigoroudis, E., Delias, P. (2003). User satisfaction and e-Learning Systems: Towards a Multi-criteria Evaluation Methodology. Operational Research An International Journal, Multicriteria Decision Aid: Theory and Applications (vol. 3 n3 2003) [26] Moskal, P.D., Dziuban, C.D., & Moskal, P.J. (1999). Faculty go online, impressions of faculty teaching web and web-enhanced courses, University of Central Florida. Unpublished manuscript. [27] Motiwalla, L., & Tello, S. (2000). Distance learning on the Internet: An exploratory study. The Internet and Higher Education, 2(4), 253264. [28] Phipps, R. A., Wellman, J. V., & Merisotis, J. P. (1998). Assuring quality in distance learning: A preliminary review. Washington, DC7 Council for Higher Education Accreditation. [29] Saaty, T. L. (1980). The analytic hierarchy process. NK: McGraw-Hill. [30] Santovec, M.L. (2003). Faculty development. Distance Education Report, 7(2), 8.

[31] Siskos, Y. (1985). Analyses de regression et programmation lineaire. Revue de Statistique Appliquee, vol.XXXII, pp. 4155. [32] Siskos, Y., Yannacopoulos, D. (1985). UTASTAR: An ordinal regression method for building additive value functions. Investigacao Operacional 1 (5), pp. 3953. [33] Song L., Singleton S. E., Hill R. J., Koh H.M., (2004). Improving online learning: Student perceptions of useful and challenging characteristics, Internet and Higher Education, 7, 5970 [34] The Institute of Electrical and Electronics Engineers, (1992), Standard for a Software Quality Metrics Methodology, December 1992 [35] Wang, Yi-Shun (2003). Assessment of learner satisfaction with asynchronous electronic learning systems. Information & Management, vol.41 pp. 7586.