Volume 2 No.

2 ISSN 2079-8407
Journal of Emerging Trends in Computing and Information Sciences

©2010-11 CIS Journal. All rights reserved.

 
http://www.cisjournal.org

The Effectiveness of Learning Management System (LMS) Case Study

at Open University Malaysia (OUM), Kota Bharu Campus
Khadijah Abdul Rahman, Siti Aswani Mohd Ghazali, Dr Mohd Nasir Ismail
Information Management Faculty, University of Teknologi MARA, Machang Campus, Kelantan
Khadijah@kelantan.uitm.edu.my, aswani211@kelantan.uitm.edu.my, nasir733@kelantan.uitm.edu.my

ABSTRACT
 
Learning Management System (LMS) provides an integrated platform for content, delivery and management of learning as
well as accessibility by a range of users that may include learners, content creators and administrators. Open University
Malaysia, the seventh private university in Malaysia that fully operated as an Open and Distance Learning environment
had successfully implemented the LMS to ease the learning process of their students as well as their administrators.
Although LMS is an important element in OUM, most of the students do not fully utilize the system to support for online
submission of course works and discuss any matters in discussion board or online discussion as well as to self-evaluate
their performance in learning. They tend to do all those things manually such as discuss any subject face to face meeting
with their peers and tutors, they prefer to have paper based tests rather than online quizzes and so on. These factors
contribute a major reason why LMS is not used effectively .T his paper is measuring the effectiveness of Learning
Management System of Open University Malaysia by testing the LMS done by three categories of users which are novice,
knowledge intermittent and expert users along with the distributions of hundred set of questionnaires to randomly selected
users in order to collect data regarding the effectiveness of OUM’s Learning Management System. The findings and result
of this survey and testing is also discussed.

Keywords: LMS, Open and Distance Learning, OUM

I. INTRODUCTION  The distance education and institute train a

Maxwell (1995) said that Open learning is
defined as a student-centered approach to education that
removes all barriers to access while providing a high
degree of learner autonomy. Distance education refers to
a mode of delivering a course of study in which the
majority of communication between teachers and
students occurs non-contiguously, and the two-way
communication between teacher and student necessary
for the educational process is technologically mediated.
Distance education may or may not be based on open-
learning ideals. Distance education programs are
becoming more common in traditional higher
education institutions. According to Aybay and Oguz
(2002), in the last ten years, many universities and
higher education institutions have started offering
distance education courses for their on-campus
students due to the following purposes:
 The university gains experience on online
course development.
 The university gains experience on the
management of online programs and this
possibly leads to the establishment of an
institute or center specialized in distance
education method.
 The students taking online courses return
quick feedback on the program.
sufficient number of teaching staff who are
qualified in developing distance educations
programs.
Considering the importance of all these
purposes, and believing that distance education will
play a more important role in the education system,
Open University Malaysia (Universiti Terbuka
Malaysia) or better known as OU Malaysia was
established on August 10, 2000 as the seventh private
university in Malaysia. Although incorporated as a
private university under the Private Higher Education
Institutions Act 1996, the University leverages on the
quality, prestige and capabilities of its strategic partners (a
consortium of the 11 public universities), OUM adopts the
motto `University for All' which is consistent with its
philosophy on democratization of education. This
philosophy underlies the belief that education should be
made available to all, regardless of time, place, age and
social economic background. Open and Distance Learning
(ODL) mode is specifically targeted at working adults who
need to support of their families as well as manage their
career while pursuing higher education. Located in the
heart of Kuala Lumpur, the university opens to all
qualified candidates to further their study in various fields
with the OUM's vision to be a leader and innovator in
open learning. From its first intake of 753 students in
August 2001, it boosts to a total of 25,000 students. There

  73

 
 Volume 2 No. 2
Journal of Emerging Trends in Computing and Information Sciences
©2010-11 CIS Journal. All rights reserved.
http://www.cisjournal.org
are 16 programs offered, 2 Diplomas, 11 Bachelor degrees
and 3 Masters.
The objective towards accomplishing the study
which are:
To measure the effectiveness of the OUM's Learning
Management System using four factors below
 Easy to learn
 Error Tolerance
 Speed
 Quality
These four factors derived from Shackel’s
Usability Model. According to Shackel (1991) usability is
a property of a system or a piece of equipment. The
property is not constant, being relative in relation to users,
their training and support, tasks and environments. Thus,
the evaluation is context-dependent. The system or piece
of equipment may be usable if it matches the combination
of users, tasks, and environment. For a system to be usable
it has to achieve defined levels on the following scales:
 Effectiveness, meaning the results of
interaction in terms of speed and errors;
 Learnability, meaning the relation of
performance to training and frequency of use, i.e.
the novice user's learning time with specified
training and retention on the part of casual users;
 Flexibility, allowing adaptation to tasks and
environments beyond those first specified;
 Attitude, meaning "acceptable levels of human
costs in terms of tiredness, discomfort, frustration
and personal effort".
So, researcher chooses only one characteristic of
Shackel’s Model which is effectiveness. This effectiveness
consists of four factors such as easy to learn, error
tolerance, speed and quality.
II. PREVIOUS RESEARCH
According to Porter, L. (1997), on-line learning
may take place in the classroom or workplace, alongside
other activities, or may be undertaken at home or at online
access centers in the community. On-line learning
provides hands-on experience using information
technologies and makes it easier for learners to access
learning and customize study. It may also refer to a range
of approaches from making resources available
electronically, to the creation of rich interactive on-line
experiences involving synchronous and asynchronous
communication.
In Malaysia, the wave of e-learning is blowing
across the entire education system from public and private
institutions (Ndubisi, 2004). According to Iahad, N.,
Kalaitzakis, E., Dafoulas, G.A and Macaulay, L.A.(2004),
ISSN 2079-8407
e-learning refers to the process of learning or training
online, meaning acquiring knowledge via Internet and the
Web. There are three main factors that encourage the
higher education institutions to get involved with e-
learning such as the increasing number of students, the
need for life-long learning and the need to prepare
students with knowledge and skills for succeeding in the
knowledge economy ( Harasim, 1995) as cited in Iahad et
al. (2004).
According to Relan and Gillani (as cited in
Lewis, 2002) the online learning environment are broadly
interpreted as an instructional delivery method in which
the Internet is the medium of the delivery. There are two
ways to delivery online learning such as:
 Synchronous: communication in which interaction
between learner-instructor and learner-learner is
simultaneous.
 Asynchronous: communication in which
interaction between learner-instructor and learner-
learner does not take place simultaneously (anytime,
anyplace).
Open and Distance Learning is a new way of
learning in Malaysia. It began in 1972 when University
Sains Malaysia introduced open and distance learning.
Since today, ten public universities in Malaysia offered
open and distance learning, followed by Open University
Malaysia (OUM), the seventh private university in
Malaysia that fully based operate on open and distance
learning environment. To define Open and Distance
Learning, we must divide the term into two parts. Open
learning and distance learning, two systems combined
together.
Open learning means that the chance for learning
is opened widely to anyone who want to learn in the
tertiary stage. The needs for tertiary stage of education
have given way for open learning. According to Harry
(1997), the term `Open Learning' stands for general aim of
opening up education and training more widely. Openness
has many dimensions and most projects that describe
themselves as open learning.
According to Linda (2001), traditional classroom
lessons are often teacher directed, cannot be interesting
and engaging, often limited to textbooks, sometimes
thought of as boring by students, can be found on the web
but resources are not readily available, and teachers
sometimes lack the initiative to experiment with varied
methods of teaching the curriculum. A traditional
classroom might be found in the workspace or an office
building as well as at home or at field site. What is
traditional as that there is an established place, where
learners meet with one or more teaches or the material
they have create and setup to work interactively with
students and learning occurs. It may contain computer
equipment but it may be picture as an older whiteboard,
classroom full of desks or table with teacher lecturing
  74
 Volume 2 No. 2
Journal of Emerging Trends in Computing and Information Sciences
©2010-11 CIS Journal. All rights reserved.
http://www.cisjournal.org
students face to face. One of the things in a traditional
class is that the student can ask questions, make
comments, interact in verbal discussions, or collaborate
with small groups.
Figure 1: The comparison between e-learning and
traditional method. (Adapted from
http://researchers.in.th/profile/sudjai)
Figure 1 above shows that the comparison in
both traditional and e-learning method. It is can be seen
clearly that the process of learning can be done easily by
internet. Students can interact with their lecturer
interactively 24 hours per day and 7 days per week via e-
mail and on-line discussion asynchronously, by chatting
synchronously as well as can confidentially collect
programs and respond via LMS, otherwise in traditional
classroom , the contents of the lectures are manually
driven. Once the class ends the lectures cannot be
archived by students. Effectiveness is measured in
terms of speed and error, refers to levels of user
performance (Lindgaard, 1994; Shackel, 1991).
Effectiveness quantitatively measures the ease of
using the system, either by speed of performance or
error rate. Effectiveness means the capability of, or
success in, achieving a given goal. Contrary to
efficiency the focus of effectiveness is the
achievement as such, not the resources spent, so
anything that is effective is not necessarily efficient,
but anything that is efficient also has to be effective.
Learning Management System provides an
integrated platform for content, delivery, and
management of learning, as well as accessibility by a
range of users that may include learners, content
creators, and administrators. An LMS acts as the
central component of an enterprise e-learning
ISSN 2079-8407
implementation. According to Schar and Krueger
(2000), an LMS must be capable of handling various
delivery modes and automate the cumbersome
process of learner enrollment, registration, records,
transcripts, schedules, and reports, and it must
incorporate evaluation, assessment, and testing
capabilities. Learning Management Systems have
been widely adopted by institutions and instructional
designers in order to fulfill certain needs and
requirements in a field of ever increasing demands
for effective, fast and pedagogically correct
education and training. LMS that are in use today are
either commercial products such as WebCT,
Blackboard, Intralearn, open-source projects such as
customized software systems that serve the
instructional purposes of particular organizations
(Avgeriou, Papasalouros, Retalis, Skordalakis 2003).
The users of LMS can be classified into three
categories:
 The learners that use the system in order to
participate through distance (in place and/or
time) to the educational process. In fact, the
learners are the focal users of LMS, in the
sense that these systems are being developed
in order to satisfy some of their needs and
resolve their problems.
 The instructors, being the teachers and their
assistants that use the system in order to
coach, supervise, assist and evaluate the
students (e.g. notify for important issues on
an electronic notice board, engage in
discussions in electronic for a communicate
and exchange personal messages with
students, collect, assess and return
deliverables, etc. ).
 The administrators of the system, who
undertake the support of all the other users of
the system and safeguard its proper
operational status
The research framework for this research is
based on the Shackel's Model of Usabiliry (1991) as
shown in Figure 2. But the researcher study based on
one of the characteristic of Shackel's model which is
effectiveness. The model is derived from Shackel's
definition of usability as illustrated below. Shackel
was one of the first Human Computer Interaction
(HCI) researchers to seriously tackle the question on
"What is a usable product or application?" According
to Kissane and Finn (2003), he offered the following
simple definition;
"A usable product is one that users find satisfaction
for the tasks for which it was designed".
  75
 Volume 2 No. 2 ISSN 2079-8407
Journal of Emerging Trends in Computing and Information Sciences

©2010-11 CIS Journal. All rights reserved.

 
http://www.cisjournal.org

Figure 2: Shackel's definition of Usability (1991)

Shackel proposes that usability can be seen in terms of
four operational criteria such as effectiveness; learn
ability, flexibility and attitude (Benyon, 1993, Folmer &
Bosch, 2004).
 Effectiveness - performance in accomplishment
of tasks by some percentage of the users within
some proportion of the environments in which Figure 3: The Research Theoretical Framework
the systems will operate and measure by a
characteristic of the interaction such as the time III. COMPONENTS OF WORK
taken to complete the task or the number of errors SITUATION
made.
 Learn ability - degree of learning to accomplish
The product acceptance is determined by the
tasks depends on some specified level of
context in which it is used such as the particular users,
competence whether it is given a specific amount
tasks and environment. In Human Computer Interaction
of training or not. Learning also includes the time
(HCI), the system is the computer system. Meanwhile,
taken to relearn the system if details are
environment includes physical aspects such as appropriate
forgotten.
heating, lighting, equipment layout, operating
 Flexibility - adaptation to variation in tasks and
circumstances as well as psychological aspects such as the
environments which can be accommodated by the
provision of help and training and socio-political features
design.
such as the organizational environment in which the
 Attitude - user satisfaction with the system interaction take place. The relationship between the
whether to continue use the system or enhance Shackel's Model of Usability and the four components is
their use of the system illustrated in Figure 3. The product acceptance is
measured by the usability factors such as effectiveness,
This study involves the testing on the learnability, flexibility and attitude in particular
effectiveness of the Learning Management System. At this environment.
phase, tasks list and questionnaire are used to gather the
data. The task list is translated into scenarios based on the
objectives of the testing. Shackel (1991) identified four IV. RESEARCH DESIGN
factors on usability which are effectiveness; learn ability,
flexibility and attitude. So, based on this model, the The researcher conducted a testing to evaluate the
researcher chooses the effectiveness, one characteristic in effectiveness of OUM's LMS. The testing approach
Shackel's Model to complete this research. Four types of requires students or users of the LMS to work on typical
questionnaire are designed to determine the effectiveness tasks using the real system. The result from the testing can
of the Learning Management System. Figure 3 indicates be used to show how the user interface, speed, quality and
that the research methodology framework used to evaluate the overall of the LMS can supports the users in their
the OUM's Learning Management System which is learning process as an Open and Distance Learners.
categorized into two areas of study, theoretical and Furthermore, Genise (2002) stated the results of the testing
exploratory. In theoretical study, the researcher review a can also be used by the designers to make any changes on
lot of literatures from various sources such as book, the system design. The testing technique used in this study
journal, articles whether printed or on the web. Whereas in is questionnaires.
exploratory study, the researcher formulate the research by
identifying variables, constructs questionnaire based n one V. VARIABLES
of the usability attributes of Shackel's Model (1991) which
is effectiveness and then the tasks list for testing is The variables used in this study can be classified
designed. Data collection is gathered by conducting the into two types such as independent variables and
testing and questionnaire given to the respondents.

  76

 
 Volume 2 No. 2 ISSN 2079-8407
Journal of Emerging Trends in Computing and Information Sciences

©2010-11 CIS Journal. All rights reserved.

 
http://www.cisjournal.org

dependent variable. Table 6 lists all the independent and
dependent variables which will be used for analysis.
VI. TESTING
In evaluating the OUM's LMS, three main tasks
with a total of four to five sub tasks were formulated based
on the three main features or functions of the system such
as to test student's performance by doing an on-line quiz
for the course that they registered, to join the discussion
forum by posting and response to their peer's message and
finally to find one book by giving them the title of the
book. All the tasks should be completed within fifteen
minutes or five minutes each tasks. After completing
every section of the task the subject has to give the
comments.

Table 1: The set of tasks

Task 1 Task 2 Task 3

1. Select one of the course 1. Post one message asking your 1. Click myLibrary
that you have registered tutor about the subject that button
2. Click on the Quizzes you don’t understand (you 2. Find a book entitled
button in LMS can choose any subject that “Taking Control with
3. Answer the test within 5 you’ registered) Time Management” in
minutes 2. Response to one of your e-Books collection
4. Submit the result friend’s message in LMS and (24x7 database)
5. Comment on Task 1 discuss about the topic raised 3. Display your result
(you can choose any subject 4. Search the book within
that you’ registered) one minutes
3. Do both question 1 and 2 in 5 5. Comment on Task 3
minutes
4. Comment on Task 2
 

VII. SUMMARY AND CONCLUDING
REMARKS
The result of this research indicates that the
overall level of the effectiveness construct in student’s
point of view is in the acceptable level. The mean score
for the overall of the effectiveness on easy to learn factor
is 3.8193 while on the error tolerance factor the mean
score for the effectiveness of OUM's LMS is 3.7383. For
the effectiveness of LMS in OUM on speed factor the
overall mean score is 3.8925 and 4.110 mean score's for
the effectiveness on quality factor which is the main factor
that contribute to the effectiveness the OUM's LMS.
From this study it showed that the Learning
Management System of Open University Malaysia is
used effectively by students although students came
from various levels of educational background, age,
marital status, course, level of thinking, computer
skills an d so on, although novice users at first they
seemed lost and cannot move around the system but
lastly they can used it effectively parallel with their
title as open and distance learners. The intermittent
users at first done several errors but they can recover
the errors easily as well as the expert users, they
were frequently used the system so they do not have
any problem in using the Learning Management
System. From the study also, it showed the OUM's
Learning Management System is easy to learn for
different levels of users, can easily recover from
errors once users made errors, can speed up learning
process as well as has a quality for students of Open
University Malaysia.
REFERENCES
[1] Avgeriou, P., Papasalouros, A., Retalis, .S,
Skordalakis, M. (2003). Towards a Pattern
Language for Learning Management Systems.
Educational Technology & Society, 6(2), 11-
24, Available at
http://ifets.ieee.org/periodical/6-2/2.html
[2] Aybay, I & Oguz,D. (2002).A Learning
Management System Developed at the Eastern
Mediterranean University: The Turkish Online
Journal of Educational Technology, Vol 2(2),
pp: 1303-652
[3] Barnum, C.M. (2002). Usability Testing and
Research. USA: Pearson Education.

  77

 
 Volume 2 No. 2
Journal of Emerging Trends in Computing and Information Sciences
©2010-11 CIS Journal. All rights reserved.
http://www.cisjournal.org
[4] Beck, E.(2005). Education Research Institute
University of Oslo In: Berthelsen. Bouvin,
Krogh and Kyng (Eds.) Critical Computing-
Between Sense and Sensibilities. Proceedings
of the Fourth Decennial Aarhus Conference,
Aarhus, August 20-24, 2005 (ACM, New York,
2005, pp. 173-176).
[5] Bouras et al (1999). Efficient Data
Management to Support Educational Activities.
In: International Workshop ICL99, Interactive
Computer aided Learning - Tools and
Application, Villach, Austria, 6-8 October.
[6] Carter, B. (1999) Effective Integration of
Multimedia Technology Into Scientific
Thematic Units Brannon Carter ( ED 608) -
Expanding Reading Ability Through Content
Area Instruction The University of Alabama in
Huntsville
[7] Chapanis, A. (1991). Evaluating Usability.
Human Factors for Informatics Usability Eds.
S. B. and S. J. Richardson. Cambridge,
Cambridge University Press: 359-395.
[8] Feldman, M.(2002) Media Reviews: LMS
Breakdown in Ebsco Publishing Folmer, E._ &
Bosch, J. (2004). Architecting for Usability ; A
Survey. Journal of Systems and Software, 70
(1-2), 61-7
[9] Gaines, B.R. (1988). A Conceptual Framework
for Person-Computer Interaction in Distributed
Systems. IEEE Transactions Systems, Man &
Cybernetics SMC-18(4) 532-541.
[10] Genise, P. (2002). Usability Evaluation:
Methods and Techniques. Version 2.0
Retrieved August 19, 2005 from
http://www.cs.utexas.edu/users/almstrum/cs370
/elvisino/usaEval.html
[11] Hall, J (2003 ) Assessing Learning
Management System, senior vice president of
Oracle University, the Education and Training
Division of Oracle Corporation.
[12] Harasim, L, (1995) Learning Networks; a Field
Guide to Teaching and learning Online
Cambridge, Ma: MIT
[13] Harry K (1997b) Research Reviews and
Agendas, Section 5, Researching the Literature
on Open and Distance Education, Block 3,
H801, Foundations of Open and Distance
Education, The Open University.
ISSN 2079-8407
[14] Iahad, N., Kalaitzakis, E., Dafoulas, G.A., &
Macaulay, L.A. (2004). Evaluation of Online
Assessment : The role of feedback in Learner-
Centered elearning. Proceedings of the 37~'
Hawaii International Conference of System
Sciences. Available in IEEE database.
[15] Kissane, A., & Finn, E. (2003), Development
of a Usability Model for VLE Deployment.
ILTA, EdTech 2003.
[16] Lewis, B.A. (2002). Learning Effectiveness:
Efficacy of Quizzes vs.Discussion in Online
Learning. Master Thesis from UMI
Dissertation Services.
[17] Lindgaard, G. ls` ed. (1994). Usability
Testing and System Evaluation: A guide for
designing useful computer system. UK;
Chapman & Hall.
[18] Maish Nichani (2001) e-Learning Post.
Retrieved from; maish c~elearninQost.com[
citied October 11, 2005]
[19] Maxwell, L. 1995. Integrating Open Learning
and Distance Education. Educational
Technology November-December, 43-48.
[20] McCormack, C., & Jones, J. D. (1997).Building
a Web-based Education System, New York:
Wiley Computer Publishing.
[21] Moore, M. G., & Kearsley, G. (1996). Distance
Education: A Systems View, Belmont, CA:
Wadsworth Publishing Company.
[22] Ndubisi, N.O. (2004). Factors Influencing e-
Learning Adoption Intention : Examining the
determinant structure of the decomposed
theory of planned behaviour constructs, 252-
262.
[23] Nielsen, J. (1990a). Evaluating Hypertext
Usability. In: D. H. Jonassen, & H. Mandl
(Eds.), Designing hypermedia for learning,
Berlin: SpringerVerlag, 147-168
[24] Oakes, K (2002) e-Learning, Aspen Enterprise
Learning Paltform. Retrieved from:
Kevin.oakes (~a,click2learn.com[citied
October 4, 2005 ]
[25] Ponta, D., Scapolla, A.M., Taini, M. (1996).
Telematics for Education: The Design of a
distributed Computer-based Collaborative
Learning System. In: Proceedings of ED-
  78
 Volume 2 No. 2 ISSN 2079-8407
Journal of Emerging Trends in Computing and Information Sciences

©2010-11 CIS Journal. All rights reserved.

 
http://www.cisjournal.org

TELECOM 96, Boston, MA,USA, 17-22 June
1996, pp. 252-257~
[26] Quint, V (2001) The user Interface Domain
retrieved from available on
http://www.w3.org/LJI/ a retrieved 14 April
2006
[27] Ravi Dharbamulla & Pamela Lawhed, (2004)
Paving the Way Towards an Efficient
Learning Management System ACMSE'04,
April 2-3, 2004, Huntsville, Alabama,
USA.Copyright 2004 ACM 1-58113-870-
9/04/04
[28] Sandra ,C.(2005)Distance Education Report :
Benefits of Upgrading to LMS, May 2005
[29] Salter, D., Richards, L & Carey,T (2004).
Educational Media International: The'TS'
design Model: An Instructional Model and
learning Environment to support the
Integration of Online and Campus-Based
Courses, Council for Educational Media
Canada
[31] Shneiderman, B. (1998) Designing the User
Interface: Strategies for Effective Human-
Computer Interaction (3rd ed ), Reading, MA:
Addison-Wesley
[32] Tella S. 1997. An 'Uneasy Alliance' of Media
Education and Multiculturalism, with a View to
Foreign Language Learning Methodology. Media
Education Centre. University of Helsinki.
Department of Teacher Education. OLE
Publications 4, pp. 14-17].
[33] Welsch, E.(2005) SCORM: Clarity or Calamity?
Available at
http//www.editor@onlinelearningmag.com
Retrieved on December 12, 2005
[34] Prasert Sudjaipraparat (2006), E-learning:
concepts and practice.Available at
http//http://researchers.in.th/profile/sudjai
Retrieved on December 12, 2006

[30] Shackel, B. (1991). Usability--context,

Framework, Definition, Design and
Evaluation. Shackel, B. and Richardson, S.,
Ed. Human Factors for Informatics Usability.
pp.21-37. Cambridge, UK, Cambridge University
Press.

  79