Using learning analytics to predict success and failure in

blended learning
Matti KOIVISTOa
a
Principal Lecturer, Ph.D., Mikkeli University of Applied Sciences,School of Engineering, Mikkeli, Finland,
matti.koivisto(a)mamk.fi

Abstract

During the last decades,universities and colleges have started to apply different forms of educational data mining techniquesto
identify factors that predict students ability to complete individual courses and full study programs. Typically, thesestudies are
based on data collected from Virtual Learning Environments (VLEs) and the main area of interest has been in e-learning or
Massive Open Online Courses (MOOCs). In this paper, the focus is instead on blended learning. The aim of the study is to find
out possible success and failure indicators on a blended learning course paying special attention in students activity both in face-
to-face and online settings. The paper begins with a short introduction to the key concepts of the study. It goes on to describe the
design and results of the empirical study carried out in a Finnish university. Finally, findings of the study are discussed and the
areas of further research are identified.

Keywords:leraning analytics, blended learning, active learning, virtual learning environment

1. INTRODUCTION

In colleges and universities, students are expected to manage their own learning. New learning methods
likeonline and blended learning highlight this and they require new kind of learning strategiesboth from the
institutions and students. New environment with higher level of self-regulationdoes not suit to all students as
indicated with an ever-increasingnumber of university dropouts. Low college completion rates do not affect
onlynon-graduating students and universities but also the society as a whole. For example, Leonhard [1] added
public universities together with the Wall Street firms and regulatory agencies on a list of the organizations whose
failures have done most damage to the American economy in recent years. Mainly because they have failed in their
core mission turning teenagers into educated college graduates. The lowgraduation rates are by no means limited
only to the USA but it is a global phenomenon as similar numbersare reported for example in the UK [2] and in
Australia [3].
During the last decades, universities have started to apply educational data mining techniques to develop
different kinds of systems to predict student academic success, to identify at-risk students and allow more timely
pedagogical interventions [4]. The main reasons for adopting learning analytics include need for financial and
operational efficiency, adaptation to new funding models and ability to respond todemands for greater accountability
[5].
Data mining in educational institutions can take many different forms with different focuses and aims.The aim of
the paper is to use learning analytics to reveal possibleacademic success and failure factors on blended
learningcontext.In this case, blended learningis definedas a learning strategy in which a portion of the traditional
face-to-face instruction is replaced by online learning. Thus blended learning is located between pure online and
offline strategies and above all it focuses on learning not on teaching. This paradigm shift from teaching to learning
has been going on for decades and it has thoroughly changed our understanding of the role of the academic
institutions (see e.g. [6] for more details). Today students are no longer considered as passive listeners but active

52
owners of the learning process. This kind of thinking often called as active learning is today the mainstream
educational method in modern universities. Thus, it is also the pedagogical core of this study.
The structure of the paper is as follows. Chapter 2 describes shortly the key concepts of the study: active learning
and learning analytics.Chapter 3 contains the introduction to the empirical part of the study carried out in a Finnish
university as well as the results of the study. Findings of the study are discussed in Chapter 4. Finally, Chapter 5
concludes the paper with summarizing the main findings, pointing out the limitations of the study and identifying
the areas of future studies.

2. Key concepts

2.1. Active learning

Active learning is not a new concept in the literature of education.It became increasingly popular after the
Association for the Study of Higher Educations report describedthe following five characteristics of active learning
in college [7]:
Students are involved in more than passive listening

They are engaged in activities (e.g. reading, discussing, writing),

There is less focus on information transmission than on development on student skills,

There is greater emphasis placed on exploration of attitudes and values and

Students are involved in higher order thinking like analysis, synthesis and evaluation.

Naturally, many new definitions of active learning have been created after that (e.g. [8] and [9])but
generally,they all define it as an instructional method that engages students in the learning process and requires
students to do meaningful learning activities and think what they are doing. Some scholars have limited it only to the
activities introduced in the classroom [10] but today most scholars connects active learning to all kind of educational
settings including Massive Open Online Courses (MOOCs) [11], flipped classrooms [12] and collaborative learning
settings [13].
Activities are also an essential part of the blended learning. Li m et al. [14] found out that students considered
opportunity to use learning activities and assignments even more important in blended than in pure online learning.
In blended learning the activities can naturally be used both in asynchronous Internet-based and in synchronous
face-to-face contexts. The fundamental aim of the activities in both contextsis in achieving better learning outcomes
and enhancing students learning in a more practical manner. Garrison and Kannuka [15] points out that one of the
strengths of blended learning is its ability to facilitate a community of inquiry both in face-to-face and online
settings. In both cases, the community of inquiry consist of three elements: cognitive, social, and teaching presence
as shown in Fig. 1.

Fig. 1. Community of inquiry [15].

53
 Every educator must make his or her own decisions how to operate on both of these environments. Although
there is no single right way to organize learning activities in blended learning some suggestions have been made.
For example,Arabi [16] has pointed out that in blended learning students will benefit most if they go through the
online content in advance and then the activities in face-to-face sessionsare clearly mapped to this content.

2.2. Data mining in educational institutions

Data mining is a technology used to find out significant relationships and patterns among different variables. The
discovery of those associations can be examined by different methods including statistical, mathematical, artificial
intelligence etc.Education institutions like colleges and universities use different kinds of data mining solutions,
which allows teachers, course designers and administrators to search for unobserved patterns and hidden meanings.
Learning analytics has many possible meanings. Van Barneveld et al. [17]dug deep to a plethora of terms and
definitions on the field and they were able to identified three different types of educational data analysis, depending
mainly on the subject and purpose of the analysis (see Table 1). According to their classification, business and
academic analytics allows executives access to indicators in order to run the business of the institution. Learning
analytics instead focuses on two areas learning effectiveness and operation excellence. It focuses on the learner
and it is based on data collected from course management systems, virtual learning environments, student
information systems and other data sources. Predictive analytics serves all participants of the learning process
including students, teachers, departments and executives. It acts as a link between collected, reported and analyzed
data and intelligent actions based on them.

Table 1. Different forms of analytics in education [17].

Analytics Agents Focus

Business and academic analytics Management Business and execution

Learning analytics Departments and students Students and learning

Predictive analytics All participants Meaning and action

Agudo-Peregrina et al. [18] developed this model further paying special attention to data-driven decision
making. They highlighted the importance of top down approach in decision making as can be seen in Fig.2. They
also pointed out that the most common approaches of learning analytics focus on students academic performancein
order to manage student success.

Fig. 2. Data-based decision makingin educational institutions [18].

3. Study design and results

3.1. Study design and data collection

The aim of the study is to find out possible success indicators on a blended learning course. The study pays
special attention to correlation between students activity and performance on the course. The data collection took
place in an e-business course at a Finnish university. The total number of the students was 72 containing the second

54
year undergraduate engineering and business students of the university as well as exchange students from the
international partner institutions.
As stated earlier, in blended learning student activities take place both in classrooms and online.In the
experiment, the classroom activitywas measured with classroom tasks done and returned during the face-to-face
sessions. Students returned the tasks individually, but they we encouraged to do collaboration and co-operation, and
to use information from different sources. The total number ofthe classroom activities was 14 and the outcome of
the activities wasconvertedinto a single number, which was used as a metric of the classroom activity. The possible
value of the metric was from 0 to 100, in which the higher number pointed out higher activity. It is essential to point
out that the idea of classroom activities was not to test knowledge of the students but to give them possibilities to be
active collaborative learners. Therefore, the evaluation of the activitieswas not only based on the correctness or
quality of the work but included otherdimensions listed in Table 2.All element listed below had an equal weight and
evaluation included both quantitative and qualitative elements.

Table 2. Classroom activity evaluation criteriaand their weights

Element Type of the element Weight

Task done Quantitative 20 %
Number of references Quantitative 20 %
Collaboration Qualitative 20 %
Quality of the work Qualitative 20 %
Effort Qualitative 20 %

The students online activity was instead measured with data collected from the universitys Virtual Learning
Environment (VLE). The data contained log entries on different events including enrolment to the course, viewing
the course material, task submissions etc. The total number of log entrieswas usedto measurea students online
activity. The number of events was converted to a single value ranging from 0 to 100 so that the student with highest
number of events got the maximum value.The values of the other students were calculated with Formula 1.

Number of the students online events

Students online activity metric = ----------------------------------------------------- * 100 (1)
Maximum number of events by any student

3.2. Results

Table 3 shows descriptive statistics for final score, classroom activity, online activity and Grade Point Average
(GPA). The final score was the result of the final exam and GPA number were calculated from the universitys
Student Information System.Also four metrics had possible values between 0 and 100 butbecause they are
measuring totally different things their values cannot be compared against each other. Therefore, the lower mean of
online activity does not indicate that students in general were less active online than in classroom environment.

Table 3. Collected data (N=72)

Final score of the blended Classroom Online Grade Point

learning course activity activity Average
Mean 81.9 78.0 50.1 75.2
SD 8.7 22.2 18.8 14.2

The Pearson correlation coefficients between these four variables are show in Table 4. Because the focus of the
study is to identify the possible success factors, we are paying our attention to the correlation between the final score
and other metrics (line 1). The numbers are suggesting that there is a moderate positive correlation (r>0.3) between
both forms of activity and the final score and strong correlation (r>0.5) between the GPA and final score. So, these
results are indicating that earlier success in studies is stronger predictor of doing well in the blended learning module
than activity in the classroom or in the online environment.

55
 Table 4. Correlations matrix (N=72)

Final score Classroom Online activity GPA

activity

Final score 1.00 0.47 0.33 0.53

Classroom activity 1.00 0.49 0.44
Online activity 1.00 0.23

GPA 1.00

The student population wasthen further divided into two subgroups: second year degree students of the
university (n=41) and exchange students from the partner universities (n=31). The means and correlations for both
groups are shown in Tables 5 and 6.According to Table 5 the only statistically significant difference between the
subgroups were in online activity. The exchanges students seemed to be a bit more activeusers of the VLE than
universitys own degree students were. Although the reason for this was not analyzed further, one possible
explanation could be that exchange students with less experience of the local systemwere looking for more
information about rules and working principles than the own students.

Table 5. Means of the universitys own degree students (n=41) and exchange students (n=31)

Final score Classroom activity Online activity GPA

Own students 82.1 73.6 45.3* 74.2

Exchange students 79.9 83.8 56.6* 76.6
* p<.05

Table 6. Correlations between final score and used variables of both subgroups

Final score Classroom activity Online activity GPA

Own students 1.0 0.47 0.45 0.66

Exchange students 1.0 0.49 0.32 0.56

One of the main areas of interest for faculty members and university administrators is how to help students
succeed academically and therefore it is essential to analyze the possible indicators of failure. In this blended
learning course totally 12 students out of 72 failed the course. The data of the failed students (Table 7) reveals that
not only the final score but also both forms of activity were significantly lower than for students who passed the
course.

Table 7. Failed students results

Final score Classroom activity Online activity GPA

Mean 19.5* 28.3* 15.7* 54.8*

Correlation 1.0 0.9 0.77 0.15
* p<.01

4. Discussion

The results of the study are indicating that activity both in classroom and in virtual learning environment has a
positive effect to academic success in blended learning courses. However, it is essential to point out that the
collected data is only showing moderate correlations between different form of activity and final score. Instead,
success in earlier university courses seem to have a stronger effect to final grade than activity. This finding is not a

56
big surpriseas other scholars (e.g. [19], [20], [21]) have reported similar results.Similarly, the numbers of the failed
students show that low activity levels are strongly correlating with low scores.
One essential application area of learning analytics is the development of early warning systems that can be used
for identifying students who are in danger to fail. The classroom and online activity were measured during the whole
course and in their current form they are not very suitable to be the early indicator of possible problems. One
possible candidate for an early warning signal was identified in this study -the VLE enrolment time. In the
university, all students must have enrolled to the courses they take during the previous semesters enrolment period.
This enrolment takes place in the Student Registration System. In addition to that, all students must enrollto the
course onthe VLE with the enrolment key provided by the teacher. Enrolment in the VLE is not an administrative
issue but provides access for a student to the course materials, tasks, and all other needed resources.Students can do
the VLE enrolment at any time during the course but naturally,they are expected to do that in the beginning of the
module. The collected VLE data includes among other things the date and time of the VLE registration for each
student. Table 8 shows failing and passing rates for two groups of students: the students joining the course in the
VLE immediately in the beginning of the module and students not doing it during the first two days.The numbers
are clearly suggesting that students enrolling later have much higher probability to fail the course.

Table 8. Failing and passing rates for students enrolling at the beginning of the course and later

Students enrolling later Students enrolling at the beginning

(n=19) (n=53)
Failing rate 52.6 % 3.8 %
Passing rate 47.4. % 96.2 %

It was surprising to notice that even a small delay (like three days) was enough to have a strong negative
effect.Of course, it is extremely unlikely that the minordelay in enrolment is the actual reason behind the higher
failure rate but it is rather an indicator of some profound factor of attrition. In this study, the reasons of attrition were
not analyzed in more details. Other scholars have analyzed factors associated with student persistence in online,
face-to-face and blended learning courses and study programs. For example, Hart [22] had identified in her integrate
literature review the following factors as being related to student persistence in courses: satisfaction with learning,
sense of belonging to a learning community, peer and family support, time management skills, and communication
with the instructor. The link between early warning signals and attrition factors is an essential area of further study.
The concept of the community of inquiry (see Section 2) and its three elements cognitive, social, and teaching
presence could serve as a solid framework for these studies.

5. Conclusions and future steps

All stakeholders of the learning process have a high interest to find out the factors that predicts students ability
to complete courses and study programs despite obstacles or adverse circumstances. The results of this study suggest
that activity during studies both in online and face-to-face settings have a moderate positive effect to the academic
performance. The results also give a strong support to the earlier findings of the causal connection between the lack
of study relatedactivity and attrition. One of the most important outcomes of the study was identification ofa
possible candidate as an early warning indicator. The study reveals that late VLE enrolment increases students risk
to fail the course remarkably.However, it is important to bear in mind that late enrolment could at best be an
indicator of possible problems not the actual cause.
All studies have their limitations and this study is not an exception to this rule. First, the study was limited only
to one blended learning course with a relatively small sample size. Second, there are number of reasons why college
students drop out from courses and degree programs. Earlier studies have indicated clear differences between
countries, universities, demographic and age groups as well as between part and full-time students (see e.g.
[23]).Therefore further studies are still needed to verify the findings of the study, to understand the actual reasons
behind the poor academic performance and finally to reach the ultimate goal of the predictive learning analytics to
make intelligent and well-timed actions to support students.

References

[1] Leonhard, D. (2009). Colleges Are Failing in Graduation Rates. The New York Times, Sep. 8th.

57
[2] Havergal, C. (2016). Rise in UK universitydroupout rate disappointing. Times Higher Education. Mar. 23rd.
[3] Hare, J. (2015). Access Broadens, Dropouts Soar. The Australian, Aug. 5th.
[4] Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an early warning system for educators:
A proof of concept. Computers & Education, 54(2), 588-599.
[5] IBM (2009). Managing the Business of Education. White paper.
[6] Barr, R. &Tagg, J. (1995). From teaching to learning a new paradigm for undergraduate education,
Change,Nov./Dec., 13-25.
[7] Bonwell, C. &Eison, J. (1991). Active Learning: Creating Excitement in the Classroom. ASHE-ERIC Higher
Education report.
[8] Felder, R. & Brent, R. (2009) Active learning: An introduction. ASQ Higher Education Brief,2 (4).
[9] Collins J. &O'Brien N. (ed.) (2011). The Greenwood dictionary of education (2nded.). Santa Barbara:
Greenwood.
[10] Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93
(3), 223-231.
[11] Martin F. (2012). Will massive open online courses change how we teach? Communications of the ACM, 55 (8),
26-28.
[12] Pierce, R. & Fox, J. (2012).Vodcasts and active-learning exercises in a "flipped classroom" model of a renal
pharmacotherapy module.American Journal of Pharmaceutical Education, 76 (10), 196.
[13] Goria, C., Speicher, O. &Stollhans, S. (2016). Innovative language teaching and learning at university:
enhancing participation and collaboration. Dublin Ireland: Research-publishing.net
[14] Lim, D. H., Morris M., &Kupritz, V. (2007). Online vs. Blended learning: Differences in instructional outcomes
and learner satisfaction. Journal of Asynchronous Learning Networks, 11 (2), 27-42.
[15] Garrison, R. &Kanuka, H. (2004). Blended learning: Uncovering its transformative potential inhigher
education. Internet and Higher Education, 7, 95 105.
[16] Arabi, E. (2014). How To Optimize Blended Learning Through Learning Activities. eLearning Industry.
[17] Barneveld, A., Arnold, K. & Campbell, J. (2012). Analytics in Higher Education: Establishing a Common
Language. ELI Paper 1: 2012.
[18] Agudo-Peregrina, ., Iglesias-Pradas, S., Conde-Gonzlez, M. & Hernndez-Garca, . (2014). Can we predict
success from log data in VLEs? Classification of interactions for learning analytics and their relation with
performance in VLE-supported F2F and online learning. Computers in Human Behavior, 31(1), 542-550.
[19] DeBerard, M. S., Spielmans, G. I., &Julka, D. L. (2004). Predictors of academic achievement and retention
among college freshmen: A longitudinal study. College Student Journal, 38, 6680.
[20] Stumpf, H., & Stanley, J. C. (2002). Group data on high school grade point averages and scores on academic
aptitude tests as predictors of institutional graduation rates. Educational and Psychological Measurement, 62,
10421052.
[21] Harrell, I. & Bower, B. (2011). Student characteristics that predict persistence in community college online
courses. American Journal of Distance Education, 25 (3), 178 191.
[22] Hart, C. (2012). Factors associated with student persistence in an online program of study: A review of the
literature. Journal of Interactive Online Learning, 11 (1), 19 42.
[23] Shapiro, D., Dundar, A. Yuan,X., Harrell, A. &Wakhung P. (2014). Completing college: A national view of
student attainment rates Fall 2008 Cohort. Signature Report No. 8. Herndon, VA: National Student Clearing
House Research Center.

58