See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/330222999

Performance Analysis of Machine Learning Classifiers for ASD Screening

using Rapidminer

Conference Paper · November 2018

CITATIONS READS

0 154

5 authors, including:

Alrence Halibas
Gulf College Oman
15 PUBLICATIONS   9 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Graduate Attributes and Learning Outcomes: Proposed Framework and Assessment Process View project

Determining the Intervening Effects of Exploratory Data Analysis and Feature Engineering in Telecoms Customer Churn Modelling View project

All content following this page was uploaded by Alrence Halibas on 03 March 2019.

The user has requested enhancement of the downloaded file.

 Performance Analysis of Machine Learning
Classifiers for ASD Screening
Alrence Santiago Halibas
Faculty of Computing Sciences
Gulf College
Muscat, Oman
alrence@gulfcollege.edu.om
Erbeth Gerald Tongco Delvo
College of Computer Studies
La Salle University
Ozamiz City, Philippines
erbethgerald.delvo@lsu.edu.ph
Abstract— Several machine learning classifiers have been
used for Autism Spectrum Disorder screening, however,
literature in finding the best classifier for this application
domain is inadequate. Hence, this paper presents a comparison
of five (5) supervised machine learning algorithms: Decision
Tree, Naïve Bayes, k-nn, Random Tree, and Deep Learning
using small datasets (n=1100) on child, adolescent and adult
ASD screening in finding the most appropriate classifier. These
algorithms, which are evaluated using a broad set of prediction
performance metrics including accuracy, precision/recall
measures, and Receiver Operating Characteristics, are
compared against each other. The experiment result suggests
that the Deep Learning classifier gives the best performance
(with more than 96%) in almost all metrics while the Random
Tree classifier came out as the least performing classifier in all
the performance metrics.
Keywords— Autism, Bioinformatics, Machine Learning, Binary
Classification, Deep Learning
I. INTRODUCTION
Autism, or otherwise known as Autism Spectrum
Disorder (ASD), is a complex mental condition that is
exhibited from early childhood and primarily characterized
by communication and social difficulties. It affects 1 in 150
children worldwide [1]. Early diagnosis and intervention are
seen to be the best treatment for this disorder, hence,
increasing interests and approaches in autism understanding
and diagnosis are prevalent [2].
Nowadays, machine learning algorithms are rapidly used
in medical science [3] that transforms biomedical data into
valuable knowledge. It is widely used in bioinformatics to
build predictive models for detection and diagnosis of
diseases [4], medical image segmentation [5], gene finding,
protein folding prediction, and so many others [3].
Supervised machine learning is now increasingly applied to
various bioinformatic problems [6]. Research on machine
learning to enhance autism diagnosis is seen to have a
potential and usefulness [7]. In fact, several studies on autism
diagnostics using machine learning have already been carried
out by [8], [9]. The ongoing medical research in this field is
attributed to an increasing availability of online data sets and
low-cost computing [10]. In this regard, this study referenced
the study of [11] on ASD Screening. An artefact of the
author’s work is a mobile application called ASD Test that is
available on Google Play and Mac App Store [12]. The
application allows the users to answer a 10 autism-related
questions and suggests the likelihood of having autistic traits.
The datasets of this application are utilized for this study.
XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE
Leslyn Bonachita Reazol
College of Computer Studies
La Salle University
Ozamiz City, Philippines
leslyn.reazol@lsu.edu.ph
Jannette Cui Tibudan
College of Engineering
St. Peter’s College
Iligan City, Philippines
jctibudan@gmail.com
Furthermore, this study attempts to determine the most
appropriate binary classification model for the given
datasets. It explores different machine learning algorithms
that can effectively classify whether a child, adolescent or
adult is likely an ASD candidate. Likewise, this paper tries to
get a good estimate and comparison of the algorithms’
prediction performance in terms of their accuracy and
classification error, precision and class recall, and Receiver
Operating Characteristics (ROC).
The succeeding sections of this paper are organized as
follows: Section II presents the background information on
the chosen machine learning algorithms and performance
metrics, Section III presents the experimental methods that
include the datasets and software used as well as the
performance analysis and results, and Section IV presents the
Conclusion and Future Work.
II. BACKGROUND INFORMATION
A. Supervised Machine Learning
Machine Learning is a branch of Artificial Intelligence
(AI) that learns and discovers meaningful patterns in data to
make predictions [13]. It is built on mathematical principles
of probability and statistics as well as computer science.
Supervised learning is a machine learning technique that
learns from mapping input variables and output variables and
uses this mapping for prediction [14]. Simply, it is learning
from examples having two sets of data, a training and a test
set [15].
A typical supervised learning problem, as shown in Fig.
1, contains instance space X contains objects or attributes, a
label space Y, and a prediction space Y’. [16] defines
classification as the task of learning a model that maps each
attribute x to one of the predefined class labels y.
INPUT OUTPUT
Classification
Attribute Set Model Class (or Label)
(x) (y)
Fig. 1 A Classification Model
A training sample (attribute) set is denoted as S =
((x1,y1),…,(xm,ym)) Є (X ×Y)m that contains a predetermined
number of examples where each xi Є X. The output is a
model hs : X  Y’ which learns from the sample set [17]. In
a binary classification problem, objects in the sample set are
associated with two classes (or labels), Y = {±1} = {-1,+1}
and the model predicts the class of new instances, Y’ = Y =
{±1}.
B. Classification Algorithms
The supervised classification algorithms that are relevant
to this study are discussed in the succeeding subsections.
1. Decision Tree
A decision tree consists of three (3) types of nodes, a root
node, internal nodes with one incoming edge and two or
more outgoing edges, and leaf or terminal nodes with only
one incoming edge and no outgoing edges. The root and
internal nodes contain the attribute test characteristics
whereas the leaf node contains the class label. A decision
tree is illustrated in Fig. 2.
Classification Model
(After training)
New
Example
? [22] provides distance computations found in (2) and (3).
Fig.2. Decision Tree classifier
A test example is classified in the decision tree by
continually applying the test conditions to the example
starting from the root node down to the respective branches
of the tree until it terminates at a leaf node.
A Decision tree affords a fast and effective method of
categorizing data sets [18]. It is usually used to classify when
response variables contain only two (2) categories. The
major advantage of a decision tree is its ability to present a
visual representation of the decision structure based on actual
values of the attributes, thus it can be very useful to extract
valuable insights. Furthermore, a decision tree can handle
categorical and numerical variables which are also present in
the datasets.
2. Naïve Bayes
Naïve Bayes is a learning classifier that is based on
Bayes rule for solving classification problem [16]. The term
“naïve” comes from the main assumption of conditional
independence of its features. Simply, this classifier assumes
that the features independently contribute to the probability
of the model which is illustrated in the succeeding formula,
where variable c is the dependent class label and F are the
several feature variables.
Equation (1) is based on Bayes Theorem for
classification where each attribute and class label are
considered as random variables. The goal is to predict the
class c and choose a value of c that maximizes
P(c|F1,F2,…,Fn) given an example with attributes (F1, F2,…,
Fn).
(1)
According to [19], Naïve Bayes is one of the effective
and efficient learning algorithms with good prediction
performance for machine learning and data mining. It can be
efficiently trained in a supervised setting. It also requires
little training data and can be likewise used for moderate and
large training data set having numerous attributes. Lastly,
this classifier can handle missing values.
3. K-nearest neighbors (k-nn)
k-nn is an instance-based machine learning method for
classification and regression. The k-nn is one of the most
common and simplest classification algorithms [20] with no
explicit training or model, thus, referred to as “lazy
learning”. This algorithm simply saves the training
observations and compares new observations using a
similarity measure and reference the saved data during test
time to make predictions [21]. This algorithm can easily
handle missing values.
There are several choices to compute distances
depending on the type of the features in the data. Mixed
distance measure is used if the data contain real and binary
values. If the data Euclidean distance is used for the real part
whereas Hamming distance is used for the binary part. Rai
The sample data set contains j attributes that forms a j-
dimensional vector, x=(x1,x2,…,xj) and the given training data
is represented by D={(x1,y1),…(xj,yj)} where xi is the input
and yi is the output/label. The xi is a vector that consists of D
features whereas xim denotes the m-th feature of x. Therefore,
the output is yi Є{1,...,C}
Euclidean Distance:
(2)
Hamming Distance:
(3)
The algorithm requires the distance function d(xi,xj) and
the value k which is the number of nearest neighbors. Kozma
[23] simply summarizes the k-nn algorithm as follows:
i. Calculate distances of text point from each training
point
ii. Sort the distances in ascending order
iii. Select k nearest neighbors
iv. Calculate majority voting
4. Random Tree
A Random Tree is much like a Decision Tree except that
for each split it performs, only a random subset of attributes
is available [24].
5. Deep Learning
Deep learning is a multi-layer feed-forward neural
network that learns from complex patterns of data. It is an
upgrade to neural network which contains an input layer,
large number of hidden layers, and an output layer to model
data much like the one illustrated in Fig.3.
Fig. 3 Feed-forward back propagation neural network
The input layer contains the input vectors xi =
[xi1,…,xip] T where p is the number of input variables.
Likewise, the hidden layer contains s neurons having
sigmoid transfer function (f1). Another sigmoid action
function (f2) is found in the output layer. Deep Learning uses
a backpropagation algorithm that defines changes in the
internal parameters [25] and calculated based on the weight
matrices of the first and second layers (W1 and W2) and
biases (b1 and b2). The output of neural network is (4). More
details about this network is available in [24, Fig. 1].
(4)
Learning exhibits outstanding performance in supervised
learning tasks [27]. Hence, its use is widespread in medical
research and it has become a new breakthrough because of
its remarkable results at discovering learnings [28].
C. Performance Metrics
This sub-section discusses the evaluation criteria that are
used to measure the quality of a classifier. In evaluating a
model’s performance, several factors come into play namely,
accuracy, classification error, precision and recall, ROC, and
others. These metrics are computed by using the four (4)
outcomes mentioned below:
 True Positive (tp) – the number of positive
examples in the test data set that are classified as
positive
 False Positive (fp) – the number of negative
examples in the test data set that are classified as
positive
 True Negative (tn) – the number of negative
examples in the test data set that are classified as
negative
 False Negative(fn) – the number of positive
examples in the test data set that are classified as
negative
There are no predefined metrics in evaluating a
classification problem. Different metrics provide different
insights. The choice of the metrics is relative to the learning
problem. Nonetheless, this experiment used the four (4)
performance metrics as follows:
1. Accuracy and Classification Error
Accuracy is often the most common performance metric
for prediction. Achieving high accuracy in modeling is the
topmost factor and in most cases when accuracy is difficult
to achieve, using approximation techniques will suffice. The
training time of a model increases as the number of
parameters increases. However, having numerous parameters
in a model is also indicative of greater flexibility [29].
The formulas for Accuracy and Classification Error are in
(5) and (6).
(5)
(6)
2. Precision and Recall
Precision is referred to as the Positive Predictive Value
(PPV). The precision for a class is computed as the number
of true positives divided by the sum of the true positives and
false positives. The formulas for Precision and Recall are in
(7) and (8).
(7)
Recall is referred to as the True Positive Rate (TPR) or
Sensitivity or Hit Rate. The recall for a class is computed as
the number of true positives divided by the sum of the true
positives and false negatives.
(8)
3. Receiver Operating Characteristic (ROC)
ROC graphs are first used in medical research and
gradually gaining popularity in machine learning and data
mining research. ROC plots or graphs are utilized in
evaluating the performance and visualizing the binary
classification models [30].
An ROC curve is a graphical plot that demonstrates the
analytical ability of the classifier. The curve is plotted with
True Positive Rate (TPR) on the y-axis and False Positive
Rate (FPR), also known as Fallout (1 – Specificity), on the x-
axis. The ROC curve is the preferred choice to select models
that can give optimal performance because it is less sensitive
to the class sizes.
III. EXPERIMENTAL METHOD
A. Selection of Algorithms
This study used five (5) classification algorithms to
predict a category, namely Decision Tree, Naïve Bayes, k-
nn, Random Tree, and Deep Learning. This list is not
exhaustive of the applicable algorithms that can be used for
this machine learning problem. There are other algorithms
that can be applied for this two-class prediction problem.
Furthermore, these algorithms are chosen using the following
criteria: the ability to handle different column types
(numerical, binary, and categorical), a binary target type, a
small number of columns and rows and an ability to handle classifier operators, namely Decision Tree, Naïve Bayes, k-
missing values. nn, Random Tree and Deep Learning.

B. Software Specification
1. Datasets Used

This experiment used three (3) standard datasets that can

be freely downloaded from UCI Machine Learning
Repository [31]. These are as follows:
 ASD Screening Data for Child Dataset contains
292 examples (151 non-ASD cases, 141 ASD-like
cases)
 ASD Screening Data for Adolescent Dataset
contains 104 examples (41 non-ASD cases, 63
ASD-like cases)
 ASD Screening Data for Adult Dataset contains Fig. 4 Process Model
704 examples (515 non-ASD cases, 189 ASD like
cases)
IV. PERFORMANCE ANALYSIS AND RESULTS
These datasets have been cleaned and prepared for
machine learning. From the three (3) datasets, only 16 out of 1. Accuracy
the 21 attributes are used for this study where
ASD_Likelihood is chosen as the class label. The other Using the three (3) datasets, Deep learning with 96.38%
attributes are irrelevant to the learning problem. The average accuracy outdone the other classifiers as shown in
common description of the datasets is summarized in Table Fig. 5. It is seconded by Naïve Bayes with 90.30% average
1. accuracy, followed by k-nn with 88.89% average accuracy,
Decision Tree with 85.87% average accuracy, and lastly,
Table 1 Problem Description Random Tree with 72.74% average accuracy. The Random
Q1-Q10 Integer Attribute Tree is the least accurate classifier as compared with the
others.
Age Polynomial Attribute
Gender Polynomial Attribute
Ethnicity Polynomial Attribute
Jaundice_History Polynomial Attribute
ASD_History Polynomial Attribute
ASD_Likelihood Binomial Label

2. Software Resources

This experiment used Rapidminer which is a commercial

and open-source software platform for data science. It
performs machine learning, data mining, predictive analytics,
and much more. It has an integrated environment for data
preparation, modeling, evaluation, and deployment.
3. Process Model

The Rapidminer process model, as shown in Fig. 4,

involves the four operators: Read Excel operator that loads
the example set; Set Role operator that sets the label role;
Generate Weight (Stratification) operator that distributes the
weight over the example set, and the Validation operator that
trains, tests, and measures the performance of a learning
operator.
Inside the Validation model are three operators: the
classifier operator, apply model operator that applies a
model, and the performance operator that evaluates the
performance of the classifier. This experiment used different
Fig. 5 Comparison of Accuracy Levels
2. Precision and Recall
The confusion matrix (also known as contingency table)
presents the number of correct and incorrect predictions as
well as the precision and recall values. The matrix, as shown
in Fig. 6, displays a visualization of the classification results
and summarizes the precision and recall percentages using
the three (3) datasets.
 Decision Naïve Random Deep
K-nn
Tree Bayes Tree Learning
Pred. NO
87.13 82.79 97.15 72.86 92.21
Class precision
Pred. YES
85.77 95.72 86.59 40 100
Class precision
OVERALL %
86.45 89.25 91.87 56.43 96.10
AVERAGE

Decision Naïve Rando Deep

K-nn
Tree Bayes m Tree Learning
True NO
85.22 95.51 81.71 83.33 100
Class Recall
True YES
84.21 82.45 92.10 42.10 92.98
Class Recall
OVERALL %
84.71 88.98 86.90 62.72 96.49
AVERAGE

Fig. 6 Confusion Matrix Summary

As shown in the class precision table, the Deep Learning

classifier achieved the best average class precision (96.10)
and class recall (96.49) percentages. It is followed by Naïve
Bayes, k-nn, and Decision Tree classifiers having close
precision and class recall percentages. On the contrary,
Random Tree trailed behind as compared to the other
classifiers having only 56.43% class precision and 62.72%
class recall.
3. ROC

In Rapidminer, the Compare Receiver Operating

Characteristic (ROC) operator is used to produce the ROC
curves of models in the same plotter diagram. The plot of the Dataset ROC Performance
curve indicates the true positive rate against the false positive 1st 2nd 3rd 4th 5th
rate. Likewise, the ROC curve is plotted in an optimistic Child Deep k-nn Naïve Random Decision
manner wherein it identifies examples with high confidence Learning Bayes Tree Tree
Adolescent k-nn Deep Naïve Random Decision
first and thereafter plot the succeeding examples with
Learning Bayes Tree Tree
decreasing confidence. Adult Deep k-nn Naïve Random Decision
As seen in Fig. 7, the ROC curves for the different Learning Bayes Tree Tree
models using the three datasets are graphically plotted for
comparison and ranked accordingly. A curve, which is Fig. 7 ROC curves for the ASD datasets
plotted on the upper leftmost or having coordinates (0, 1) of
the ROC space, signifies the best possible prediction. This V. CONCLUSION AND FUTURE WORK
means that this classifier has achieved 100% true positive
rate and 0% false positive rate, thus, is the considered the This paper has provided a comparative study on the
most accurate classifier. prediction performance of several learning classifiers. It
investigated which model(s) provides high accuracy, good
As summarized in the ROC Performance table, Deep precision and recall percentages, and ROC performance for
Learning and k-nn are the top performing classifiers the given machine learning problem. This study concludes
according to their ROC curves. Conversely, the Naïve Bayes, that Deep Learning classifier achieved the highest
Random Tree, and Decision tree consistently maintain the classification performance in terms of accuracy, precision
3rd, 4th, and 5th place, respectively, in terms of their ROC and recall percentages and ROC, whereas, Random Tree
performance. classifier is the least performing classifier in all metrics.
Good classification performances are also achieved by k-nn,
Naïve Bayes, and Decision Tree classifiers. These results can
be used as references for future work on ASD screening.
This paper hopes to promote subsequent work on
performance evaluation for the given binary classification
problem by exploring other machine learning algorithms
such as Chaid, Decision Stump, Rule Induction, and Random
Forest. Further, the small datasets used in this experiment
may affect the generalization of the models that are built,
thus, it is recommended to provide more training data to
improve the performance of the classifiers.
Lastly, high prediction performance is essential in
medical research especially in ASD screening so that the best
intervention strategies are provided.
ACKNOWLEDGMENT
The author thanks UCI Machine Learning Repository for
allowing the use of the three (3) Autism Screening datasets.
REFERENCES
[1] I. Rapin and R. F. Tuchman, “Autism: Definition, Neurobiology,
Screening, Diagnosis,” Pediatric Clinics of North America, vol. 55,
no. 5. pp. 1129–1146, 2008.
[2] F. Hauck and N. Kliewer, “Machine Learning for Autism
Diagnostics: Applying Support Vector Classification,” Int’l Conf.
Heal. Informatics Med. Syst., pp. 120–123, 2017.
[3] Y. Q. Zhang and J. C. Rajapakse, Machine Learning in
Bioinformatics (Vol.4). John Wiley & Sons, 2009.
[4] A. R. Olivera et al., “Comparison of machine-learning algorithms to
build a predictive model for detecting undiagnosed diabetes - ELSA-
Brasil: accuracy study,” Sao Paulo Med. J., vol. 135, no. 3, pp. 234–
246, 2017.
[5] S. Wang and R. M. Summers, “Machine learning and radiology,”
Medical Image Analysis, vol. 16, no. 5. pp. 933–951, 2012.
[6] X. Chen, M. Wang, and H. Zhang, “The use of classification trees for
bioinformatics,” Wiley Interdiscip. Rev. Data Min. Knowl. Discov.,
vol. 1, no. 1, pp. 55–63, 2011.
[7] D. Bone, M. S. Goodwin, M. P. Black, C. C. Lee, K. Audhkhasi, and
S. Narayanan, “Applying Machine Learning to Facilitate Autism
Diagnostics: Pitfalls and Promises,” J. Autism Dev. Disord., vol. 45,
no. 5, pp. 1121–1136, 2015.
[8] M. Duda, J. A. Kosmicki, and D. P. Wall, “Testing the accuracy of an
observation-based classifier for rapid detection of autism risk,”
Translational psychiatry, vol. 5. p. e556, 2015.
[9] D. P. Wall, J. Kosmicki, T. F. Deluca, E. Harstad, and V. A. Fusaro,
“Use of machine learning to shorten observation-based screening and
diagnosis of autism,” Transl. Psychiatry, vol. 2, 2012.
[10] M. I. Jordan and T. M. Mitchell, “Machine learning: Trends,
perspectives, and prospects,” Science, vol. 349, no. 6245. pp. 255–
260, 2015.
[11] F. F. Thabtah, “Autism Spectrum Disorder Screening: Machine
Learning Adaptation and DSM-5 Fulfillment,” in Proceedings of the
1st International Conference on Medical and Health Informatics
2017, 2017, pp. 1–6.
[12] F. F. Thabtah, “Autism Spectrum Disorder Tests App.” 2017.
[13] Wei-Lun Chao, “Machine Learning Tutorial.” Graduate Institute of
Communication Engineering, National Taiwan University, Taiwan,
2011.
[14] D. Greene, P. Cunningham, and R. Mayer, “Machine Learning
Techniques for Multimedia: Case Studies on Organization and
Retrieval (Cognitive Technologies),” in Machine Learning
Techniques for Multimedia, 2008, pp. 51–90.
View publication stats
[15] E. Learned-Miller, “Introduction to Supervised Learning.”
Department of Computer Science, University of Massachussetts,
Amherst, 2014.
[16] T. Pang-Ning, M. Steinbach, and V. Kumar, Introduction to data
mining. 2006.
[17] S. Agarwal, “Introduction. Binary Classification and Bayes Error.”
pp. 3–4, 2013.
[18] A. T. Azar and S. M. El-Metwally, “Decision tree classifiers for
automated medical diagnosis,” Neural Comput. Appl., vol. 23, no. 7–
8, pp. 2387–2403, 2013.
[19] H. Zhang, “The Optimality of Naive Bayes,” Proc. Seventeenth Int.
Florida Artif. Intell. Res. Soc. Conf. FLAIRS 2004, vol. 1, no. 2, pp.
1–6, 2004.
[20] S. Gadat, T. Klein, and C. Marteau, “Classification in general finite
dimensional spaces with the k-nearest neighbor rule,” Ann. Stat., vol.
44, no. 3, pp. 982–1009, 2016.
[21] “Modern Machine Learning Algorithms: Strengths and Weaknesses,”
EliteDataScience, 2017. [Online]. Available:
https://elitedatascience.com/machine-learning-algorithms.
[22] P. Rai, “Supervised Learning: k-Nearest Neighbors and Decision
Trees.” School of Computing, The University of Utah, Utah, 2011.
[23] L. Kozma, “k Nearest Neighbors algorithm (kNN).” Helsinki
University of Technology, Helsinki, 2008.
[24] Rapidminer, “Random Tree,” Rapidminer Documentation, 2018.
[Online]. Available:
https://docs.rapidminer.com/latest/studio/operators/modeling/predicti
ve/trees/random_tree.html. [Accessed: 13-Jan-2018].
[25] Y. A. LeCun, Y. Bengio, and G. E. Hinton, “Deep learning,” Nature,
vol. 521, no. 7553, pp. 436–444, 2015.
[26] Y. Lv, J. Liu, and T. Yang, “Comparative studies of model
performance based on different data sampling methods,” in 2013 25th
Chinese Control and Decision Conference (CCDC), 2013, pp. 2731–
2735.
[27] J. Wu, Yinan Yu, Chang Huang, and Kai Yu, “Deep multiple instance
learning for image classification and auto-annotation,” in 2015 IEEE
Conference on Computer Vision and Pattern Recognition (CVPR),
2015, pp. 3460–3469.
[28] Y. Bengio, “Deep learning of representations: Looking forward,” in
Lecture Notes in Computer Science (including subseries Lecture
Notes in Artificial Intelligence and Lecture Notes in Bioinformatics),
2013, vol. 7978 LNAI, pp. 1–37.
[29] G. Ericson and W. A. Rohm, “How to choose algorithms for
Microsoft Azure Machine Learning,” Microsoft Azure, 2017.
[Online]. Available: https://docs.microsoft.com/en-us/azure/machine-
learning/studio/algorithm-choice. [Accessed: 12-Jan-2017].
[30] T. Fawcett, “An introduction to ROC analysis,” Pattern Recognit.
Lett., vol. 27, no. 8, pp. 861–874, 2006.
[31] M. Lichman, “UCI Machine Learning Repository.” University of
California, School of Information and Computer Science, Irvine, CA,
2013.