Beruflich Dokumente
Kultur Dokumente
Abstract— Automation of question generation from a text this question generator program can help learners to assess
has been one of the focus of research in recent years. In the what has been studied independently with answer questions
education field, question generation can be used to assist in the that arise from the material that has been studied [5]. There
generation of questions to be used as evaluations of learning have been several previous studies that have examined the
outcomes. The process of generating questions with different question generation of using the Indonesian and English. The
difficulty levels manually is not easy. Firstly, someone must study carried out among others is the generation of questions
understand the whole matter and then she or he is able to categorizing definitions, methods and reasons using template
make questions according to the material. Generation of base [6], then there is study on generating questions on the
questions in large quantities and various learning materials
medical domain using predicate argument (PA) and the
will certainly require lot of effort and time. Therefore, it is
necessary to automate the process of generating the question.
introduction of Named Entity relation (NER) [7]. Study in
This research introduces question generation automation the medical field is then developed again using Semantic
methods based on Bloom's Taxonomy using text analysis. The Role Labelling (SRL) and (PA) [8]. Another method used in
method proposed in this study yielded an accuracy of 81.35%. study related to question generation in Indonesian is the use
The accuracy proves that the proposed method can be used to of syntax analysis that consists of phrase identification
generate questions automatically process using pattern matching method, gramatical function
determination using rule-based method for creating question
Keywords— Question Generation, Text Analysis, Taxonomy from compound sentence [9]. The recent study on question
Bloom generation in Indonesian is done using syntactical approach
to generate Open Domain Indonesian Question [10]. In
I. INTRODUCTION addition to Indonesian, question generation is also developed
in foreign languages, one of which is English [11][12]. They
Question generation has been the focus of research for generate a question in english from a paragraph with selected
this several years. Question generation is a technique for features like first sentence, last sentence, words that often
generating questions that come from a sentence or text in the appear, sentence length, number of nouns, adverbs, and
form of natural language [1]. The use of question generators hyphens from paragraph[11], and another research used
will reduce the dependence of an expert in question combination of POS Tags, Named Entity Relation (NER),
generation [2]. Question generation is usually used in 2 and Super Sense Tags (SST) for generate question [12].
ways, to help create Question Answering System (QAS) and From the description, it is known that there has been no
to assist in making questions in the academic field [3]. In the research on education domains that do the generation of
QAS field, question generation can be used to enrich the questions as well as classify it according to certain difficulty
rules while in the academic field the questions can be used level. Therefore, the contribution of this research is to create
for evaluation of learning outcomes. The process of rules that can generate question generation automation based
evaluating learning outcomes can be done through on Bloom's Taxonomy using text analysis methods to
questioning with varying difficulty levels [4]. In 1950, facilitate the process of generating questions that will be
Benjamin Bloom has introduced a concept known as the used as material for evaluation of learning outcomes. This
bloom’s taxonomy. The concept introduces a level of paper is organized as follows. In section 2 we will describe
assessment of the capabilities possessed by learners. On the the literature study. In section 3 we will introduce the
other hand, making questions with different difficulty levels implementation of our approach. In section 4 we will
according to the bloom’s taxonomy is not easy. Everyone evaluate the result. In section 5 we will describe the
has a different perspective regarding the classification of conclusions of this method and future work.
each type of question. It is necessary to have a program
which is capable to generate questions as well as classifying
questions based on different difficulty levels in accordance
with the concept of a bloom’s taxonomy. The results of this
question generation will not be only beneficial to the teacher
but will be also beneficial to the learner. The existence of
270
C. Bloom’s Taxonomy Start
271
Start
Sentence
Pattern
from Paragraph Identification of
1 2 Keyword tion Target learning extraction sentence pattern
1 2 Adalah Jelaskan apa D Arg1 materials
(is,am, yang
are) dimaksud
(Explain T
what is Identified?
meant)
2 1 Adalah Disebut B Arg2
Bloom’s Taxonomy
(is,am, (called)
are) Generating
3 2 Yaitu Yaitu B Arg1 End questions based
on difficulty level
(that is) (that is)
4 1 Yaitu Disebut B Arg2
(that is) (called)
5 4 Berdasar Berdasarkan B Arg1 Question
kan (based on) based on
difficulty level
(based
on) Fig. 3. Scheme of Generation and Question Classification
6 1 Yakni Yakni B Arg1
(that is) (that is) Paragraph:
7 1 Bertuju- untuk Tujan dari B Arg2 Serif adalah huruf yang memiliki tanda dekoratif pada ujung dari
an (for) (the purposes setiap huruf. Jenis huruf ini umum digunakan pada bagian heading
(aim) of) dari dokumen. Sans-serif adalah huruf yang tidak memiliki serif.
8 4 Digunak untuk Kegunaan B Arg2 (“Serif is the letter that has decorative sign in the end of letter. These
an (for) dari type of letters are generally used for heading part of the documents.
(used) (usability of) San-serif is the letter that doesn’t have serif ”)
9 3 Terdiri dari Terdiri dari B Arg1
(consist) (from) (consist of) The process of extracting paragraphs into every sentence:
1. Serif adalah huruf yang memiliki tanda dekoratif pada ujung
dari setiap huruf. (“Serif is the letter that has decorative sign in
C. Generation Process and Question Classification the end of the letter”)
2. Jenis huruf Serif umum digunakan pada bagian heading dari
Question generation process is done based on pattern dokumen. (“These letters are generally used for heading part of
identification on every sentence to be questioned. The use of the document”)
pattern matching is chosen because it is considered easy to 3. Sans-serif adalah huruf yang tidak memiliki serif. (“San-serif is
implement, has high accuracy and does not require any the letter that doesn’t have serif”)
additional resources or tools. The question generation
scheme is shown in Fig. 3. The paragraphs entered by the Fig. 4. Paragraph Extraction Process
user will be broken down into per sentence. Each of these
TABLE II. RESULT OF PATTERN IDENTIFICATION
sentences will be identified. The process of checking the
question generation pattern is done through checking the Pattern Sentences Description
existing keywords in the sentence. Keywords found are then id
1 Serif Arg1
matched against a list of existing patterns. The process of
adalah Keyword
classifying questions is done through keyword identification huruf yang memiliki tanda dekoratif pada Arg2
and pattern identification of each question. Each question ujung dari setiap huruf.
that has been raised then identified the level of difficulty 2 Serif Arg1
based on the keywords and patterns that the question has. adalah Keyword
After the identification process is completed, then formed huruf yang memiliki tanda dekoratif pada Arg2
ujung dari setiap huruf.
questions that have been classified based on the level of
56 Serif adalah huruf yang Arg1
difficulty. Examples of the results of the implementation memiliki Keyword
process are shown in Fig. 4, Table 2 and Table 3. Because tanda dekoratif pada ujung dari setiap huruf. Arg2
the second sentence uses pronouns that refer to the previous 40 Jenis huruf Serif umum Arg1
subject then the sentence will be adjusted first so that next digunakan pada Keyword
questions are not ambiguous. Adjustment is to replace the bagian heading dari dokumen. Arg2
pronoun to be the subject referred. The second sentence From table 3, we know that not all the generated questions
changes to “Jenis huruf Serif umum digunakan pada bagian result a standard question. It is because the question is
heading dari dokumen”. generated according to the given sentence that used for
question generation . It means if the sentence from text is
not standard sentence, than neither is the generated question.
As long as the generated question is understandable it is not
a problem if the generated question is not standard of
writing [10].
272
TABLE III. RESULT OF QUESTION GENERATION TABLE IV. COMPARISON OF THE RESULT
Pattern Level Question Sentence:
Id Berikut ini contoh algoritma menghitung luas lingkaran.
1 2 Jelaskan apa yang dimaksud Serif! “The following is an example of the algorithm calculates the area of
(“Explain what is Serif?”) the circle”
2 1 Huruf yang memiliki tanda dekoratif pada ujung Generation of questions based on the proposed Kusuma &
dari setiap huruf disebut? method: Alhamri
(“the letter that has decorative sign in the end of Buatlah algoritma untuk menghitung luas lingkaran!
letter is called?”) “Make the algorithm to calculate the area of the
56 2 Serif adalah huruf yang memiliki…. circle!”
(“Serif is the letter that has?”) Bagaimana algoritma menghitung luas lingkaran?
40 3 Jenis huruf Serif umum digunakan pada…. “How the algorithm to calculate the area of the
(“These type of letters are generally used for?”) circle?”
Generation of questions based on Basuki & Kusuma Basuki &
method [10]. Kusuma
Bagaimana algoritma menghitung luas lingkaran?
D. Evaluation “How the algorithm to calculate the area of the
circle”
Evaluation process is needed to see the success rate of a
developed method. The success rate of the developed method
can be seen from the achievement of the accuracy value Therefore, the identification of the original subject is done
owned, the greater value of the accuracy the better method through the identification of the word before the keyword.
developed. The process of calculating accuracy is shown in When the process of checking the pronoun or pattern has
equation 1. been completed it will be done question generation
according to the pattern found in the sentence. After
Accuracy = Amount of Correct Data (1) generating questions, then these questions are classified
Amount of All Data according to the level of difficulty in the bloom’s taxonomy.
From 60 paragraphs in this study, there are 214
The development of this method involves 3 experts.
sentences. Then from 214 sentences can generate 563
Therefore, it is necessary to calculate Kappa to ensure that
questions with different difficulty levels based on Bloom’s
there is no significant difference in judgment between the
experts in conducting the assessment [17]. Taxonomy difficulty levels. From 563 questions, 458
generated questions are calassified valid and the rest are
Kappa calculation formula is shown in equation (2). classified invalid. The generated question is valid if it is
K= Pr (a) – Pr (e) (2) understandable and it can be answerd. The accuracy of the
1 – Pr (e) question generation reaches 81.35%. If compared with the
Which latest research about question generation that have been
Pr (a) = the percentage of consistent measurements among done by Basuki & Kusuma [10], both of these studies are
experts equally used a keyword to generate a question. The
Pr (e) = percentage of the number of changes measurement difference is the research that have been done by Basuki &
between experts Kusuma [10] just generates question according to 5W + 1H,
while this study generates question not only common
Kappa calculation results can be interpreted into 5 question but also an imparative sentence. It is important in
categories. If the calculation value Kappa <0.2 then this study because to make an academic question that suits
considered the result of the agreement among researchers is with Bloom’s Taxonomy sometimes need imperative
poor, if the value between 0.2 - 0.4 means considered as sentence, not only a question. For example “Jelaskan
fair, if the value of 0.41 to 0.60 then considered as pengertian dari … !” (“Explain the meaning of … !”). Table
moderate, if 0.61-0.80 is considered good and if 0.81-1.00 is 4 is the example of questions that can be generated by both
considered very good. methods.
V. CONCLUSION
IV. RESULT AND DISCUSSION
This study yielded an accuracy of 81.35%. It proves that
The experiments conducted in this study were using 60 the proposed method can be used to generate questions with
paragraphs that came from 9 subjects of study program different difficulty levels according to the levels in the
Informatics Engineering. The entire paragraph is then used bloom’s taxonomy. This method can recognize the use of the
as input in this study. The developed method will be pronoun "ini" or "itu" so as to reduce the ambiguity when
implemented into a web. This web creation process is done generating questions. For future works, the researchers will
to simplify the process of testing methods. The method try to improve the effectiveness of existing templates so as to
workflow is shown in Fig. 3. The method of testing begins cover sentences that are not structured in order to increase
by entering an input paragraph then the program will the value of accuracy.
execute paragraphs into sentence units. The sentences are
then identified to know the existence of the pronoun and the ACKNOWLEDGMENT
pattern of the sentence. If a pronoun is found in the sentence
then it is necessary to change the pronoun according to the The authors say thank you to Kemenristek Dikti for
original subject. The subject usually resides before the supporting this research and all of participants who have
explanatory keyword like "adalah" or "merupakan". helped in the process of collecting datasets, classification and
validation in this research.
273
REFERENCES Majemuk,” elib-Unikom, pp. 1-7, 2016.
[10] S.Basuki and S.F.Kusuma, “Automatic Question Generation For 5W-
1H Open Domain Of Indonesian Question By Using Syntactical
[1] W. Suwarningsih, I. Supriana and A. Purwarianti, “Tantangan dan Template-Based Features From Academic Textbooks,”JATIT (Journal
Peluang pada Question Generation,” Jurnal Sistem Informasi (JSI), of Theoretical and Applied Information Technology., vol. 96, no. 12,
pp. 719-730, 2014. pp. 3908-3923, 2018.
[2] H. Ali, Y. Chali and S. A. Hasan, “Automation of Question [11] D.Swali, J.Palan and I.Shah, “Automatic Question Generation from
Generation From Sentences,” in Proceedings of the Third Workshop Paragraph,” International Journal of Advance Engineering and
on Question Generation, Canada, 2009. Research Development, vol. 3, no. 12, Pp. 73-78, Desember 2016.
[12] M.Blstak and R.Viera, “Automatic Question Generation Based on
[3] H. Hussein, E. Mohammed and S. Guirguis, “Automatic English Analysis of Sentence Structure,” Springer International Publishing
Question Generation System Based on Template Driven Schame,” Switzerland, Pp. 223-230, 2016.
IJCSSI, pp. 45-53, 2014.
[13] N. Yusliani and A. Purwarianti, “Sistem Question Answering Bahasa
[4] S. F. Kusuma, D. Siahaan and U. L. Yuhana, “Automatic Indonesia’s Indonesia Untuk Pertanyaan Non Factoid.,” Jurnal Ilmu Komputer
Questions Classification Based On Bloom’s Taxonomy Using Natural dan Informasi, pp. 10-14, 2011.
Language Processing,” in International Conference on Information
Technology Systems and Innovation (ICITSI), Bandung, 2015. [14] N. Sukajaya, M. H. Purnomo and I. K. E. Purnama, “Intelligent
Classification of Learner's Cognitive Domain using Bayes Net, Naive
[5] D. Lindberg, Automatic Question Generation From Text For Self- Bayes, and J48 Utilizing Bloom's Taxonomy-based Serious Game,”
Directed Learning, Canada: Simon Fraser University, 2013. International Journal Of Emerging Technologies in Learning (IJET),
[6] M. Fachrurrozi and N. Yusliani , “Sistem Pembangkit Pertanyaan pp. 46-52, 2015.
Otomatis Dengan Metode Template-Based,” Journal of Research in [15] A. A. Yahya and A. Osman, “Automatic Classification Of Questions
Computer Science and Applications, Vols. Vol. 2, No. 1, pp. 24-29, Into Bloom's Cognitive Levels Using Support Vector Machine,”
2013. IEEE, 2011.
[7] W. Suwarningsih, I. Supriana and A. Purwariyanti, “Discovery [16] M. Taher, “Urgensi Taksonomi Bloom Domain Kognitif Versi Baru
Indonesian Medical Question-Answering Pairs Pattern With Question Dalam Kurikulum 2013,” Balai Diklat Keagamaan Medan, Medan,
Generation,” in International Journal of Applied Engineering 2013.
Research, 2015.
[17] A. Tenriawaru, A. Djunaidy and D. O. Siahaan, Pengembangan model
[8] W. Suwarningsih, I. Supriana and A. Purwariyanti, “Indonesian pengukuran meaningful learning berdasarkan semantik aktivitas
Medical Sentence Transformation for Question Generation,” in Asia pelajar dalam lingkngn E-Learning, Surabaya: Teknik Informatika
Pacific Conference on Wireless and Mobile, Bandung, 2016. Institut Teknologi Sepuluh November, 2018.
[9] D. D.Ginanjar and K.K.Purnamasari, “Pembangkitan Pertanyaan
Otomatis untuk Teks Berbahasa Indonesia yang Mengandung Kalimat
274