EMPIRICAL STUDY UPON SOFTWARE

TESTING LEARNING WITH SUPPORT FROM

EDUCATIONAL GAME
Marcello Thiry, Alessandra Zoucas and Antnio C. da Silva
Master`s degree course in Applied Computing
Universidade do Vale do Itajai - UNIVALI
Florianopolis, Brazil
thiry, azoucas, antonio.carlos}univali.br


Abstract Although software testing is seen as one of the
main quality measurements of software, test practices and
techniques are still seldom applied by software development
companies. One possible reason to this scenario is the lack of
skilled and available professionals to implements such
techniques and practices. The teaching of software testing is
typically approached as a topic of Software Engineering
lectures. In this context, one of the main challenges is to allow
the student to apply the concepts seen in classroom. The
developed educational game aims to allow the students to
practice through experimenting concepts, techniques and
practices of software testing in a simulated environment. To
evaluate the game contribution, we present the results of two
experiments performed with undergraduate students. In the
first experiment the learning effectiveness was compared to
students who did not play the game. In the second experiment,
the learning effectiveness was compared to a traditional pencil-
and-paper exercise.

Keywords: Software testing; educational game; learning;
Experimental Software Engineering.
I. INTRODUCTION
The quality control and assurance in soItware projects
consist in activities and techniques that, when systematically
applied, permit to assess the chances oI success oI a soItware
project concerning the IulIillment oI the expectations oI
internal and external clients |1|. Models as CMMI |2|
include concepts and practices to increase capacity and
maturity oI an enterprise`s processes |3|. Lined up with the
models and standards oI process, SoItware Engineering (SE)
gathers activities and techniques that excel by anticipating, in
a systematic, organized and controlled way, the identiIication
oI nonconIormities in soItware. These activities and
techniques are called VeriIication and Validation (V&V).
Although soItware testing is present in the developing
process, researches show that its techniques and practices are
not totally employed by the soItware development
organizations |4| |5|. In the research perIormed in |4| it was
evidenced that, even though the organizations admit they
apply soItware tests, almost halI (48,5) oI soItware testing
practices are regarded as not applied and not important.
Among the practices regarded as not applied and not
important are all those related to measurement and analysis,
including test coverage |6|. Considering these results, we can
conclude there are a great amount oI practices regarded as
not important and not applied, regardless how big the
companies are |4|. The reasons Ior that include the lack oI
knowledge about the practices, lack oI skilled and available
human resources to its implement, lack oI support oI high
level managers, lack oI approaches, among others.
The results obtained in |4| conIirm the results oI another
study perIormed in 2004 that assessed soItware testing in
soItware development enterprises |6|. It was observed a great
distance between what is produced in the academic
environment and what is put into practice by soItware
enterprises. The results allow veriIying how immature the
assessed enterprises are, concerning soItware testing.
One oI the challenges oI teaching SE is the necessity to
assure that the student acquires enough experience in
applying the concepts through laboratory practices |7|. As an
alternative to the practical approach in SoItware Engineering,
an experiment perIormed with Computer Science
undergraduate students explored the possibility oI having the
students working on a real soItware project |8|. BeIore the
experiment application, it was veriIied that the SE students
Irom the third semester had little capacity to remember
concepts already approached and the possible cause was the
little practice the students had. The experiment main goal
was the application oI concepts and techniques oI SoItware
Engineering |8|. The result identiIied greater motivation and
interest Irom the participants in the SE subject.
Other approach to the practice in SE teaching reIers to
the application oI educational games |9||10||11||12|.
However, it is possible to veriIy a small number oI
experiments perIormed to veriIy the games learning
eIIectiveness |9||10||11||12|. In this context, this paper
intends to promote learning soItware testing techniques and
practices through an educational.
The next section presents researches related to this work.
Section III presents the developed game 'U-TEST. The
details oI the game learning eIIectiveness assessment are
presented in section IV. Section V presents a discussion on
the results and the conclusions oI this paper.
II. RELATED WORK
The analysis oI similar solutions, besides the deIinition oI
the assessment criteria was based in a search using diIIerent
inIormation sources: Google Scholar, CiteSeer, IEEExplorer
and ACM Digital Library. At Iirst, speciIic games to support
SoItware Engineering teaching were sought, using a protocol
containing terms such as: software testing game learning,
software testing game based learning, software testing
teaching approach, among others. As a result, it was
observed the absence oI games in soItware testing.
ThereIore, games to support SoItware Engineering teaching
in general were chosen. In this new search, seven games
were Iound: (1) SimSE |9|, (2) SERPG |10|, (3) TIM -
The Incredible Machine |13|, (4) Planager |14|, (5)
SESAM |11|, (6) X-MED |12| e (7) SimulES |15|.
The games were analyzed under seven criteria: (1) It has
the deIinition oI educational goals; (2) Game genre: action,
adventure, puzzle, simulation, strategy, etc.; (3) It gives a
Ieedback to the students about their perIormance; (4) The
game is available Ior Iree use; (5) There is game learning
eIIectiveness assessment; (6) Game platIorm: non digital,
web, desktop, etc. ; and (7) Field oI knowledge on SE.
Characterization oI the works assessed uses the Iollowing
description: (T) Totally present; (P) Partially present and (N)
Not present. Table I presents criteria and assessment results.
TABLE I. ASSESSED GAMES RESULT
SimSE
SE
RPG
TIM
Plana
ger
SESA
M
X-
MED
Simul
ES
1 1 1 1
2
Slmula
Llon
Slmul.
8C
Slmula
Llon
Slmula
Llon
Slmul
AdvenL
Slmula
Llon
Slmula
Llon
3 1 1 1 1 1 1
4 1 1 1 1 1 1 1
3 1 1 1 n 1 1
6
desk
Lop
web
desk
Lop
desk
Lop
desk
Lop
desk
Lop
noL
dlglLal
7
ro[ecL
MgmL
ro[ecL
MgmL
ro[ecL
MgmL
ro[ecL
MgmL
ro[ecL
MgmL
SofLw.
measu
rem.
SofLw.
pro
cess

There is a concern in most studies according to the
importance oI experiments to the game learning eIIectiveness
assessment, although they was not applied to all assessed
games. Based on the results, little can be concluded about the
game learning eIIectiveness and it continues to be a topic to
be explored in Iuture studies |10|. It was also observed that
most oI the games Iocus on project management or general
SoItware Engineering. Thus, speciIic games in soItware
testing may increase the didactic content and oIIer a
complement to traditional capacitating.
III. THE EDUCATIONAL GAME 'U-TEST
This section presents the educational game 'U-TEST
developed to assist soItware testing teaching. The game has
an instructional design based on ADDIE model |16| and it
was designed to support SoItware Engineering students. It is
expected the students have basic concepts oI programming,
SoItware Engineering and soItware testing. The educational
goals were based on the Bloom`s revised taxonomy |17| and
comprehend the Remember, Understand and Apply levels.
The game goals are: (1) Recognize and Understand the main
concepts oI soItware testing in a general way and (2)
Understand and apply the techniques oI data entry selection,
equivalence class partitioning and boundary-value analysis.
'U-TEST is a simulation game to support soItware
testing with Iocus on unit tests and black box techniques,
approaching theoretical and practical questions. The game is
based on a case where the player is seen as a candidate to a
position in a soItware company. AIter an interview the player
must solve challenges to prepare unit test cases. The game
presents brieI comments about the company and the project
the player will take part. Next, the player must build the test
cases to the presented Iunctions. As a result oI their
perIormance, the player is inIormed about his position on the
players` ranking. The challenges proposed by the game Iocus
mainly on the data entry selection, using a combination oI
equivalence classes partition and limit value analysis. It
allows the player to identiIy the necessity oI application oI
test techniques and practices. Figure 2 presents the screen oI
the Iirst challenge oI the game.

Figure 2- Screen oI the "U-TEST" game.
During the game, the player will undergo ten stages, and
in six oI them there are the Iollowing challenges: 1)
Presentation oI the artiIact; 2) Setup the equivalence classes;
3) DeIine limit values Ior the identiIied classes; 4) Select the
correspondent value to the identiIied value; 5) Setup a
cause-eIIect graph or decision tree; and 6) Project Iinal
Ieedback. The Ieedback is provided by a graphic indicator
placed on the interIace and at the end oI each challenge. The
player is inIormed about their perIormance and its indicator
is updated. At the end oI the game, the player is inIormed
about their general perIormance and position on the players`
ranking.
IV. LEARNING ASSESSMENT
A. The Empirical Studv
During this study, two research questions were deIined:
1) Is the learning eIIect on the remembering, understanding
and applying level in the group oI students that played the
game higher than in the group that didn`t play?; 2) Is the
educational game considered appropriate in terms oI content
relevancy, correctness, suIIiciency and degree oI diIIiculty,
sequence, teaching method and duration in the context Ior
which it is intended? Is the game considered engaging?
Based on the Iirst research question, it was established
the Iollowing hypotheses:
H
0
: There is no signiIicant diIIerence in relative learning
eIIectiveness between group A (experimental group) and
group B (control group).
H
1
: There is signiIicant diIIerence in relative learning
eIIectiveness between group A and group B.
These hypotheses were assessed Irom a statistical test
whereas the second research question was assessed Irom the
qualitative assessment based on the questionnaire answered
by students.
Two experiments were perIormed, the Iirst one involving
Computer Science undergraduate students and the second
one involving InIormation Technology undergraduate
students. Each experiment was based on the proposal deIined
by Kochanski |18|: 1) the assessment was designed
considering all the contents IulIilled in the game; 2) the
students signed a term showing interest in taking part on the
experiment; 3) the students Iilled a questionnaire about their
experience and previous knowledge to establish each student
and group backgrounds; 4) all students attended theoretical
lectures about soItware testing beIore playing the game; 5)
all students perIormed a pretest measuring their knowledge
aIter the lectures; 6) the students were randomly partitioned
into two groups in a balanced manner, one experimental
group (A) and other control group (B); 7) in the Iirst
experiment, the group A played the game and the group B
played another game with no relation to SoItware Testing
(this game was considered a placebo). In the second
experiment, the group B took part in a traditional pencil-and-
paper exercise with problems to solve on the same content
explored in the game; 8) all students perIormed a posttest
measuring again their knowledge aIter the previous step; 9)
using the data collected Irom pre and posttest, hypothesis
tests were conducted to veriIy any improvement on student`s
perIormance aIter playing the game; 10) all students in the
group A answered questions related to their perception oI the
game (this questionnaire served as a base to a qualitative
assessment oI the game).
The pre and posttest questions (steps 5 and 8) were
developed based on the Bloom`s revised taxonomy |17| and
comprehend the Remember, Understand and Apply levels.
B. Hvpothesis Testing and Data Analvsis
In this series oI experiments, our principal concern Ior
accuracy and to overcome problems with statistical power is
due to the very small sample size. For such small data sets, it
is basically impossible to tell, iI the data come Irom a
variable that is normally distributed |19|, as with small
sample sizes (n 20), tests oI normality may be misleading.
UnIortunately, with small samples, parametric tests lack
statistical power and it may be almost impossible to generate
a p-value oI 0.05, whatever the diIIerences between the
groups oI sample data. But, on the other side, nonparametric
tests are not robust. However, inspecting the data distribution
we could not assume a normal distribution oI the variables.
ThereIore, we used non-parametric test (one-tailed Mann-
Whitney U) as it is considered the most powerIul
nonparametric alternative to the t-test Ior independent
samples. Due to the small samples size (n20), we also did
not use a z-value to approximate the signiIicance level Ior
the test, but compared the minimum U to tabellized U values
|19|.
The Iirst experiment was perIormed with ten Computer
Science undergraduate students. They are independently and
randomly sorted into two groups, the Iirst oI size n
A
5
(experimental group) and the second oI size n
B
5 (control
group). The students oI the group A played the game 'U-
TEST, while those oI the group B played another game with
no relation to SoItware Testing (this game was considered a
placebo). Both groups perIormed a pretest (beIore the
treatment) and a posttest (aIter the treatment). Each test has
20 questions distributed according to the Remember,
Understand and Apply levels (Bloom's revised taxonomy).
The Iirst step was to assembly the measures (each
measure represents the diIIerence between the student pretest
and posttest scores) Irom groups A and B into a single set oI
size N n
A
n
B
10. These measures were then rank-
ordered Irom lowest (#1) to highest (#N). When measure
entries are tied Ior ranks, each measure receives the average
oI those ranks. AIter they have been sorted out, the rankings
are then returned to the group, A or B, which they belong
and substituted Ior the original measures that gave rise to
them.
The next step was to calculate w
A
(the sum oI the n
A

ranks in group A) e w
B
(the sum oI the n
B
ranks in group B).
In this experiment, w
A
40 e w
B
15. Using the Mann-
Whitney U Iormula, we calculated U
A
w
A
n
A
(n
A
1)/2
40 5*(51)/2 25 e U
B
w
B
n
B
(n
B
1)/2 15
5*(51)/2 0. Considering the smaller value oI U, we
obtained U
obt
0. Then, we consulted the critical value U
crit

2 Ior the Mann-Whitney Test (U) considering n
A
5 and n
B

5 with the level oI signiIicance 0.05. According to the Mann-
Whitney Test, once U
obt
U
crit
(0 2) is true, we could
reject H
0
.
The second experiment was perIormed with ten
InIormation Technology undergraduate students. They are
also independently and randomly sorted into two groups, the
Iirst oI size n
A
6 (experimental group) and the second oI
size n
B
7 (control group). The students oI the group A
played the game 'U-TEST, while those oI the group B took
part in a traditional pencil-and-paper exercise with problems
to solve on the same content explored in the game. Both
pretest a posttest were the same oI the Iirst experiment.
In this case, we have N n
A
n
B
13. AIter Iollowing
the same procedure described Ior the Iirst experiment, we
calculate w
A
44 e w
B
47. We also obtained U
A
23 e U
B

19. Considering the same level oI signiIicance 0.05, we
have U
crit
7. Once U
obt
U
crit
(19 7) is Ialse, we could
not reject H
0
.
To assess the second research question, we applied a
perception questionnaire with students oI the experimental
groups. In both experiments, students answered they liked
the game and Ielt motivated, besides they thought the game
contributes to learning. Based on these results, we veriIy that
the game, besides promoting a signiIicant improvement at
the students` scores in the Iirst experiment, was positively
evaluated by them. In the second experiment, although it was
not possible to conIirm higher learning eIIectiveness in
relation to a traditional pencil-and-paper exercise, the game
was also considered more motivating that the exercise. It
may be an indication that, even when educational games are
not superior to the traditional exercises, the students`
motivation may be a decisive Iactor to long term learning. A
question to be assessed in the Iuture is: iI the students who
played the game had a greater capacity to keep the
knowledge than the ones who only did the exercise.
V. CONCLUSION
The experiments were conducted in a systematic and
documented way, establishing a Iormalized assessment that
can be repeated countless times. This repetition allows the
comparison with previous assessments, oIIering better
conditions to analyze historical results and identiIy positive/
negative learning tendencies. Although the number oI
experiments is still reduced, as well as the number oI
participants, the positive results (eIIects with statistical
validity) are a stimulus to the continuation oI this research.
The results also helped to identiIy strong and weak points oI
the game and it will guide its development in the Iuture.
Based on the Ieedback obtained, the game is already been
improved, mainly to increase its diIIiculty levels and
variability oI directions. Moreover, we intend to increase the
number oI soItware testing topics considered by the game.
An initial concern was the comparison oI the game with a
traditional pencil-and-paper exercise, once the game is still
very direct and based on problems solution. However,
according to the Ieedback received Irom the qualitative
questionnaires, we veriIy that the game is a motivational
diIIerential, increasing interest and raising greater curiosity
on knowledge. As a continuation oI this study, researchers
have been developing and experiment educational games in
other areas oI SoItware Engineering. Currently, games on
Project Management, Requirements Engineering and
SoItware Improvement Process are being studied.
The results obtained so Iar, even under the statistic rigor,
cannot be generalized. Among the main threatens to the
assessment, it must be considered that the experiments were
perIormed inside two institutions only, where teachers are
part oI the research group. This way, it is necessary to widen
the set oI experiments and number oI students so that it will
be possible to assess learning tendencies with more accuracy.
Other threaten was the inequality on the students knowledge
and experience in SoItware Testing students. However, it
was treated with a previous evaluation oI the proIile
questionnaire answered by the students beIore the
experiments. In this case, no student who could have been a
threat to the obtained results was Iound.
REFERENCES

|1| J. Tian, "SoItware Quality Engineering: Testing, Quality Assurance,
and QuantiIiable Improvement, Hoboken: John Wiley & Sons. Los
Alamitos: IEEE Computer Society Press, 2005.
|2| SEI SoItware Engineering Institute, 'CMMI Ior Development,
Version 1.3, Pittsburgh: SEI, Carnegie Mellon University, 2010.
|3| B. MutaIelija, and H. Stromberg, 'Process Improvement with CMMI
V1.2 and ISO Standards, CRC Press, 2008.
|4| A. R. C. da Rocha, A. C. Dias-neto, A. C. C. Natali, and G. H.
Travassos, 'Caracterizao do estado da pratica das atividades de
teste em um cenario de desenvolvimento de soItware brasileiro, In:
simposio Brasileiro de Qualidade de SoItware, 5., 2006, Vila Velha.
Vila Velha: SBC, 2006. p. 27-41.
|5| A. Bertolino, 'The (Im)maturity level oI soItware testing, ACM
SIGSOFT SoItware Engineering Notes. New York, v. 29, n. 5, p. 1-4,
set. 2004.
|6| H. Zhu, P. Hall, and J. May, 'SoItware unit test coverage and
adequacy, ACM Computing Surveys. New York, v. 29, n. 4, p. 366-
427, dez. 1997.
|7| CEEInI-MEC, 'Diretrizes curriculares de cursos da area de
computao e inIormatica, accessed at 2008 august:
http://www.mec.gov.br~.
|8| M. Gnatz, L. KoI, F. Prilmeier, and T. SeiIert, 'A Practical Approach
oI Teaching SoItware Engineering, In: ConIerence on SoItware
Engineering Education and Training - CSEE&T, 16., 2003, Madrid.
Proceedings. Madrid: IEEE, 2003. p. 120-128.
|9| E. Navarro, 'SimSE: a SoItware Engineering simulation environment
Ior soItware process education, 321 p. Dissertation (Doctor oI
Philosophy in InIormation Computer Science) University oI
CaliIornia, Irvine, 2006.
|10| F. B. V. Benitti, and J. S. Molleri, 'Utilizao de um RPG no ensino
de gerenciamento e processo de desenvolvimento de soItware, In:
Workshop sobre educao em computao - WEI, 16., 2008, Para
SBC/UFPA, 2008. p. 258-267.
|11| J. Ludewig, 'Models in SoItware Engineering an introduction, In
SoItware and Systems Modeling 5-14. Springer Berlin / Heidelberg.
Volume 2, Number 1 / March, 2003.
|12| C. G. von Wangenheim, M. Thiry, and D. Kochanski, 'Empirical
evaluation oI an educational game on soItware measurement,
Empirical SoItware Engineering, v. 14, n. 4, p. 418-452, ago. 2009.
|13| A. Dantas, M. Barros, and C. Werner, 'Treinamento experimental
com jogos de simulao para gerentes de projeto de soItware, In:
Simposio Brasileiro de Engenharia de SoItware SBES, 18, Brasilia:
SBC/UNB, 2004. p. 23-38.
|14| R. Rosa, and E. Kieling 'Planager - Um Jogo para Apoio ao Ensino
de Gerncia de Projetos de SoItware, TCC Bacharelado em Sistemas
de InIormao PUC RS, 2006.
|15| E. Figueiredo, C. Lobato, K. Dias, J. Leite, and C. Lucena, 'Um jogo
para o ensino de engenharia de soItware centrado na perspectiva de
evoluo, XV Workshop sobre Educao em Computao (WEI),
Rio de janeiro, 2007, pp. 37-46.
|16| M. Molenda, 'The ADDIE model, In Kovalchick, A. Dawson, K.
(Eds.), Educational Technology: An Encyclopedia. 201-215. Santa
Barbara: ABC- CLIO, 2003.
|17| L. W. Anderson, and D. R. Krathwohl, (Eds.). 'A taxonomy Ior
learning, teaching, and assessing: a revision oI bloom's taxonomy oI
educational objectives, New York: Longman, 2001.
|18| D. Kochanski, 'Um Iramework para apoiar a construo de
experimentos na avaliao empirica de jogos educacionais,
Dissertao apresentada a Universidade do Vale do Itajai como
requisito para a obteno do titulo de Mestre em computao. So
Jose, Brasil. 2009.
|19| J. Levin, and J. Fox, 'Elementary Statistics in Social Research,
Allyn & Bacon, 2006.