Proceedings of the Electrical and Electronic Postgraduate Colloquium EEPC2009

Motivation for Developing a High Combinatorial Interaction Strength Testing Suit via
the Grid Computing Approach
Bestoun S. Ahmed1 and Kamal Z. Zamli 2
1
Software Engineering Group, School of Electrical and Electronic Engineering
Universiti Sains Malaysia, Engineering Campus
14300 Nibong Tebal, Seberang Perai Selatan, Pulau Pinang, Malaysia
E-mail: besoon82@gmail.com
2
School of Electrical and Electronic Engineering
Universiti Sains Malaysia, Engineering Campus
14300 Nibong Tebal, Seberang Perai Selatan, Pulau Pinang, Malaysia
E-mail: eekamal@eng.usm.my

Abstract parameters and configuration may give rise to the

An essential purpose of software testing is detecting the
defects for a given set of inputs and estimating the
effectiveness of a program operation. This allows the
software program to be viewed as the functionality of an
input to an output to ensure that the program realizes its
function. However, considering all input possibilities is
impossible for a complex configuration and large input
parameter software due to the different factors including
time, cost, and resource constraints. Thus, the need for a
more intelligent sampling mechanism is realized. Recently,
the widespread use of combinatorial interaction testing has
been shown to provide a dramatic solution to such a problem.
Although there are combinatorial test techniques that can
solve the problem of input defects, the increasing complexity
of configurable software systems creates the need for more
complex interaction strengths that current solutions do not
have. In this paper, we present our approach to this problem
and the possibility of using Grid computing to take a better
development direction.
Keywords:
Testing, Combinatorial interaction testing, Grid computing
Introduction
combinatorial explosion problem.
The combinatorial explosion problem appears in a highly
configurable system, such as those used in industrial
applications. The testing of such a system needs a large
number of test cases due to the different input parameters the
system may accept. These parameters and their interactions
with each other must be considered to ensure the accurate
detection of different software bugs. However, exhaustive
testing is often impossible due to time and resource
limitations. As a result, there still exists a need for a more
intelligent mechanism that ensures that interaction coverage
is approached systematically with a minimum number of test
cases. A combinatorial strategy is a method or mechanism
for selecting test cases using a combination of test input
parameters [1].
In this paper, we address the different problems in
combinatorial interaction testing that motivate us to go
further and investigate more. To this purpose, we first
present different combinatorial strategies by providing the
definition and the salient features of each strategy, including
their advantages and disadvantage. Our focus will center on
the more recent strategies. In doing so, the motivation points
will be made clearer. From this, we will build our approach
to the problem. Finally, we will give a preliminary analysis
of the problem and declare our intended approach.

As an essential part of software quality assurance, software

testing must span the entire development life cycle. To
Motivation
address quality standards, two different aspects of the
Literature reveals more than 19 different combination
problem must be considered: (1) revealing the faults and (2)
strategies described in more than 40 papers. One of the early
asserting the reliability of the product [1]. To this end, for
techniques for solving the combinatorial explosion problems
given any software, test cases must be generated to
is random testing [2]. Such testing method, however, has
undertake the test process and address the faults that may
both advantages and disadvantages. In particular, it has been
occur. Nowadays, our dependencies on software are
found that different values may produce the same behaviors
straightforward and increasing, as many kinds of software
in random testing. In addition, the probability that one would
have become a part of our daily lives. We use software in
choose a test value that would cause buggy behavior is very
life-saving or safety applications, for example. As a result,
low. Thus, the randomness of choosing the input parameters
rigorous software testing is immensely important. For this
is considered as a non-deterministic strategy [3]. To address
reason, many combinations of input parameters and system
this problem, a systematic method for choosing the right
configurations need to be tested to ensure that a program
inputs becomes an essential demand. This method is called
functions properly. However, high numbers of input
the deterministic strategy.
 Proceedings of the Electrical and Electronic Postgraduate Colloquium EEPC2009

In contrast to random strategies, systematic or deterministic and strategies in detail.

strategies produce the same results from a specific input
parameter domain. Currently, there are different
combinatorial strategies under this category including Table 1. Summary of combination strategies
Orthogonal Array (OA) [4], Covering Array (CA) [5], Each Combination Strategy
Choice (EC), Base Choice (BC) [6], All Combinations (AC),

(4,5,6) - Way
2-Way
3-Way

(7,8) - Way

Higher (t)
Anti Random (AR), Partly Pair Wise (PPW), and IPO,
among others. The following diagram shows the detailed T-Way strategy
categorizations for most of the available testing strategies.

AETG N N N
Non-deterministic mAETG N N N
mTCG
GA N N N
ACA N N N
OA
CA
TConfig
AllPairs
TCG N N N N
Deterministic IPO
IPOG
Jenny
TVG
CTS

Figure 1. A detailed categorization of the available testing

strategies
In the deterministic direction, a combinatorial approach is
used to ensure that the parameters interact in such a way that
the minimum number of test cases is produced. This type of
testing combines parameter values with T-way input
parameters that are covered at least once in a generated test
case.
One form of T-Way testing is pairwise or 2-way testing. It is
used to ensure the pairwise interaction between two
parameters. There are several tools used for pairwise testing
such as AETG [7], TConfig [8], [9], IPO [10], TCG, and
Jenny, among others. However, different studies show that
some faults may also be caused by interactions involving
more than two parameters [11]. Thus, it is necessary to
enable a higher strength of coverage (or t value). IPOG [5]
and its improvements, IPOG-D [12] and IPO-s [13], are
perhaps three of the most recent strategies for T-Way
interaction testing implemented in a tool called FireEye [14].
A comparison of combination strategies has been made with
respect to coverage criteria, size of generated test suits, time
of execution, and generation. Based on these research
studies, it appears that the problem of generating and
constructing a minimum test set from the exhaustive test
space is an NP-complete problem [10]. To summarize these
features, the following table compares most of these tools
Referring to Table 1, the existing T-way strategies are scarce
in higher (t) interaction strengths. We must note that, the
above table is with a certain parameter number not with a
flexible situation of parameter. As a result, there is a scarce
also in the parameter number also. It is important to mention
that (N) in the table refers to the fact that not enough
information exists because of time constraints. Most
strategies aim to support t=6 at most. As mentioned before,
due to the increasing complexity of configurable software
systems, parameter interaction tends to increase
dramatically.
Approach
From the above discussion, clearly there is a need for a new
approach for developing new strategies to support high
strength combinations of interacting variables. Higher
interacting degrees of coverage can be remarkably more
effective in revealing faults.
To address this problem, our research group has made two
efforts to increase the strength of parameter interaction. The
first effort is called the G2Way strategy. This strategy is used
for conducting T-Way test generation using the backtracking
approach [15]. The second effort is a modification of the
IPOG strategy called MIPOG for conducting T-Way testing
[16]. Empirical results demonstrate that both strategies, in
most cases, outperform other strategies as far as the number
 Proceedings of the Electrical and Electronic Postgraduate Colloquium EEPC2009

of T-way test sets is concerned. From the results of both For highly configurable systems, the likelihood that one will
strategies, it is clear that there are problems that exist for obtain an out-of-memory error increases when performing
higher T-ways and time. high interaction test generation. Even when the interaction is
within an appropriate range, the time necessarily spent on
There are two efforts in performing improvements through
test case generation will also increase. To prevent such
the modification of the MIPOG strategy for parallel T-way
situations, the strategy called Grid computing appears to be a
test generation.
favorable solution.
The first effort uses multi-core systems, and it is given the
Grid computing provides two main directions. First, it
strategy name MC_MIPOG [17]. The second effort uses
develops a new strategy for T-way test case generation
Grid computing, and it is given the strategy name G_MIPOG
because it is able to deal with a large number of parameters
[16]. Although both cases show an improvement in terms of
and values. Second, the Grid system provides a method for
T-interaction (around t=11), problems still arise due to time
the better distribution of computational processes. This can
and higher T-way interactions. Table 2 summarizes these
lead to higher T-way interactions that utilize less time for
results.
processing.

Preliminary Analysis Conclusion

From the discussion above, it is evident that some problems
This paper aims to address the motivations for developing
still exist with regard to the methods of input selection for
and building a test suit for a system with a high T size. Such
software testing and that these problems require further
a test suit must include the minimum possible number of test
study. These problems include the interaction factors for
cases. When this number is obtained, we can take advantage
generating test cases, the size of these generated cases, and
of real parallelism by distributing them over a set of grid
the time it takes to generate these cases. Each problem gives
resources.
rise to other factors that must also be considered.
Currently, we are developing a test generation strategy that
When the number of parameter coverage increases, the
can derive a minimal test size within an acceptable time limit.
number of T-way test sets also increases exponentially. As
Such a strategy must be able to work with high parameters
such, large and complex systems require higher numbers of
and values. When such a strategy becomes available, we can
parameters and higher orders of T-way tests. Thus, the size
use a real parallel technique to distribute the task around a
of generated test cases will increase dramatically. This large
Grid system that gains time and interaction strength.
number of test cases may further lead to a combinatorial
explosion problem. Testing may become tedious and
consume more of the main memory of the system.

Table 2. Comparative results using the TCAS module for T=2 to 11(source [17])
 Proceedings of the Electrical and Electronic Postgraduate Colloquium EEPC2009
References
[1] Baresi, L., 2006,"An introduction to software testing",
Electronic Notes in Theoretical Computer Science,
148: 89-111.
[2] Mandl.,R., 1985, "Orthogonal Latin Squares: An
application of experiment design to compiler testing",
Communications of the ACM, 28(10):1054-1058.
[3] Grindal, M., Offutt, J., Andler, F., 2005, "Combination
testing strategies: a survey", Journal of Software
Testing, Verification and Reliability, 15(3): 167-199.
[4] Hedayat, A.S., Sloane, N.J.A., and Stufken., J.,
"Orthogonal Arrays: Theory and Applications", New
York: Springer, 1999.
[5] Hartman, A., and Raskin, L., "Problems and
Algorithms for Covering Arrays", Discrete Math., July
2004, pp. 149-156.
[6] Grindal, M., Lindstrom, Offutt, J, and Andler, S., "An
Evaluation of Combination Strategies for Test Case
Selection, Technical Report", Technical
ReportHS-IDA-TR-03-001, Department of Computer
Science, University of Skovde, 2003.
[7] Cohen, D.M., Dalal, S.R., Fredman, M.L., and Patton,
G.C., "The AETG System: An Approach to Testing
Based on Combinatorial Design", IEEE Trans. on
Software Engineering, 23(7), July 1997, pp. 437-444.
[8] TConfig tool. Available:
http://www.site.uottawa.ca/~awilliam.
[9] Williams, A. W., and Probert, R. L., "A practical
strategy for testing pair-wise coverage of network
interfaces", In Proceedings of the 7th International
Symposium on Software Reliability Engineering
(ISSRE), White Plains, New York, 1996.
[10] Lei, Y., and Tai, K. C., “In-parameter-order: a test
generation strategy for pairwise testing,” Proceedings
of 3rd IEEE Intl. Conf. on High-Assurance Systems
Engineering Symposium, 1998, pp. 254-261.
[11] Fouche, S., Cohen, M., and Porter, A., "Towards
incremental adaptive covering arrays", Proceedings of
the 6th joint meeting of the European software
engineering conference and the ACM SIGSOFT
symposium on The foundations of software engineering,
Dubrovnik, Croatia, 2007.
[12] Lei, Y., Kacher , R., and Richard, D., 2008, "
IPOG/IPOG-D: efficient test generation for multi-way
combinatorial testing", Journal of software testing ,
verification and reliability, 18: 125-148.
[13] Calvaga, A., and Gargantini, A., "IPO-s: incremental
generation of combinatorial interaction test data based
on symmetries of covering arrays", Proceedings of
IEEE Int. Conf. on software testing verification and
validation workshops, Denver, Colorado – USA, 2009.
[14] Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., and
Lawrence, J., “IPOG: A General Strategy for T-way
software testing,” Proceedings of 14th IEEE Intl. Conf.
and Workshops on the Engineering of Computer Based
Systems, August 2007.
[15] Klaib, M., Zamli, K., Mat .Isa, Younis, M., and
Abdullah, R., “G2Way – A Backtracking Strategy for
Pairwise Test Data Generation”, in the 15th IEEE
Asia-Pacific Software Engineering Conference,
Beijing, China, 2008, pp. 463-470.
[16] Younis, M.I. Zamli, K.Z. Isa, N., "
A strategy for Grid based t-way test data generation",
In proc. First International Conference on Distributed
Framework and Applications, DFmA, 2008.
[17] Younis, I., and Zamli, Z., 2008, "MC_MIPOG: Parallel
T-Way Test Generation Strategy for Multi-core
System", ETRI Journal.