Sie sind auf Seite 1von 10

(IJACSA) International Journal of Advanced Computer Science and Applications,

Vol. 1, No. 4, October 2010

Improving the Technical Aspects


of Software Testing in Enterprises

Tim A. Majchrzak
Department of Information Systems
University of Mnster
Mnster, Germany
tima@ercis.de

AbstractMany software developments projects fail due to quali- developed. But not all software development projects fail; in
ty problems. Software testing enables the creation of high quality fact, many companies produce software systems of notable
software products. Since it is a cumbersome and expensive task, quality. We propose to study effectual development to discover
and often hard to manage, both its technical background and its best practices for reaching quality especially with regard to
organizational implementation have to be well founded. We testing. In combination with the processes and techniques for
worked with regional companies that develop software in order the development of software, software testing is the foundation
to learn about their distinct weaknesses and strengths with re- of software quality [17].
gard to testing. Analyzing and comparing the strengths, we de-
rived best practices. In this paper we explain the projects back- To better support businesses with results from academic re-
ground and sketch the design science research methodology used. search, a combination of research in information systems and
We then introduce a graphical categorization framework that software engineering is a feasible approach [21]. We undertook
helps companies in judging the applicability of recommendations. a project with regional enterprises and tried to learn what
Eventually, we present details on five recommendations for tech- makes software development projects successful. After identi-
nical aspects of testing. For each recommendation we give im- fying the companies' status quo [21], we analyzed the myriad
plementation advice based on the categorization framework. of observations we made and the experiences the projects par-
ticipants shared with us. Eventually, we derived a set of novel
Keywords: Software testing, testing, software quality, design
best practices.
science, IT alignment, process optimization, technical aspects
It appears to be easy to say how software development
I. INTRODUCTION should be done. But although techniques are described in the
literature and there is knowledge about successful develop-
Striving for improved software quality is no new emer- ment, this knowledge has not necessarily been transferred into
gence. The idea to optimize technical aspects respectively to business reality. Some of the best practices we found have been
use technology to achieve this aim is known for decades. Un- denoted earlier e.g. in different contexts or with different pre-
surprisingly, the term software engineering [28] has been requisites. However, adopting them seems to be very challeng-
coined in the 1960s and the software crisis is knownand un- ing. We thus give details on how to implement the recommen-
fortunately still lastingsince the 1970s [8]. dations and which conditions have to be met. We also name
Especially large-scale projects that end in disasters nurture related work for each recommendation rather than discussing
the public's picture of unreliable software. An example is the them in a section of their own. Best practices presented in this
NASA Mars Climate Orbiter, which crashed because metric work have a technical focus; suggestions for the organizational
and imperial units were mixed in a software subsystem [27]. embedding of testing can be found in [20].
The miscalculation leading to the crash would most likely have This paper is organized as follows. Section II introduces the
been detected by detailed software testing. Unfortunately, there project's background. We sketch our research methodology in
are many other examples of failed major projects that have Section III. A framework for categorization is explained in
similar root causes: inscrutable, ill-designed or not exhaustively Section IV. Five effective technical recommendations are dis-
tested software [17]. cussed in Section V. A conclusion is drawn in Section VI and
Despite the widely perceived disasters, the main problem is future work is highlighted in Section VII.
failure of everyday projects [6][10]. Even after decades of re-
search, no silver bullet has been found and many problems II. BACKGROUND
remain unresolved [4]. Complexity of software obviously in-
Mnster is located in North Rhine-Westphalia, Germany. In
creases faster than methods to control it are developed [16]. As
the city and its surrounding region a lot of IT-based companies
a consequence, problems of varying severity can be found in
have been sited. Most of them are medium-sized and specialize
projects in any industrial sector and for any kind of software
This article is an extended and revised version of the paper presented by
Majchrzak [22] at the i-Society 2010 conference in London on 28 June, 2010. 1|P age
http://ijacsa.thesai.org/
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 1, No. 4, October 2010

on software development. Some larger companies with far over course is impossible to describe the perfect testing process or to
1.000 employees do not develop software for customers; as offer a general description on how to test software. However,
financial service providers their individually developed soft- we searched for a larger number of satisfactory solutions that
ware enables their business processes. The number of their address typical problems. Finding such satisficing [31] solu-
developers exceeds the number of employees most of the tions helps enterprises even though not all possible problems
smaller companies have in total. can be addressed.
All companies are members of the local chamber of com- Since we wanted to learn about problems from the point of
merce which supports the Institut fr Angewandte Informatik view of the participating companies, we decided for a qualita-
(IAI Institute of Applied Informatics). The IAI is hosted by tive approach [26]. Best practices can hardly be found with a
the University of Mnster and based on the work of both com- simple questionnaire. Thus, we conducted semi-structured ex-
puter scientists and economists. Projects undertaken by the IAI pert interviews. Using only a rough guideline for the interviews
are run by academics that seek both research progress and [19], we were able to learn about how testing is done in the
mean to support the local industry. companies. As the interviews developed, distinct weaknesses
and strengths could be identified as well as common problems
By frequent exchange with companies the IAI learned and successful strategies discussed with the participants. The
about their dissatisfaction with software testing. While most data gained in each interview is far too verbose to be published
companies were ambitious to improve the quality of the soft- as such. But each of it forms a kind of case study [36] which
ware they developed and to cut down costs for testing, they did greatly aids further analysis.
not know how to achieve this. Additionally, many enterprises
lack the time to try out new technologies or to evaluate changes Recommendations derived from the discussion with the in-
to their processes. However, the companies were not economi- terview partners are meant to complement the literature. Even
cally endangered and apparently developed software of quality. comprehensive work on software testing processes [2] or quali-
Thus, two conclusions could be drawn. Not a single company ty improvements [19] does not cover all problems typically
has a perfect testing process. All of them face a number of test- faced by practitioners. Some ideas published also do not seem
ing related problems. Nevertheless, each company has devel- to be directly accessible to practitioners. Along with literature,
oped distinct strengths that help it in creating good software that promotes result-driven testing [13], we want to help clos-
products. ing this gap. Technical aspects as depicted in this work should
be given special attention. If conducting IT research, it should
Based on these observations the IAI project to improve be kept in mind that information technology is studied [25]
software testing was initiated. Two main purposes were set: even if organizational aspects are likewise important.
Firstly, the status quo of software testing in the regional enter-
prises was to be brought to light. Secondly, successful strate- A quantitative analysis would augment the qualitative ap-
gies used by the companies were to be identified and aggre- proach. Without quantitative data it is hard to prove that a rec-
gated to best practices. In this work we present five major best ommendation is effective and efficient. However, deducing
practices that change or influence the technical way of software best practices is a first step and was very laborious; verifying
testing or the technology used. results was identified as a further step (see Section VII).
From the exchange with the enterprises and due to the di- The course of action we took can be sketched as follows.
versity of software developed as well as the differences in cul- We began by contacting IAI supporting companies and by
ture in each company, we expected strengths to be complemen- identifying staff for the interviews. Both managers and techni-
tary. Hence, it could be estimated not only to find a few known cally skilled employees were chosen. In a second step we inter-
methods for successful development but a plethora of promis- viewed the participants. While there usually was only one
ing attempts to increase software quality and to optimize longer interview done with smaller companies, medium-sized
processes. and larger companies were visited more than once. We were
able to address both organizational and technical issues with
Diversity is both a blessing and a curse. It helps to identify the respective experts. In the interviews we tried to identify
best practices that form recommendations unknown to most who is responsible for testing, when it is done, what is included
companies and therefore highly beneficial to them. At the same in tests (graphical user interface, interfaces to other systems,
time, prerequisites have to be met so that a recommendation etc.), which methods are used and how testing is generally
can be adopted at all. Consequently, a framework is needed to done. We also tried to learn about the usage of testing tools
support companies in choosing which recommendation to im- [23].
plement. The framework is described in Section IV and used
for each recommendation in Section V. After discovering the status quo, we discussed general
problems met and successful strategies found. This included
III. RESEARCH METHODOLOGY evaluating which improvements the participating companies
desired. Eventually, potential best practices were discussed
The project was meant to combine scientific rigor with re- with them. This part of the interview was the most open one. A
levance and efficiency as demanded by businesses. We decided lot of ideas were exchanged and many interesting approaches
for a methodology based on design science which addresses considered.
important unsolved problems in unique or innovative ways or
solves problems in more effective or efficient ways [15]. It of

2|P age
http://ijacsa.thesai.org/
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 1, No. 4, October 2010

The third step was to analyze the results and to aggregate


data. As it is of high interest to the regional companies, an

Target
state
overview of the status quo has been drawn. For reasons of

Level of demand
space and scope it is not included in this paper but can be found

Advanced
in [21]. By finding interdependencies as well as aligning and
judging best practices identified by the participants, we ex-
tracted recommendations. Of course, particularities of the com-
panies' situations were taken into account. This lead to the con-

Basic
struction of a framework (see Section IV) that describes the
conditions under that a recommendation applies. Component Integration System Acceptance
Phase of testing
Project size
IV. FRAMEWORK
Small Medium Large
Recommendations for a topic as complex and intertwined
on various levels as software development require a sophisti- Kind of development
cated categorization. Their full value can only be accessed if it Individual software Mass market
is known how to use them and which prerequisites have to be
met. Besides, support on deciding which best practices are Releases
most applicable for the own business is advisable. We thus use One Several Regular
the framework first described in [20] to classify recommenda-
tions.
Figure 1. Exemplary use of the framework
The level of demand shows how great the organizational
change required to adopt a recommendation is. Basic recom- As represented in Figure 1, the level of demand and the
mendations should be adopted by any company. If recommen- phase of development are used to set up a matrix. A tick indi-
dations are not only basic hints but require considerable effort cates that a recommendation is meant to be beneficial for the
to be implemented, they are considered to be advanced. Even- depicted phase and level. Ticks might be shown in brackets
tually, target states are ideals that cannot be reached unlabored. which indicate that benefits will be observable but might be
In fact, they are guidelines on what level of perfection can be less pronounced than for other phases and levels. The three
reached and require a process of continuous optimization. other determinants are shown as bars. A shade of (dark) gray
However, the benefits of an actual implementation will be im- means that a recommendation applies under the specified con-
mense in the long run. ditions. Fading indicates that adoption of the recommendation
should be considered if the depicted determinant is met. Rec-
It is important to consider the project size. Small-sized ommendations still require more detail so that companies can
projects commonly have a single team that does development judge them. However, the framework can be used to get a
and testing. For medium-sized projects these tasks are underta- quick overview of the main prerequisites for it.
ken by at least two teams. Large projects can comprise hun-
dreds of employees and include general departments that con- Please consider Figure 1 for clarification:
tribute to it. If thinking about a recommendation, not just the
The recommendation requires advanced effort. It
sole number of employees that participate in it should be taken
is possible to be extended in order to mark a target
into account. In fact, the typical size of projects as well as their
state in which beneficial effects will be much
character should be kept in mind.
stronger.
Another important determinant is the kind of software de-
veloped. Based on contracts, individual software is developed Implementing it especially aids integration and
for a single customer. Usually, there is close contact to the system testing. Positive effects are also likely to be
principal. Standard or mass market software often is developed observed for acceptance testing.
over a long period of time. This makes regression testing im- The recommendation is meant to be adopted for at
portant. Many recommendations can be applied to both kinds least medium-sized projects and it aims at indivi-
of software. dually developed software.
Similarly, the number of releases of a software product has It aims at individually developed software. Theo-
to be taken into consideration. It is differentiated between one, retically, there could be a fading of the gray shade
several and regular releases whereas regular means that there into the box for mass market software. This would
will be releases for some month or years. mean that it would also benefit while the main fo-
The fifth determinant distinguishes between the phases (or cus was individual software.
stages) of testing. It is divided into the phases of component For full effect, there should be a greater number or
test, integration test, system test and acceptance test that also regular releases of the software developed.
can be found in the literature [35].

3|P age
http://ijacsa.thesai.org/
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 1, No. 4, October 2010

V. TECHNICAL RECOMMENDATIONS

Target
state
The following sections present five recommendations for

Level of demand
the optimization of technical aspects of software testing. Their
order reflects the implementation complexity.

Advanced
A. State-of-the-art Development Environment
The first recommendation is pretty straightforward. We en-

Basic
courage using the latest development environments available,
particularly integrated development environments (IDE) that Component Integration System Acceptance
are customizable and support plug-ins. They offer magnificent Phase of testing
opportunities to increase the quality of the developed software. Project size
Using the latest IDEs is especially appealing since development Small Medium Large
software is used anyway and many of these products or at least
plug-ins for them are free. Kind of development

Admittedly, using an IDE is not only about testing. But the Individual software Mass market

support it offers significantly helps to increase development Releases


quality. If the developer is aided in his routine work, testers do
not have to struggle with defects in programs that originated in One Several Regular

unthoughtfulness. Testers can then concentrate on finding ac-


tual bugs e.g. in algorithms. Consequently, this recommenda- Figure 2. Classification of Development Environment
tion is a testing best practice even though parts of it do not di-
rectly deal with testing; they have a noticeable indirect impact. the most powerful IDEs. It supports Java, C/C++ and (by using
extensions) many other languages such as PHP. Even though a
Unlike expectation, companies do not necessarily use state- lot of functionality is built-in, there is a four-digit number of
of-the-art IDEs. It is common to do so for individual develop- plug-ins to enhance it further (an exemplary site that lists them
ers in small enterprises. However, once the choice of develop- is [9]). To benchmark the development environment used, it is
ment tools is not solely based on developers' discretion but a good idea to compare it to leading IDEs. Speaking with the
there are general guidelines or even mandatory directives, tools participants showed that some of them used IDEs that were far
that do not offer as much functionality as would be possible are from offering what Eclipse or Microsoft Visual Studio (the
used. This is especially true for situations in which developer leading tool for .NET) do. Partly the functionality does not
PCs are centrally set-up by IT organization staff rather than by even reach what the leaders provided years ago.
the developers themselves. Changing development tools could
not be possible since tools for cooperative work or versioning, Coloring the source code to point up the syntax (syntax
or software to access corporate-wide storage systems or re- highlighting) [7] and automated suggestions while typing (code
source pools might not be exchangeable. completion) are common. Documentation fragments can be
shown directly to e.g. prevent the usage of methods marked as
Some of the participants drew a picture of the way their de- deprecated. Many IDEs also offer direct checking of the code
velopment is supported by the tools used that reminded us of so mistakes are immediately highlighted. Partial compilation
the 1980s. There was no kind of syntax highlighting and no can provide error information without the need to explicitly
built-in supportive functionality to aid the developer with cod- invoke the compiler. Thus, software with syntax errors will not
ing. There was no direct access to the programming languages even be tried to compile and will be fixed by the developer
or library documentation; developers would look it up on the before they consider it to be finished.
Internet or use books even for the simplest questions. And,
probably worst, there was no testing and debugging support. Semantic correctness cannot be guaranteed automatically
Debugging was done by putting print()-statements into the but many typical mistakes can be prevented. For example, le-
code that almost arbitrarily supplied the developers with frag- vels of warning can be defined. We strongly encourage enabl-
ments (or rather shreds) of information. ing this feature. Eclipse can for example show Java warnings
by underlining code in yellow color. A variable that may take
Seeing how much more productive developers using mod- the value of null but is used without checking for this will be
ern IDEs are and how much these tools aid them in achieving marked. Consequently, code that provokes so called Null-
high quality software, we strongly recommend using up-to-date PointerExceptions can be fixed. Many other mistakes
development environments and the functionality that comes can be prevented from being made. Despite an unfamiliar feel-
with them. This general recommendation is suitable for any ing programmers might have in the beginning, they are getting
company. It is extremely helpful for component testing (see used to the warnings quickly. Superfluous warnings usually can
Figure 2). be disabled; in Java this e.g. can be done by using so called
If IDEs are used that do not offer some of the more sophis- annotations [3].
ticated functions and cannot be extendede.g. with plug-ins The next step is checking code rules. IDEs do not offer this
upgrading to a more recent version or another IDE is recom- functionality but there are tools and plug-ins available. While
mended. Eclipse arguably is the most widely known and one of

4|P age
http://ijacsa.thesai.org/
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 1, No. 4, October 2010

the above described warnings are generated by the compiler


and shown by the IDE, tools for checking code rules have a

Target
state
logic on their own which makes them more powerful. Moreo-

Level of demand
ver, they are customizable and allow having corporate-wide

Advanced
coding standards enforced. While developers should not feel
patronized, having common standards is highly recommended.
Many problems arise when several developers work on the
same code and probably misunderstand what their colleagues

Basic
did. This is particularly problematical if developers introduced
the style of their choice and then leave the company while the Component Integration System Acceptance

code they wrote has to be maintained. This can be prevented by Phase of testing
having corporate-wide schemes and conventions. We suggest Project size
using tools or plug-ins to check compliance with general cod- Small Medium Large
ing standards suggested by the programming language vendors
(e.g. [33]), literature (e.g. [34]) and company-specific addi- Kind of development
tions. Individual software Mass market

We also advocate using the debugging functionality of Releases


modern IDEs. Instead of printing out variable contents, modern
One Several Regular
trace debuggers visualize the complete state of a program at a
point of the developer's choice. Pointers can be followed and
variables modified; execution can be continued step-by-step. Figure 3. Classification of Test Case Management
Visualizing graphs for control flow and data flow further aids
debugging. Combined with knowledge on modern debugging Thus, the main purpose is formalization and structuring.
techniques [11] the debugger of a state-of-the-art IDE is a tool Ideally, each employee knows exactly what he has to do at any
of immense power and versatility. time and can look up that information in a test case manage-
ment tool. To a certain degree he can choose from tasks yet
To sum up, we strongly recommend using a modern IDE, unassigned. When pursuing these tasks, he likely will spend his
even if giving up old libraries, methods, paradigms or even efforts with high efficiency. The tool should also be able to
programming languages is a precondition. Along with this report a project's status which is especially helpful for large
process, binding standards for formatting source code and for projects. The added effort for entering test cases can be mini-
naming variables, methods etc. should be set up. For a better mized with intelligent help from the tool. Besides, regression
understanding how the optimal usage of a programming lan- testing is improved.
guage can be supported by an IDE, practitioner literature such
as [3][24] is recommended. There is a plethora of work on pro- Despite not many facts on test case management being pub-
gramming best practices that can be utilized to augment this lished, we know of one detailed work. Parveen et al. present a
recommendation. case study [30] on the implementation of a centralized test
management using TestDirector, a tools by then sold by Mer-
cury Interactive. While the study is different in context and
B. Test Case Management and Database
scope, experiences are similar to our observations of the bene-
In small projects testing often is seen as a stateless task. ficial effects of test case management.
Tests are done once a module is finished and found defects are
corrected directly. This is repeated at the levels of integration The test case management's functionality can be extended
and system testing. Unfortunately, it is inefficient and cannot successively. Not only can it be used more precisely but addi-
be combined with a holistic view [20] of testing. Hence, we tional functionality can be added. It is a good idea to include
recommend using a test case management tool. It already helps support for requirements engineering. Tasks can be derived
medium-sized projects that have at least a couple of releases. from requirements and test cases can be linked to them. Should
While the later phases of testing are supported with little effort, test cases fail, the employee responsible for the requirement
the solution can be expanded and will be beneficial for all might be able to help. Reporting can also help to find modules
phases of testing (see Figure 3). that have a high rate of defects which probably result from mis-
takes in their requirements.
Typical functions include the compilation and categoriza-
tion of test cases, ideally using a highly customizable interface Especially for products that are continuously refined, inte-
that supports users with suggestions to disburden them of repe- gration of a bug tracker is recommended. This software is used
titive tasks, and setting statuses of test cases. Optionally, as- to report and manage defects (bugs) and therefore ideal for
signment of tasks and responsibilities can be done. A tool integration with test case management. Bug trackers can be-
should further support cooperative work and offer reminders come an interface to the technical staff of the customer. A
(via e-mail) for employees about assigned tasks, nearing dead- wealth of further functionality can be easily added.
lines and other important dates. Connections to the environ- Test case management is thought to be an interface between
ments that run test cases are another amenity. They allow test- steps of processes. Erstwhile informal and hardly checkable
ing to be triggered automatically. process components are represented by it. Information is pro-

5|P age
http://ijacsa.thesai.org/
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 1, No. 4, October 2010

vided in a structured form. The management software is used only once. The test cases can simply be reused. It will only be
for any operative testing procedures. In fact, each testing needed to add more test cases if the new algorithm has an ex-
process begins and ends with utilizing it. tended functionality. With a good test case management, this is
even true if the second algorithm has been implemented month
Beginning with using table calculation sheets can foster the or years after the first one. Without such a system, the old test
creation of an integrated system that offers interfaces to other cases most likely have been deleted in the meantime, were lost
tools. Expanding the test case management should be done along abandoned data, or there will be no knowledge how to
step-by-step. Both a bottom-up (beginning with component use them.
tests) and a top-down approach (beginning with system tests or
even acceptance tests) are possible. Media disruption should be We advocate both using test case management and a test
avoided as it lowers efficiency. An example for media disrup- case database. They are especially successful if they are inte-
tion is to write results from a test run to a piece of paper and grated (see Section V.D).
later type the results from the paper into a tool. If a strategy for
implementation is worked out in advance, a delay of projects is C. Aligning Systems for Testing and Production
unlikely and a return-on-investment (ROI) should be achieved Utilizing modern programming languages and paradigms
timely. While implementation details are out of scope of this for developing complex distributed applications does not only
paper, we strongly advise setting up a test case management. bear advantages. Developing applications on workstations but
On the first look a test case database appears to be equal to deploying them to servers or mainframes is prone to compati-
test case management software. While both purposes can be bility and scaling problems.
combined in one tool, there is a functional distinction between In the very beginning of programming, software only ran
them. Test case management serves towards the aim of struc- on the system it was developed for. For any other platform the
turing and documentation. A test case database is driven tech- code at least had to be adjusted. It might even have been easier
nically. It is used to collect test cases in executable form and to rewrite it from scratch if architectures were entirely differ-
stores components such as test stubs and mock objects. The ent. Nowadays the environment used for development and test-
main aim is to increase the rate of test case reuse and hence to ing usually differs from the one software is developed for.
facilitate regression testing. Moreover, at least an operating system is mediating with the
Test case databases are usually integrated into tools but can hardware. In most cases virtualization hypervisors, application
be implemented separately. Test cases have to be saved in a servers and other components form additional layers. This has a
structured way and it should be easy to find and retrieve them. plethora of amenities. Using high level programming languages
Ideally, the database system can directly invoke the environ- allows for the compilation of the same code for different plat-
ment test cases are coded for and run them. It is very helpful forms; virtual machines and other components can even offer
for data-driven applications if (e.g. relational) databases can be hardware abstraction. However, the productive system often is
stored along with test cases. Arbitrary testing results are pre- far more powerful and not only its hardware is different but
vented since the database can be reset to a defined and consis- often the software is different, too.
tent state for any test cases that requires this. In simple cases, differences only apply to the workstations
A test case database has amenities beyond the mere reuse of and servers' operating systems. However, additional compo-
test cases. A good strategy for testing larger software systems nents such as libraries, database management systems (DBMS)
is to have a defined suite of test cases and run it both for an old, or application servers are likely to be different as well. Server
correctly working version and the new version of the software. versions of these systems will probably not even run on
If results differ, defects are likely. The same applies to test da- workstations. Consequently, problems arise. To give an exam-
tabases. First the database is set to a defined state. Then the test ple: Java EE applications are commonly run in a sophisticated
suite is run for the old version of the software. The same is application server such as IBM WebSphere. Workstations often
repeated for the newer version of it after the database has been run a lightweight Apache Tomcat. Even though an application
reverted to the defined state. Resulting states are compared that runs on Tomcat should seamlessly do so on WebSphere,
since differences hint to problems. If results are identical but practice shows that unexpected behavior or crashes can be ex-
the old version is known to be buggy, problems have apparent- pected. This can be explained with a different interpretation of
ly not been fixed. While such testing is possible manually, tool specifications, differing versions, conflicting libraries and simi-
support avoids mistakes and saves much manual labor. lar issues.
Test case databases are also useful if libraries are developed We recommend aligning development and testing systems
that are incorporated into several other systems. They can be with the intended productive environment. By alignment we
tested even if changes were made. Changes to interfaces or mean to reasonably adjust development and productive hard-
defined functionality are noticed immediately without deploy- ware and software while keeping the effort economically feasi-
ing the library to productive systems. ble. It will in most cases e.g. not be justified to buy a second
mainframe system just to have a testing platform that is equal
The strengths of test case databases are most apparent if re- to the productive system. Nevertheless, options are often avail-
gression testing is used. Consider an example: If two algo- able that guarantee a high technical compatibility but are cost-
rithms for the same purpose but with different runtime charac- effective. Exactly to find these options alignment is about.
teristics have to be tested, test cases have to be implemented

6|P age
http://ijacsa.thesai.org/
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 1, No. 4, October 2010

Business/IT alignment in general is subject to lively scientific


discussion [5].

Target
state
Level of demand
Aligning systems is suggested for at least medium-sized
projects with a couple of releases. It is especially useful for

Advanced
individually developed software and early development phases
(see Figure 4). Due to our observations we even deem addi-
tional effort to align systems justified. Surprisingly, no work

Basic
seems to be published on system alignment for the reasons of
testing. Component Integration System Acceptance
Phase of testing
The more advanced a testing phase is the more alike should
systems be. Compatibility problems should however be re- Project size
solved as early as possible. Achieving this can be easier than Small Medium Large
thought. For example, lightweight versions are available for
common server applications. This applies to the earlier WebS- Kind of development
phere example; Tomcat should be used on the client only if the Individual software Mass market

target system is Tomcat either.


Releases
Instead of installing a DBMS on the developing system, the One Several Regular
one installed on the server can be used remotely. A separated
database should be created to protect productive data from cor-
ruption. Modern servers and to an even higher degree main- Figure 4. Classification of Aligning Systems
frames offer virtualization that allows to completely separate
instances not only of databases but of any application. Thus, D. Integration of Tools
testing is possible on the same machine and with the same sys- We learned from the participants that using tools for testing
tem software that the application will eventually run on. Re- is common. A general observation was that tools are hardly
source usage should be protected so that a tested application integrated. However, exactly this is recommended.
running into a deadlock or massively using resources does not
endanger productive applications running in parallel. Testing tools are applications on their own in most cases.
Common formats or defined interfaces seldom exist. Only larg-
Applications accessed by a number of parallel users require er tools such as IBM Rational products provide an interchange
realistic testing. Problems that arise with memory usage or pa- of data. Most participants desired the integration whereas only
rallel execution can hardly be found with systematic testing. few of them actually had experiences with it. We recommend it
Such problems will not reveal themselves if just trying out for medium-sized and larger projects with at least a couple of
the application on the testing system. An acceptable perfor- releases. Due to the high complexity some effort is required
mance on the testing system cannot be assumed for the produc- before benefits can be observed for the phases of integration
tive system even if it is more powerful. Not yet considered de- and system testing. Ultimately, amenities can be realized for all
pendencies, growing data and similar issues can cause prob- phases (see Figure 5).
lems in the (far) future. Thus, testing has to be done under rea-
listic conditions. Defects in parallel algorithms might only re- Several kinds of integration are desirable. First of all, do-
veal themselves under certain conditions. Race conditions in cumentation systems should be linked with systems for testing.
which several threads of an application obstruct each other will Undocumented tests are only worth a fraction of documented
probably occur on fast systems only. Reasonable conclusions ones. Automatically synchronizing the results of execution with
about an application's performance can solely be drawn when the test case management system (cf. Section V.B) disburdens
thoroughly testing them in a productive environment. testers of repetitively entering test cases. A well structured do-
cumentation as described in [16] can be achieved more easily.
Besides all advocating to testing under realistic conditions, An improved database of testing results can also be used for
we strongly advise not to test on productive systems without statistical examination. Test managers can easily learn about
making sure that productive data cannot be modified and that running times, success rates of test cases and similar data. For
the performance remains unaffected. Negative (side-) effects on regularly released software integrating the bug tracker with the
productive systems would render any benefit of realistic testing management system is another option. Reported bugs can be
useless. adjusted with known defects and test cases. This decreases re-
dundancy.

7|P age
http://ijacsa.thesai.org/
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 1, No. 4, October 2010

Target

Target
state

state
Level of demand

Level of demand
Advanced

Advanced
Basic

Basic
Component Integration System Acceptance Component Integration System Acceptance
Phase of testing Phase of testing
Project size Project size
Small Medium Large Small Medium Large

Kind of development Kind of development


Individual software Mass market Individual software Mass market

Releases Releases
One Several Regular One Several Regular

Figure 5. Classification of Integration of Tools Figure 6. Classification of Customizing of Tools to Fit wih Processes

Linking systems for test case execution supports regression business processes are changed in order to fit with a tool's re-
testing. Test cases run in earlier phases can easily be repeated. quirements. Without changing the processes, many tools can
Automated synchronization again relieves testers of repetitive hardly be used. Alternatively, customizing tools is possible but
tasks and ineffective work. Connections to test case manage- very laborious. However, tools should be tailored to fit with
ment systems and a test case database can make tasks econom- business processes and not the other way around.
ically feasible that would be too laborious if done manually.
Especially companies that have defined testing processes
The above ideas will seem utopian in a development land- pointed out, that changing processes to enable the usage of
scape without integration. They should motivate alignment and tools is a particularly bad idea. In fact, tools should be custo-
encourage challenging the status quo. To our knowledge there mizable in order to seamlessly integrate into the processes.
is no exhaustive solution offered and there are no well-defined Therefore, we recommend selecting tools based on their cus-
standards. Individually developing tools for integration will be tomizability. While introducing a new tool could be used to
unavoidable. Nevertheless, for tools newly bought integration benchmark the affected business processes, well performing
capabilities can be checked. Even small tools for data transfor- processes should not be changed. Customizing tools should be
mation can yield dramatic reductions of manual workload. A done in larger projects with at least several releases of a soft-
tool for aggregating data and computing statistical reports from ware product. The benefits will become most obvious if a com-
the test documentation can e.g. be implemented with little ef- pany strives for a continuous optimization of its testing
fort and refined continuously. processes. Optimizations will be most apparent in tool-driven
phaseshence, there will be hardly an effect on acceptance
By undertaking a strategy of small refinements, integration testing (see Figure 6).
is possible without much trouble or high costs. Growing know-
ledge will bolster further development. We found open source In general, introducing new tools or installing upgrades of
software to be convenient for integration. It can be modified to existing tools entails changes. They for example are caused by
work with existing software with ease. the implementation of additional phases of testing or the addi-
tion of new functionality. This kind of changes is both normal
Full integration of tools enables new possibilities. This in- and desirable. Companies should try to optimize their
cludes installing a test controlling which is used to keep an processes, though. Adapting the course of action and proce-
overview of the testing process and to calculate key figures dures given by the tool should be a starting point for own con-
[20]. The vision is an integration of systems that comprise test siderations. Only in a small number of cases these presets will
case management, development (project) planning, test sche-
align with a company's standards. Consequently, a well-
duling, staff assignment, time control, task management, con- founded strategy of integrating a tool has to be found. Moreo-
trolling and even a management cockpit. ver, evaluations of the tool's performance should be scheduled.
Experiences gained after using it for a while should be used for
E. Customizing of Tools to Fit wih Processes further improvements.
In most cases, testing tools are driven by the underlying
technology. Even if they can be customized, they induce a cer- Implied processes are often based on technical details of
tain way in which they have to be used. As a consequence, tools. We learned in the underlying project of this article that
only a small number of testing tools can be intuitively used.

8|P age
http://ijacsa.thesai.org/
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 1, No. 4, October 2010

Thus, tools should be checked for their customizability upon preferred over changing processes in order to be able to work
evaluation and selection. Steps for creating a test case should with tools.
be designed to align with employees' flow of work. If the do-
cumentation, demonstration materials, or tool presentations hint We found a discrepancy of testing knowledge described in
to fixed and unchangeable presets, tools have to be carefully the literature and the reality of testing in enterprises. To give an
checked. It is particularly impedimental if enforced processes example: Even practitioners literature such as [1] distinguishes
cannot be divided into substeps or if tools lack interfaces. A between black box and white box testing. However, hardly any
common problem would be tools for test execution that cannot of the project participants made an explicit distinction like this.
be integrated with documentation software. Not a single participant had ever heard of gray box tests. The
above described recommendations might thus be partly found
Adaptability and customizability can be given in several in the literature but many companies have not implemented
dimensions. Technically speaking, it should be possible to in- them, yet. Apparently, literature is inaccessible for some practi-
terrupt tests during execution in order to save intermediate re- tioners, not practically usable in the everyday work, or un-
sults. Moreover, interfaces to import and export data are very known [30]. This has also been found for organizational as-
helpfulin particular, if they can be used in real time (cf. Sec- pects of testing [20]. Research progress and testing improve-
tion V.D). With regard to the usability, a configurable interface ments that were hoped for [14] seem to have reached the indus-
positively affects the acceptance of a tool. The possibilities to try only partly.
tailor a tool should be based on its complexity. Customizing in
the technical dimension (e.g. by writing scripts) is acceptable Developing software of high quality is not a mere economic
for small tools only. Ideally, tools should offer the possibility to obligation. Neither is it just needed to improve the idea the
load plug-ins. Furthermore, tools that are plug-ins by them- general public has about software quality. Preventing that soft-
selves and can be loaded into an integrated development envi- ware harms humans in any way is an ethical obligation [11].
ronment (IDE) are well suited. They help to design continuous We thus encourage further research in both organizational areas
processes. (i.e. information systems research) and in the technical field
(e.g. computer science and formal methods). Moreover, we
The experiences we gained in the project suggest that it is a encourage enterprises to reach a culture of testing instead of
successful strategy to carefully calculate the effort required for perceiving testing as a costly delay in the development process.
changes to tools. This effort commonly is preferable to the dis- We therefore propose a structured approach and to keep re-
advantages of adapting the processes. Besides, customizing search bound to cooperation with enterprises.
tools offer the chance to reflect on the testing processes and
optimize them. In the long run, even small changes have great VII. FUTURE WORK
effect. Irregularities caused by hardly changeable tools are like-
ly to cut productivity. Moreover, when tools are not customized The project this work is based on is continued in order to
or no tools are introduced at all due to the strategy of saving the evaluate the recommendations found. Future work will contain
effort of selecting and adapting them, the company might loose a discussion of the results with practitioners and probably a
competitiveness. Improving processes and using cutting-edge quantitative study. Ideally, a study could be done on a national
tools will improve testing and raise the quality of the developed or even global scale. It could not only try to capture the success
software. of the recommendations but check how the literature on soft-
ware testing is used.
VI. CONCLUSION It is without question that a quantitative analysis would per-
In this paper we presented results from a project that aimed fectly augment the qualitative approach. For example, measur-
at finding best practices for software development and especial- ing a return-on-investment (ROI) of the improvements made
ly testing. We described its background, the research approach would be ideal [28]. Without quantitative data it is hard to
and the framework used to categorize recommendations. Out of prove that a recommendation is effective and efficient. Deriv-
about 30 recommendations found and classified with the ing best practices is a first step and was very laborious due to
framework, we presented five recommendations that make the problematic nature of software testing. Verifying results
novel contributions to the technical dimension of how testing implemented by companies was identified as a further step.
can be done in enterprises. Design sciencethe research approach of our choiceis in-
crementally iterative [15]; adding additional rigor and verifying
Using a modern IDE greatly supports development. It results actually implemented by companies was identified as a
enables testing to focus on finding bugs rather than on eliminat- further step.
ing mistakes that entered the code by neglectfulness. Using test
case management and a test case database leads to a structured ACKNOWLEDGMENT
testing process. Moreover, it supports regression testing.
Alignment of testing and productive systems prevents many The author would like to thank Herbert Kuchen for discuss-
problems that arise due to incompatibilities and scaling issues. ing the recommendations and for his continuous support and
Integrating testing and development tools requires continuous encouragement. Special thanks go to the companies that parti-
governance but increases testing performance and efficiency. cipated in the project. It would not have been possible without
Consequently, regression tests can be run much more efficient- their willingness and help.
ly. Finally, customizing tools to fit with processes should be

9|P age
http://ijacsa.thesai.org/
(IJACSA) International Journal of Advanced Computer Science and Applications,
Vol. 1, No. 4, October 2010

REFERENCES H. Grob, B. Hellingrath, S. Klein, H. Kuchen, U. Mller-Funk, and


G. Vossen, editors, Arbeitsbericht Nr. 127. Institut fr
[1] R. Black. Pragmatic Software Testing. Wiley, Indianapolis, 2007. Wirtschaftsinformatik, WWU Mnster, 2010.
[2] R. Black. Managing the Testing Process. Wiley, Indianapolis, 3rd [24] S. Meyers. Effective C++: 55 SpecificWays to Improve Your Programs
edition, 2009. and Designs. Addison-Wesley, 3rd edition, 2005.
[3] J. Bloch. Effective Java. Prentice Hall, Upper Saddle River, 2nd edition, [25] J. Morrison and J. George. Exploring the software engineering
2008. component in MIS research. Commun. ACM, 38(7):8091, 1995.
[4] F. P. Brooks, Jr. The mythical man-month (anniversary ed.). Addison- [26] M. D. Myers. Qualitative research in information systems. MIS
Wesley, Boston, 1995. Quarterly, 21(2):241242, 1997.
[5] Y. E. Chan and B. H. Reich. IT alignment: what have we learned? [27] NASA. Mars climate orbiter mishap investigation board phase I report,
Journal of Information Technology, 22, 2007. 1999.
[6] R. N. Charette. Why software fails. IEEE Spectrum, 42(9):4249, 2005. [28] P. Naur and B. Randell. Software Engineering: Report of a conf. spon.
[7] M. F. Cowlishaw. Lexxa programmable structured editor. IBM J. of by the NATO Science Committee, Garmisch, Germany. Scientific
Research and Development, 31(1):7380, 1987. Affairs Division, NATO, 1969.
[8] E. Dijkstra. The humble programmer. Communications of the ACM, [29] W. J. Orlikowski and C. S. Iacono. Research commentary: Desperately
15:859866, 1972. seeking the IT in IT researcha call to theorizing the IT artifact. Info.
[9] Eclipse marketplace. Online: http://marketplace.eclipse.org/ (Access Sys. Research, 12(2):121134, 2001.
date: 14 September 2010). [30] T. Parveen, S. Tilley, and G. Gonzalez. A case study in test
[10] R. L. Glass: Computing Calamities: Lessons Learned from Products, management. In ACM-SE 45: Proceedings of the 45th annual southeast
Projects, and Companies that Failed, Prentice Hall, Upper Saddle River, regional conference, pages 8287, New York, 2007. ACM.
1999. [31] H. A. Simon. The sciences of the artificial. MIT Press, Cambridge, 3
[11] D. Gotterbarn and K. W. Miller. The Public is the Priority: Making edition, 1996.
Decisions Using the Software Engineering Code of Ethics. Computer, [32] S. Slaughter, D. Harter, and M. Krishnan. Evaluating the cost of
42(6), 6673, 2009. software quality. Commun. ACM, 41(8):6773, 1998.
[12] T. Groetker, U. Holtmann, H. Keding, and M. Wloka. The Developers [33] Sun Microsystems, Inc. Code Conventions for the Java Programming
Guide to Debugging. Springer, 2008. Language. Online: http://java.sun.com/docs/codeconv/ (Access date: 14
[13] D.-J. de Grood. TestGoal: Result-Driven Testing. Springer, Heidelberg, September 2010).
2008 [34] H. Sutter and A. Alexandrescu. C++ Coding Standards. Addison-
[14] M. J. Harrold. Testing: a roadmap. In ICSE Future of SE Track, pages Wesley, 2004.
6172, New York, 2000. ACM. [35] J. Watkins. Testing IT: an off-the-shelf software testing process.
[15] A. R. Hevner, S. T. March, J. Park, and S. Ram. Design science in Cambridge University Press, New York, 2001.
information systems research. MIS Quarterly, 28(1), 2004. [36] R. K. Yin. Case Study Research: Design and Methods. Sage
[16] IEEE. IEEE Std 829-2008: IEEE standard for software ans system test Publications, London, 3rd edition, 2002.
documentation. New York, 2008.
[17] C. Jones. Software Quality: Analysis and Guidelines for Success. AUTHORS PROFILE
Thomson Learning, 1997.
[18] D. Kopec and S. Tamang. Failures in complex systems: case studies, Tim A. Majchrzak is a research associate at
causes, and possible remedies. SIGCSE Bulletin 39(2):180184, 2007. the Department of Information Systems
[19] W. E. Lewis. Software Testing and Continuous Quality Improvement. of the University of Mnster, Germany,
Auerbach, Boston, 3rd edition, 2008. and the European Research Center for
[20] T. A. Majchrzak. Best practices for the organizational implementation of Information Systems (ERCIS). He
software testing. In Proc. of the 43th Annual Hawaii Int. Conf. on received a BSc and MSc in Information
System Sciences (HICSS-43). IEEE Computer Society, 2010. Systems from the University of Mnster
[21] T. A Majchrzak. Status Quo of Software Testing - Regional Findings and currently finishes his PhD. His
and Global Inductions. Journal of Information Science and Technology, research comprises both technical and
7(2), The Information Institute, 2010. organizational aspects of software
testing. He has also published work on
[22] T. A. Majchrzak. Best Practices for Technical Aspects of Software several other interdisciplinary IS topics.
Testing in Enterprises. In Proc. of the Int. Conf. on Information Society
(i-Society 2010), IEEE Computer Society, 2010.
[23] T. A. Majchrzak and H. Kuchen. Handlungsempfehlungen fr
erfolgreiches Testen von Software in Unternehmen. In J. Becker,

10 | P a g e
http://ijacsa.thesai.org/