Sie sind auf Seite 1von 44

An AUTOMATED TESTING INSTITUTE Publication - www.automatedtestinginstitute.

com

A
S
utomated .......
T
oftware esting MAGAZINE
March 2011 $8.95

The Art of
Linking Just Like
Clockwork
Measuring and
Improving Automation
reliability

Image Based
Automation
A non-intrusive, GUI
neutral approach

TestLink/Hudson
Automation
Open source automated
Test management

Finishing the Internet: Performance Implications of Moving from IPv4 to IPv6


Verify/ATI Conference
2011
The Test Conference You’ve Been Waiting For

Save The Date


September 26 - 28, 2011
Crowne Plaza Hotel
Arlington, VA

Venture Beyond the Abstract


www.verifyati.com
A
S
utomated
T
oftware esting
March 2011, Volume 3, Issue 1

Contents
The Art of Linking
No software project has unlimited resources, therefore, the ability to engineer optimal solutions based on the
limited resources made available on a project is definitely an art worth mastering. This issue provides guidance on
how to link commercial, open source and home grown resources together in order to end up with an automated test
solution that maximizes your ability to achieve your goals.

Features
Image Based Automated Software Testing 14
Need a non-intrusive, GUI neutral automated test solution? Image based automation may be for you. Read this article to learn
about image based automation and the tool requirements that point to this type of automation as a solution. By Elfriede Dustin
Manage Your Automated Tests With TestLink and Hudson 20
This article offers an effective, efficient and... free solution for managing automated tests via open source tools. Read this article
to learn about Hudson and TestLink, and about integrating these tools for managing your tests. By Bruno P. Kinoshita

Just Like Clockwork 24


Most effective automation implementations use some type of framework for test development and execution. This article
provides information that will allow you to determine your automated test framework reliability. By David Fern

Columns & Departments


Editorial 4 AST Comics 32
The Art of Linking Enjoy a testing comic strip that is funny and informative.
Linking tools and concepts for effective automation.
Go On A Retweet 38
Authors and events 6 Read featured microblog posts from the web.
Learn about AST authors and upcoming events.
Hot topics in automation 40
Sweat the technique 8 Finishing the Internet!
Asserting Yourself
Performance test implications of IPv4 and IPv6 Internet
Learn how to create a custom assertion. communications.
Open Sourcery 10 The Automated Software Testing (AST) Magazine is an Automated Testing
Institute (ATI) publication.
Mobile Source For more information regarding the magazine visit
Demand for and impediments to open source mobile apps. http://www.astmagazine.automatedtestinginstitute.com

March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 3


Editorial

The Art of Linking


by Dion Johnson
My professional career has been greatly
impacted by what I learned in college,
which may seem odd given the fact that
I’m not doing anything remotely close
to what I studied in school. I graduated
as an Electrical Engineer skilled in
building circuits, processing signals, and
calculating electromagnetic induction. I
had a little red tool box that we engineers
carried around with resisters, wires of
varying gauges, diodes, LEDs, small
circuit boards, and more. To this day I
still have that red tool box in my closet,
although I’m not sure why because most
of that stuff makes little sense to me now.
I did surprise my wife a couple of years
back when I dusted off that kit and used
its contents to repair the wiring in the software project has unlimited resources, provides an approach for managing
cord of our treadmill that was chewed up therefore, when I join a contract, I first automated tests by linking together two
by a pet, but that is about the extent of assess the resources that are available and open source tools - TestLink and Hudson
my electrical engineering abilities these - via an open source plug-in. Finally, Mr.
then come up with an optimal solution David Fern adds a special perspective
days.
for linking those resources together to on linking together automation elements
Among the skills that I still find useful achieve the desired goals. into a framework that is optimized with

...it wasn’t enough just to complete it. It had to be completed


such that the output from the finished product was maximized.

from my engineering days, however, is This issue of the AST magazine is respect to reliability. Entitled, “Just Like
the ability to come up with an optimal dedicated to the art of linking together Clockwork”, Fern’s article continues AST’s
solution given some finite, limited set resources and the quality implications of quality attribute series by introducing three
of resources. As electrical engineers, we doing so. Ms. Elfriede Dustin provides a characteristics of reliability and identifying
were often given problems and projects approaches by which they may be measured
featured article entitled “Image Based
to solve and complete with a specific set and improved.
Automated Software Testing” that
of resources and inputs. With the given introduces image based test automation The department articles also focus on
resources and inputs it was possible to and addresses linking commercial image links; specifically the “Hot Topics in
complete the project in many different Automation” department. In this issue
based tools with homegrown solutions
ways, but it wasn’t enough just to the hot topic is “Finishing the Internet”,
for meeting test automation needs. In a
complete it. It had to be completed such and it focuses on linking the existing IPv4
featured article entitled “Managing Your
that the output from the finished product Internet to the emerging IPv6 Internet
Automated Tests With TestLink and
was maximized. and the performance testing implications
Hudson”, Mr. Bruno P. Kinoshita also of doing so.
Although I’m now in the world of talks about links by discussing a tool with
software, these skills still follow me. No the word “link” in its name. Kinoshita So let the linking begin!

4 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011


Training That’s Process Focused Yet Hands On
Software Test Automation Training
www . training . automatedtestinginstitute . com

Public Courses
Software Test Automation Foundations
Automated Test Development & Scripting
Designing an Automated Test Framework
Advanced Automated Test Framework Development

Virtual Courses
Automated Test Development & Scripting
Designing an Automated Test Framework
Advanced Automated Test Framework Development
Test Automation Framework Workshop: A Shortcut to Success

Come
participate in a set
of test automation courses Featured Course
that address both fundamental
and advanced concepts from a
theoretical and hands on perspective.
These courses focus on topics such
as test scripting concepts, automated
framework creation, ROI calculations
and more. In addition, these courses
may be used to prepare for the
TABOK Certification
exam.

Public and Virtual Training Available


Authors and Events

Who’s In This Issue?


Elfriede Dustin is considered a leader in the
automated software testing industry. She’s authored
A
S
utomated
T
oftware esting
various white papers on the topic of automated
software testing, as well as the book Effective
Managing Editor
Software Testing. Elfriede was also the lead author
Dion Johnson
of Automated Software Testing and Quality Web
Systems, and co-author of Testing SAP, The Art Contributing Editor
of Software Security Testing and her latest book, Edward Torrie
Implementing Automated Software Testing. Among her many accolades
Elfriede has also graced the Automated Software Testing Magazine and the Director of Marketing and Events
Automated Testing Institute Online Reference with multiple contributions Christine Johnson
on the topic of software test automation. Elfriede has over 20 years of IT
experience, implementing effective testing strategies, both on government
A PUBLICATION OF THE AUTOMATED
and commercial programs, and currently works for IDT (www.idtus.com) –
TESTING INSTITUTE
an innovator in automated software testing solutions. Elfriede is also on the
ATI’s Test Automation Body of Knowledge (TABOK) board.

David Fern is a seasoned software testing


professional with more than 12 years of experience
breaking software in the public and private industry.
Currently he is an Automation Test Engineer and
Software Testing Technical Lead at the Social Security CONTACT US
Administration. He has developed and implemented AST Magazine
software-testing strategies for large government Internet astmagazine@automatedtestinginstitute.com
application deployments as well as enterprise wide Point
ATI Online Reference
of Sale (POS) systems in the private sector. He has trained and mentored contact@automatedtestinginstitute.com
teams on processes and strategies for testing new technologies such as Web
Services. David has had multiple articles and techniques published in various
testing publications and online sites including the Automated Software
Testing Magazine, and has also given presentations at many of the industries
leading conferences. Additionally, he is a Certified Project Management
ATI and Partner Events
Institute (PMI) Project Manager Professional (PMP), an ISTQB/ASTQB
Certified at the Tester-Foundation Level (CTFL), Mercury Interactive (HP) May 2011
Quick Test Professional 9.0 Specialist and Microsoft MCSE and MCDBA. ATI 2011 Automation Training
Schedule Begins
Bruno P. Kinoshita lives in Sao Paulo in http://training.automatedtestinginstitute.com
Brazil. He currently works as a consultant with Sysmap
Solutions (www.sysmap.com.br). Bruno has 7 years September 26-28, 2011
commercial IT experience, mainly in telecom services, Verify/ATI Conference 2011
with Claro, Tim and Vivo (mobile carriers from Brazil). www.verifyati.com
His experience covers Java development, requirements
analysis, system architecture, test management and
quality analysis. Bruno holds SCJP, SCWCD, ITIL and Cobit certifications
and is a member of various open-source projects such as Hudson, TestLink
and Crux. You can find more about Bruno P. Kinoshita through his personal
website www.kinoshita.eti.br.

6 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011


For more information, visit http://www.networking.automatedtestinginstitute.com
March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 7
Sweat the Technique

Asserting Yourself
Create Your Own Custom Assertions in 4 Steps

A
ssertions are functions that Step 1. Identify the pertinent
check a true/false statement function arguments
within a software program.
These functions, which only pass when Line 1 of the function defined in Figure 1
the statement is evaluated as True, are an begins the function definition and reveals
excellent way to implement verification some typical assertion arguments. Table 1
points within an automated test because defines these arguments.
they provide a mechanism for concisely
identifying a condition that must be
checked and presenting an error message
if the check fails. Many programming
languages have standard libraries with
built-in assertions, but not all of them.
Even if a programming language has
assertions, it is still useful to be able
to create custom assertions that can
perform activities not included in built-
in assertions; activities such as aborting
tests and/or the application in the event of Table 1: Function Parameters
a test failure.
assigned a value somewhere within an the
A simple condition assertion can be
automated script, therefore upon checking
created via the following 4 steps:
the equality of these two variables, a
1. Identify the pertinent function value of either True of False will
arguments be returned. If the condition returns
Step 2. Evaluate Condition True, then line 3 is executed. Line 3,
2. Evaluate condition
in an effort to keep the sample function
3. Evaluate additional arguments Lines 2 through 13 reveal how a simple, merely generates a message box
4. Test the function typical if-then-else function may with a defined pass message, but in an
be used for evaluating a basic condition. actual implementation you may want
Figure 1 provides a VBScript example A condition may be something as simple to include a statement that sends a pass
of how such a function may be produced. as a = b. The variables a and b will be message to your test report and performs
some additional actions. If the condition
returns False, then the block of code
represented in lines 5 through 12 will
be executed. Line 5 simply generates a
message box with a defined fail message.

Step 3. Evaluate Additional


Arguments
While line 5 represents the basic steps
to take in the event of a failure, lines 6
through 12 provide a structural example
of how to process additional arguments in
the event of a failure. Some application
failures render further test activity
useless, and may even render the current
Figure 1: Custom Assertion Function (VBScript) context of the application useless. In these

8 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011


Sweat the Technique

and/or AUT.

Step 4. Test the function

Figure 2 provides an example of


how the function may be tested. Lines 1
through 5 set the pertinent variables while
line 7 actually defines the condition that
Figure 2: Function Unit Test is evaluated by the assert function. Since
a and b are not equal, the assert condition
situations, it may be advisable to abort the Each of these blocks will execute in the will fail. Thus, the failingMsg text
test run and/or the AUT in the event of event that their respective variables are will appear in a message box. The
a failure. Such runtime decisions may be set to the value of True. For the purposes abortTest argument is True, but
set in motion by the values entered in the of this article, message boxes are used to the abortApp argument is False,
abortTest and abortApp function show that each block is being successfully therefore the code block associated with
arguments. Lines 6 through 8 handle the entered into. In a real implementation, the the former will be executed while the
abortTest argument, while 10 through message boxes will be replaced with the code block associated with the latter will
12 handle the abortApp argument. appropriate syntax for aborting the test not.

Training That’s Process Focused Yet Hands On


Software Test Automation Training
www . training . automatedtestinginstitute . com

Public Courses
Software Test Automation Foundations
Automated Test Development & Scripting
Designing an Automated Test Framework
Advanced Automated Test Framework Development

Virtual Courses
Automated Test Development & Scripting
Designing an Automated Test Framework
Advanced Automated Test Framework Development
Test Automation Framework Workshop: A Shortcut to Success

March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 9


Open Sourcery

Mobile Source
The Demand For and Impediments to Mobile Open Source Applications

A wise person once said, “Wherever software from taking over


there is software technology, there will the mobile development
be a demand for free software tools space. That is not how reality
associated with that is playing out, however. While
technology.” Actually, Black Duck reveals notable
no one ever said movement in mobile open
that… but someone source development trends,
should have said it the 9000 open source mobile
because it certainly projects make up only a
seems to be holding small percentage of the total
true. As you 365,000 projects listed in
probably know, free their Knowledgebase (http://
and open source software www.blackducksoftware.
(FOSS) has become a force com/knowledgebase). Also, there are
to be reckoned with in Google’s still some roadblocks that open source
the world of desktop Android. Of the software must overcome including
applications. 3,800 new mobile projects, security concerns and cheap commercial
But, the same can 55% are for the Android, while software.
reasonably said for the iOS is in second with 39%. Much to the chagrin of many open
mobile applications as well. And This is noteworthy, because Android is source advocates, security continues
once a few hurdles are cleared, open itself open source and is also the leading to be a roadblock for open source
source mobile software may become a mobile operating systems used in mobile applications whether desktop or mobile.
threat to commercial mobile software in devices. Developers are taking note of For businesses, it is a chief concern,
the same way that open source desktop this trend and are focusing their efforts which is borne out by the results of
software has become a serious competitor on the creation of tools such as Sencha’s a Forrester 2009 survey. According
to commercial desktop software. Event Recorder which is a test suite and to TechRepublic, this Forrester study
Over the past few years, the number infrastructure for automating the testing revealed that:

...once a few hurdles are cleared, open source mobile software


may become a threat to commercial mobile software ...
of mobile open source software projects of all web apps on Android devices • 58% of large companies had
has grown tremendously. Black Duck (http://www.prweb.com/releases/sencha/ security concerns about open
Software, a global provider of products eventrecorder/prweb8184572.htm). source.
and services for automating the
According to CNET, Apple is still • Two-thirds of small to midsize
management, governance and secure
the leading provider of smartphones, businesses had security concerns
use of open source software analyzed
but Android (which is used in multiple with open source.
data from nearly 9000 mobile FOSS
projects listed in their Knowledgebase devices) is the leading mobile OS
• 9% of enterprises said they were
and discovered that 3,800 new projects (http://reviews.cnet.com/8301-19736_7-
“very concerned” with open
were started in 2010 alone (http:// 20039240-251.html). source security.
www.blackducksoftware.com/news/ With a popular open source operating • 45% of small to midsize
releases/2011-03-02) system that is used on multiple devices, businesses were “very
And guess which platform is leading and an upward trend in mobile open concerned” with open source
the charge? If you said Apple’s iOS you source development, it would seem security.
are incorrect! The correct answer is that there’s nothing to stop open source

10 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011


Open Sourcery

(See the following link for because it provides a larger


more information: http:// window of opportunity for
www.techrepublic.com/blog/ cybercriminals to do their work.
opensource/companies-still- PCWorld, in an article discussing
concerned-about-open-source- the recent Android DroidDream
security-really/682) malware – which was more of an
Given the recent rise of open market problem as opposed
mobile malware, security to an open source problem –
is likely to be a concern states:
for mobile device users as “Android is an open source
well. McAfee has recently platform with much more lenient
access to the Android Market. That
released its “McAfee Threats freedom can also be exploited,
Report: Fourth Quarter 2010” Figure 1: New Mobile FOSS Projects though, to slip malicious apps
(Credit: BlackDuckSoftware.com)
document, which not only into the mainstream.” (http://www.
noted the rise of malware but currently have a window of opportunity pcworld.com/businesscenter/
identified threats against the SymbOS article/221247/droiddream_becomes_
to exploit a variety of mobile platforms.
android_market_nightmare.html)
(free) and Android (open source) More consumers are using mobile
operating systems as the most interesting devices and tablets in their daily lives as So while security problems are not
(http://www.mcafee.com/us/resources/ well as at work. Enterprises must now necessarily isolated to mobile open source

So if an app is more readily available, of higher quality and


almost free (i.e. $0.99) then a consumer may more likely pay a
small fee.
reports/rp-quarterly-threat-q4-2010.pdf). support more devices than ever before, software, this type of software still must
in effect extending their corporate overcome the perception of insecurity
The report also includes the following
firewalls and services to places they
statements: before it can really compete with mobile
may not be prepared for.”
“Threats to mobile platforms (primarily commercial software.
smartphones) are not new; however, Some suggest that the “open” nature Cheap commercial software is also
cybercriminals seem to have renewed of mobile open source software, while apparently a big road block for open source
their interest for a variety of reasons.
As in most crimes it’s a matter of attractive in some respects, may make software as well. Part of the problem is
opportunity, and cybercriminals it the most vulnerable to malicious code that open source apps for mobile devices
are clashing with some mobile app
marketplaces. For example, open source
apps that fall under the GPL license type
are getting push back from Microsoft’s
and Apple’s market places according to
an article in The Register (http://www.
theregister.co.uk/2011/03/04/open_
source_in_mobile/). This may make it
difficult for consumers to get access to
some open source apps. Then, at times,
commercial products are of higher
quality than open source products that
do the same thing. So if an app is more
readily available, of higher quality and
almost free (i.e. $0.99) then a consumer
may more likely be willing to pay the fee.
While mobile open source software is
definitely on the rise, it still has a ways
to go and a few hurdles to overcome
before it is a serious contender to mobile
Figure 2: Manufacturer Operating System Share – Smartphones commercial software.
(Credit: CNET)

March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 11


12 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011
March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 13
Image Based
Image Based
Automated Software
Testing
5 Indicators That Image Based
Automation Is For You

By
Elfriede Dustin

14 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011


M
ost all of the current
vendor provided automated
software testing tools, such
as Hewlett Packard (HP) QuickTest
Professional (QTP) or IBM Rational
Functional Tester (RFT) are GUI technology
dependent. That means if non-compatible
proprietary programming GUI language
features or non-compatible third party GUI
controls are used to develop the GUI for a
system under test (SUT), such automated
testing tools will fail in those areas.
Some automated testing groups might
use workarounds for those specific
incompatibility issues, but others might
want a tool that is GUI technology neutral.
One such class of GUI technology neutral
automated testing tool utilizes a concept
known as “imaged based” automated
software testing. With image based
technology, automated tests recognize GUI
elements of the SUT via sophisticated image
search algorithms that are independent of the
type of GUI that may be in use. The image
based automated testing tool baselines the
test script images as expected results and
then compares the baselined scripts during
test execution against actual results.
This article discusses image based
automated testing technology, describes
some of the tools that are image based,
chronicles my organization’s incorporation
of imaged based technology, clarifies some
of the myths that exist surrounding the
technology, and finally provides some of its
caveats and limitations.

Automated Software Testing Magazine 15


5 Image Based Indicators
1 Must be non-intrusive to the system under test (SUT)
Many major vendor provided tools require that at least portions of their tool be installed on the SUT, and
thus are intrusive to the SUT. In some environments such intrusion is prohibited. (*SL1, SL4, SL5, SL6)
2 Must be Operating System (OS) independent
Some SUTs work on multiple operating systems (i.e. Windows, Linux, Solaris, etc.), thus requiring
test tools that are able to do the same. (*SL1, SL4, SL5, SL6)
3 Must be Graphical User Interface (GUI) technology independent
Many SUTs provide GUIs that are written using multiple technologies (i.e. Motif, C#, etc.) and use a
wide variety of third party, non-custom, GUI controls. Effective automation therefore requires a tool
that is able to handle such diverse environments. (*SL1, SL2, SL3, SL4, SL5, SL6)
4 Must be able to handle display and non-display centric automation
A tool must often handle operations on an SUT’s GUIs as well as on an SUT’s backend (i.e. for
verifying the transmission of various messages and data). (*SL6)
5 Must be able to handle a networked multi-computer environment
A tool may be required to operate effectively in an environment with multiple servers and multiple
monitors/displays all interconnected to form one SUT. (*SL4, SL5, SL6)

(*There are several useful image based tools currently in existence. Each requirement above references one or more image
based testing tools - from a short list (SL) of tools listed in this article - that meet the requirement. A brief description of each
tool - SL item - may be found on the subsequent pages of this article.)

Figure 1: Automation Test Tool Requirements Met By Image Based Tools

1
5 Image Based Indicators decided to continue our search for other
tool options that allowed for non-intrusive
automated testing and discovered
independent. Our client had a variety
of operating systems, but initially we
performed our proof of concept on
Must Be Non-Intrusive To SUT
eggPlant, which at the time was owned by Windows and Linux. eggPlant only
Originally our search for tools that Redstone. eggPlant had great potential. ran on a Mac and at the time they were
met this specific requirement led us to Like VNCRobot we found that it used working on a Windows appliance. With
VNCRobot, which at the time was a VNC technology and met our requirement the Windows appliance one still would
freeware tool. This tool used the Virtual of not being intrusive to the SUT. After have to use a Mac with the Windows
Network Computing (VNC) technology evaluating eggPlant and VNCRobot for appliance connected to the Mac. TestPlant
to connect one computer to another. three months we decided that eggPlant had eventually developed a solid Linux
VNCRobot itself was the “VNC client” features important to us that were absent eggPlant license, allowing our framework
and could access and control the remote in VNCRobot. For example eggPlant to meet the required OS independence. In
computer running a “VNC Server” via the provided an “image collection” feature addition, TestPlant released their official
RFB (remote framebuffer) protocol. See which during test execution allowed Windows license in 2010.
Figure 1 for a pictorial view of this setup. for image searching and comparison of
When we got in touch with the
VNCRobot author we ascertained that
he was the sole developer working this
various similar images. We therefore
chose to incorporate eggPlant into our
overall automated testing framework that
3
Must be Graphical User Interface
(GUI) technology independent
Many of the current vendor provided
project, so my colleagues and I had a we called Automated Test and Re-Test
automated software testing tools are GUI
critical question to answer: (ATRT).
technology dependent. That means if
“Did we want to hinge the success of
our project on one developer?” 2
Must be OS independent
The tool we selected as part of our
proprietary programming languages or
third party controls are used within the
SUT, the tool would not be compatible
Our answer was “No”. Therefore, we ATRT framework also needed to be OS thus presenting problems for testing.

16 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011


node 1
Image-based Testing
Tools (Short List)
node 2
Capture Playback SL1: SeeTest
Tool
node 3 (http://experitest.com)
Experitest’s
System Under Test SeeTest product
suite is based
on a patented
Figure 1: VNC and Capture Playback Tools technology using
self-learning
eggPlant, through the use of VNC, and plugins can be added, removed, diagnostic
interacts with all GUI elements of the or modified easily through the Eclipse and matching
SUT as images, independent of the GUI framework. With this we created algorithms and a modular self-enhancing
technology used. This is a common an extension point that defined an image recognition technology. SeeTest
feature of many image based automated interface for our framework and the uses a client-agent architecture and
test tools, and should not be confused user to interact with an external tool in a plugs-into all existing test automation
with bitmap, coordinate-based checks. common manner. environments such as QTP, C#, JUnit,
Unlike with traditional bitmap checks, Python and Perl. In addition, it provides
After defining an efficient specific techniques to connect to external
images that move on the screen can
development environment, we now devices, such as using existing SDKs,
typically still be located via the image
needed a way to develop message data. and more.
based tool’s search algorithm.
We initially evaluated Fireeye from
NIST for an orthogonal array approach

4
Must be able to handle display of data generation, but it didn’t allow SL2: Project Sikuli
and non-display centric for the complexity of data interactions
automation and relations required. We decided
(http://sikuli.org/)
Another requirement our customer upon Matlab for our data generation Sikuli is a visual
had was that a tool/framework be able (http://www.mathworks.com/products/ technology to automate
to test various backend (non-GUI) matlab/) and test graphical
interfaces using an array of protocols. user interfaces (GUI)
using images. Sikuli
5
Unfortunately, eggPlant did not Must be able to handle
support this requirement out-of-the networked multi-computer includes Sikuli Script,
-box, so we had to develop an in-house environment a visual scripting
solution for this backend “interface” API for Jython, and Sikuli IDE, an
While most of the industry mainly seems integrated development environment for
testing ourselves. This was a complex to focus on web based applications and
undertaking because we first needed writing visual scripts with screenshots
web testing, the DOD world in which easily. Sikuli Script automates anything
to understand the requirements, the we worked consisted of networked
make-up of the backend system and you see on the screen without internal
computers (i.e. multiple servers, systems API’s support.
the message data. Each SUT used of systems, monitors and displays)
a different protocol (TCP/IP, UDP, interconnected to form one SUT.
CORBA, proprietary) and all used
different message data formats. Thus,
eggPlant, with its VNC use, handled SL3: RoutineBot
the interconnected systems well and we
we developed an approach where all developed ATRT in such a way that it (http://www.routinebot.com/)
protocols and data could be tested via could support this networked mesh of RoutineBot is a
our ATRT framework. environments. For example, it handled visual automated
This approach used the open source distributed and concurrent testing over a test tool that
Eclipse development environment. We network, allowing automated tests to be provides its users
chose the Eclipse Rich Client Platform executed concurrently over a network. with the capability
because it could be expanded on the This was useful for automating test cases of creating and
ATRT base environment to allow that involved various GUI or message executing tests
for configurability/extensibility via based outputs that were dependent on based on image
a powerful plugin framework. New each other, moved over a network or patterns in the application under test.
feature sets and tools are integrated, ran in parallel. As long as a VNCServer

March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 17


was installed on any of the networked baseline, if feasible
computers, ATRT (which also served as
a VNCClient) could connect to them.
• Use image recognition tolerances. Image-based Testing
100% image pattern recognition is
not always required and a lesser Tools (Short List)
Caveats And Limitations Of tolerance can still pass the test,
Imaged Based Automation making the tests less brittle SL4: eggPlant
• “Re-skinned GUIs are often a (http://www.testplant.com)
As previously mentioned, if your
problem. Image based technologies
automated testing tool needs to meet eggPlant is
and many other automated testing
one or more of the requirements listed, TestPlant’s
tools currently fail horribly in the
image based testing might be the image-based
case where a GUI interface has
solution. test automation
been re-skinned. Currently no
There are many pros to having a major workaround exists for this product that uses VNC technology.
GUI technology and OS independent limitation. My group intends to
solution available. However, the current work on implementing a feature
crop of image based testing tools comes in our framework that can mass- SL5: T-Plan Robot
with some caveats and limitations. replace the skin on the images (http://www.t-plan.com/robot/)
Following is a list of limitations as well used, so the image baseline is not
as some guidelines to help overcome
T-Plan Robot is a Java
rendered useless. Functional automated
those limitations:
testing tool that is
• Use text input for navigation, Summary developed on generic
whenever feasible, such as tabbing image based testing
to desired fields, using hot keys, While image based automated testing principles, and provides
using function keys, etc. solutions solve some of the existing a human-like approach to software
• Cut/paste text into a field, automated testing tool woes, it is testing.
whenever feasible instead of using not yet the automated testing silver
Optical Character Recognition bullet many may seek. Image based
(OCR). While OCR reliability has automated testing is, however, a viable SL6: ATRT
increased over the years, it is still solution when you have to solve unique (http://www.idtus.com)
not 100%. challenges such as the ones presented in
this article. For more
• Capture the smallest image information
possible. Large images leave a lot Wait! There’s More! on Automated
of room for duplicate images to be Test and Re-
found or other errors when trying Test (ATRT) please contact http://idtus.
to find a match. com/contact
For more information on image based
• Resolution matters. Though some testing tools and other types of tools, feel
vendors claim resolution doesn’t free to visit the ATI Online Reference
matter, we have found it still often Tool Index at:
does. Use the same resolution
as part of your SUT baseline, if http://www.tools.automatedtestinginstitute.
feasible. com

• Color depth matters. Use the same


color depths as part of your SUT

Note
It’s worth noting that the evaluation discussed at the beginning of this article was nearly 4 years ago and both VNCRobot
and eggPlant have undergone several changes since then. In 2009 VNCRobot was acquired by T-Plan Ltd. and renamed
T-Plan Robot (http://www.t-plan.com/robot/). T-Plan Robot now also comes with the “image collection” feature and more.
Also, a reduced feature, open source version of T-Plan Robot is available under a GPL license.
eggPlant has also undergone various changes since our initial evaluation as well, including being acquired by TestPlant
(1998).

18 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011


3rd Annual

ATI Automation
Honors
Celebrating Excellence in the
Discipline of Software Test
Automation

Nominations Open April 1!


www.atihonors.automatedtestinginstitute.com

March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 19


Successful, effective and efficient
test automation often
involves Linking
several tools together

straightforward task, therefore I decided


to build a Hudson plug-in to integrate
TestLink and Hudson. In this article we
will discuss this plug-in and an effective
and efficient automation solution built
with TestLink and Hudson.

What is TestLink?
By Bruno P. Kinoshita TestLink is a web based test management
system. Its use is governed by the terms of
the GNU General Public License. TestLink
contains features that allow for the creation
of Test Projects, Test Specifications, Test

Manage Your Automated Plans, Test Execution, Automated Tests,


Reports, Metrics and many others entities.
The basic organizational units in TestLink

Tests With TestLink and are Test Projects. Within a Test Project you
can create your Test Specification, add
Test Plans and execute your tests.

Hudson I have worked with two other companies


here in Sao Paulo helping them to choose
a test management tool, and TestLink was


chosen in both companies.
Find a way to manage our automated tests.”
What is Hudson?

T
his was the directive I received in September of last year from my employer and
I know from past experience that it actually meant “find an effective and efficient
solution for managing our automated tests.” I had a teacher at my university Continuous Integration (CI) has become
that was obsessed with the etymology of words, so I remember exactly what effective a popular development practice in many
and efficient mean. Effective is something that accomplishes a purpose, and efficient companies. Among CI tools Hudson
is being effective without wasting time or effort or expense. Since I had previously has had some noticeable success since
participated in the decision to use TestLink as a test management tool and Hudson as a it won the Java Duke’s Award in 2008.
continuous integration server in my employer company, the idea of using TestLink and Hudson is extremely lightweight, fast,
Hudson together to manage our automated tests seemed to be the best. TestLink already easy to install and extensible. The project
had an API to facilitate external integration, and extending Hudson can be a pretty started in Sun with the Software Engineer

20 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011


• The plug-in was built with the support of the Hudson
community and since a majority of the Hudson
community voted for the renaming of Hudson, it
makes sense for the plug-in to be renamed as well and
maintained for Jenkins.

For the purposes of this article, however, Hudson will continue


to be referred to as Hudson.

Setting up your environment

Now let’s discuss the actual implementation of TestLink


and Hudson. Here is a list of the tools and their associated
dependencies for setting up your automation environment for
Figure 1: VNC and Capture Playback Tools TestLink and Hudson:
1. TestLink 1.9.0
Kohsuke Kawaguchi and now is maintained by a large and active a. Web Server with PHP support (e.g.: Apache Http
community. Server)
In May 2008, Dan Dyer (http://dandyer.co.uk) wrote a post b. Relational Database (MySQL, Postgres, MS SQL
with the title “Why are you still not using Hudson?” From his or Oracle)
article you can pick up some key reasons for using Hudson: 2. Hudson 1.395
• Easy to install a. Java 1.5 or superior
• Ease-of-configuration 3. TestLink Hudson Plug-in 2.1
• Distributed builds
• Out of the box support for CVS, SVN and many plug-ins TestLink Installation and Setup
to extend it Upon ensuring these dependencies are met, the next step is
And as an update to this list, I would add community support, to install TestLink. TestLink 1.9.0 may be installed using the
large set of documentation and examples, and the price, of following steps:
course, as it is free. 1. Download TestLink 1.9.0 from http://www.teamst.org
By the end of last year some issues involving the infrastructure and extract the contents of the compressed file into the
of Oracle (that had acquired Sun Microsystems and all its root directory of your Web Server.
products, including Hudson) arose in the Hudson community, as 2. Create the database for TestLink.
well as some concernment about Oracle stewardship and control 3. Access the TestLink application from the browser and
of the project. So in the beginning of this year the community follow the wizard to complete the installation.
voted for a rename of the Hudson project to the Jenkins project.
Oracle decided to continue maintaining the original Hudson, Once you’re done with the installation, log into TestLink and
while Jenkins is maintained by most of what was the Hudson create a Test Project. Next create a custom field with name and
community. label set to ‘Class’ and assign it to your Test Project.
Now, create a Test Specification that consists of at least one
Test Suite and then create a Test Case with the Execution Type
What is the future of the plug-in? set to “Automated” and in the Class custom field fill in with the
name of a TestNG Java Class (see Figure 1 for a illustration of
With the split of Hudson into two tools, comes a split in the a test case in TestLink, and Figure 2 for a sample TestNG Java
user community. Some will stay with Hudson, while a large Class).
majority of the community (including its founder and principle The next step is to create a Test Plan. During this creation,
contributor, Kohsuke Kawaguchi) will use and help maintain do not forget to check the public and active options. Once that is
Jenkins. Therefore, a decision had to be made about the current complete, add the Test Case to the Test Plan.
TestLink Hudson Plug-in. It has thus been decided that from this
point forward the plug-in will be called the TestLink Jenkins Finally, before installing Hudson, the XML-RPC availability
Plug-in with no further updates being made to the existing in TestLink must be enabled. Open the config.inc.php file
Hudson Plug-in. The reasons for this decision are: that is in the TestLink installation folder. Find the following line:

• It would be too troublesome to maintain both versions $tlCfg->api->enabled = FALSE;

March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 21


1 package br.eti.kinoshita.sample;
2
3 import org.testng.Assert;
4 import org.testng.annotations.Test;
5
6 /**
7 * Sample test.
8 * @author Bruno P. Kinoshita - http://www.kinoshita.eti.br
9 */
10 public class MyTest
11 {
12
13 @Test
14 public void testCurrentTimeMillins()
15 {
16 Assert.assertTrue( System.currentTimeMillis() > 0 );
17 }

Figure 2: Sample TestNG Java Class

Replace it with the following line: By default Hudson listens to port 8080, so it may be accessed
at http://localhost:8080/. To install the TestLink Plug-in, open
$tlCfg->api->enabled = TRUE; Hudson and go to Manage Hudson -> Manage Plugins and
select it for installation.
Go to TestLink and look for the “My Settings” link in the top
of the page. There the devkey used by the plug-in to access Once the plug-in installation is completed Hudson must be
TestLink may be generated. restarted for the installation changes to take effect. Next, go to
Manage Hudson -> Configure System and add your TestLink
Installing Hudson and the TestLink Plug-in installation information to the TestLink section as illustrated in
Figure 3.
Hudson is relatively easy to install. Go to http://www.hudson-
ci.org and download the Hudson 1.395 WAR file. Then, use the Now, create a FreeStyle Job and add a step to invoke TestLink
command line to open the directory where the WAR file was (see Figure 4). Any value can be entered into the Build name
saved, and execute the following statement: field. When the plug-in is not able to find a build in TestLink it
creates a brand new one. The name of the newly created custom
java –jar hudson-1.395.war field must be added, however, because this custom field is also

Figure 3: TestLink Information In Hudson Figure 4: FreeStyle Hudson Job

22 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011


the key custom field. JUnit
The next step is to go back up and find the Advanced button. For JUnit, the plug-in uses the name of the Test Suite. If the
Click on it and check the “Use custom workspace” checkbox. Test Suite name is the same as the value of the key custom field
This will show an input text field used to indicate what will be the of a Test Case, then the plug-in checks if the failure or error
build workspace. For simplicity the workspace in this example is attributes of the Test Suite are greater than 0 (zero). If so, the
pointing to the Java project with the automated tests. test case is updated with the status failed. Otherwise it is updated
Now, save the configuration and execute the build by clicking with passed.
on Build Now. The plug-in uploads the resulting TestNG file as
evidence of the run.
Conclusions

Strategies used by the plug-in to find TestLink can be effective for managing your tests, be they
test results manual or automated. Hudson has many features to facilitate the
execution and schedule of your automated tests. By using the
After the automated tests are executed the plug-in has to look TestLink Hudson Plug-in you can put them to work together. The
for test results to determine whether or not an automated test was plug-in is responsible for integrating both tools and orchestrates
executed correctly and to update the test status in TestLink (as the activities. Another advantage on using the plug-in is that it is
shown in Figure 5). The TestLink Hudson Plug-in currently has extremely easy to add your automated tests to your continuous
three strategies for finding Test Results: TAP, TestNG and JUnit. integration process, as you are already running them inside the CI
All three of these strategies use a key custom field to associate a server. All you will need to use this solution is a set of automated
test result with an automated test case. tests that can output in TAP, JUnit or TestNG and a few minutes
to prepare your entire environment.
TAP

Contribute Content Today


For TAP, the plug-in uses the TAP file name as the key. If the
TAP file name is the same as the value of the key custom field of a
Test Case, then the plug-in checks whether the TAP file contains
errors (not OK). If so, the test case is updated in TestLink with
the status set to failed. Otherwise it is updated with passed.

TestNG
For TestNG, the plug-in uses the class name of the test as the Announcements &
key. Usually a TestNG XML report contains a suite with many Community
Comments Box Blog Posts
tests. The plug-in looks for the name of the classes of the tests. If
a TestNG class name is the same as the value of the key custom
field of a Test Case, then the plug-in checks the status of the
methods of the class. If a class contains a method that failed,
then the Test Case is updated with the status failed. Otherwise it Automation
is updated with passed.
Events

As a registered user you can submit content


directly to the site, providing you with content
control and the ability to network with like
minded individuals.

Learn more today at


http//www.about.automatedtestinginstitute.com
Figure 5: TestLink Results

March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 23


Like Clockwork

by David Fern

Just Like Clockwork


Making Your Test Automation Architecture
Reliable
Does your software test automation architecture work like clockwork?

Most software test automation projects rely on some type Reliability is defined as:
Functionality
of framework or architecture to operate as efficiently and “A set of attributes that
effectively as possible, and we the test engineers are often Reliability
bear on the capability of the
developers of these software test architectures or frameworks. Usability software to maintain its level
The Idiom “like clockwork” refers to: Efficiency of performance under stated
“With machine like regularity and precision; perfectly.” Maintainability conditions for a stated period
of time .” (ISO 9126: 1991,
http://www.thefreedictionary.com/like+clockwork Portability
4.2)
This article will focus on Software reliability can also
Most software test automation or frameworks. As we “Reliability” which has be defined as “the probability
projects rely on some type of design and develop these the following three sub of failure-free operation of
framework or architecture frameworks, we must characteristics: a computer program in a
to operate as efficiently and follow good development specified environment for
effectively as possible, and practices, which ensure that a specified time”. (http://
Maturity
we the test engineers are the following six quality en.wikipedia.org/wiki/
often developers of these characteristics identified in Fault Tolerance Software_quality#Software_
software test architectures the ISO 9126-1 are met: Recoverability reliability)

Reliability is defined as:


“A set of attributes that bear on the capability of
software to maintain its level of performance under stated
conditions for a stated period of time
(ISO 9126: 1991, 4.2)

24 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011


Like Clockwork

When using software we expect that it


will be mature enough to provide the
Evaluating the maturity of an
architecture can be very complex, but Reliability
functionality that we need and robust
enough to continue to function in the
the following formula sums up product
or software maturity quite nicely: Analogy (Part 1)
event of an unexpected occurrence.
Product PC * (PS+PR+PM) This reminds me of an experience
Think of the last time you were using a Maturity
=
3
software application that did not meet I had last week while sitting in
your needs because it suddenly stopped
functioning or even crashed without Where: awkward silence in the meeting
notice resulting in a loss of work. Do • PC = Product Capability room waiting as the presenter
you want to use that software again after
your time and effort was lost? • PS = Product Stability was having trouble getting the
• PR = Product Repeatability demo application started. I heard
This article provides information that
will allow you to answer the question: • PM = Product Maintainability the familiar “BEEP BEEP” breaking
“Is your automated test architecture
The source of this formula is http:// the silence and announcing that
reliable?”
www.stsc.hill.af.mil/crosstalk/1997/08/ it was 2:00 PM and time for the
When discussing the reliability of product.asp.
an automated test architecture in this meeting to start. It was my watch
The repeatability, or in our case,
article I am referring to its ability to “Old Reliable” beeping on the
percentage of reusable components may
perform all requested testing tasks or
be most important to us as automation hour as it has for over 10 years
to shutdown and recover gracefully in
engineers. This refers to the reach
the event a failure is encountered. In keeping me on time for meetings,
and ability of architecture code or
this case, failures are deemed to be test functions to be used in the testing of any
steps, processes or functions that fail to waking me up every morning and
application (global) as compared to how
complete successfully (or as expected). much of the code or how many of the timing my exercise routines, it is
For example, the proper page, field or functions are application specific.
object cannot be found or manipulated always with me and never let me
on a Web or mainframe application. The concept of global code or functions down. Wouldn’t the world be a
Another example is the failure of a can be very advantageous in keeping
query to properly execute on a database architectures that are used by multiple better place if all software worked
or the failure of a Web Service test step people consistent. By having all as reliably as my watch?
to properly execute. automators share a single set of global
functions placed in a mutual location
the chances of each automator creating
Maturity a new function to accomplish a similar
task are greatly reduced.
Maturity is defined as:
Here is a formula to calculate the
“Attributes of software that bear reusability of your automated test
on the frequency of failure by architecture:
faults in the software.” (ISO
9126: 1991, A.2.2.1) % Gen LOC
= * 100%
Reusability Tot LOC
When we talk about a mature automated
testing architecture, we are asking Where:
if the software is ripe or ready for
the “Real World”. We have all seen • Gen LOC = # of lines of code,
pilots, prototypes and even immature functions or keywords that can
architectures that may be able to test a be used in all applications
single application but as you test each • Tot LOC = Total # of lines of
new application, you find the need to code, functions or keywords
continually enhance the architecture’s
functionality for it to work properly. The higher the percentage of global

March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 25


Verify/ATI Con
Venture Beyond

www.ver
26 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011
nference 2011
d the Abstract

Save The Date


September 26 - 28, 2011
Crowne Plaza Hotel
Arlington, VA
The Test Automation Conference You’ve Been Waiting For

rifyati.com
March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 27
reusable code or keywords the more unexpected situation as opposed to
mature your architecture is. waiting for human intervention or
crashing.
Reliability
How can you build maturity into your
automation architecture? Is your automated test architecture fault
Analogy (Part 2)
tolerant?
The first thing to remember is you are Maturity
building software to test software. There are formulas for determining
So, just as with building software in how fault tolerant your automation
the Software Development Lifecycle We are only five minutes late in
architecture is. One such formula is
(SDLC), it is important to have Mean Time Between Failures (MTBF) starting the meeting, but I can live
complete automation requirements and or the average time to the next failure.
a proposed strategy for future versions. with that. I know that it is exactly five
Not everything can be thought of or The formula for MTBF is: minutes late because Old Reliable
built at once so the architecture will
need to evolve. You may begin with # TS
always displays the exact time
a Web testing architecture, but when MTBF =
# FE and date as I check it periodically
planned properly it can evolve to also
test database, mainframe and Web against the atomic clock and
Where:
Service applications. never do any maintenance on it.
• # TS = # of Test Steps
Completed Old Reliable is mature enough to
Fault Tolerance • # FE = # of Failures display the correct date and time
Fault Tolerance is defined as: Encountered for months with 30 or 31 days and
“Attributes of software that The higher the MTBF, the more fault even during leap year.
bear on its ability to maintain a tolerant your architecture is.
specified level of performance The next formula, Mean Time Taken
in cases of software faults or to Repair (MTTR), is used with MTBF
of infringement of its specified to calculate the availability of the
interface.” (ISO 9126: 1991, automation architecture or measures of
A.2.2.2) the impact of failure on an application
Fault tolerance or graceful degradation or project.
is the property that enables a system The formula for MTTR is:
(often computer-based) to continue
operating properly in the event of the
# HR
failure of (or one or more faults within) MTTR =
# TR
some of its components. (Source: http://
en.wikipedia.org/wiki/Fault-tolerance)
Where:
What if you script an automated test to
press a button on a page, but the page • # HR = # of Hours Spent
never comes up or the button is not Repairing
shown? What will your architecture • # TR = Total # of Repairs
do? Does the test stop and wait for you?
Does it go to the next test try the next The formula for Availability is:
command and probably keep failing
until it goes all the way through the % MTBF
= * 100%
Availability
entire test? Does it fail, document the MTTR+MTBF
event in a log and continue on to the
next test? The source of this formula is http://
www.cs.utk.edu/~parker/publications/
Being fault tolerant means that an PERMIS_06.pdf.
automation architecture successfully
continues upon encountering an As the failure/fault count and the

28 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011


Like Clockwork

amount of time used to repair those executed. But, what if one step fails?
failures/faults increases, the availability
of that architecture for actually running Notice column H, the “Recovery”
Reliability
automated tests decreases. This links
directly back to the earlier discussion
column, has a keyword called Exit_
Iteration in some of its rows. This
Analogy (Part 3)
on maturity. A mature architecture will keyword causes the framework to
have a very high availability. automatically exit this test and move Fault Tolerance
to the next one if anything on that row
How can you build fault tolerance into fails to execute properly. This concept
your automated test architecture? can be taken to the next level as shown
As the presenter was attempting
in rows 6, 7 and 8 with a keyword to get the software working, I
Many test automation tools have some called Report_Defect. If a row with
sort of fault tolerance already built in. this keyword in it fails, the test not only
looked down at my watch to find
Such as setting the time you want a test exits and moves to the next test, but also that 10 minutes had past. I began
to wait and retry if it cannot perform automatically creates a defect report in
a certain action. Also some tools the defect database.
to remember all that watch and
will perform an action as long as an I had been through together. It
application page meets some minimal This ensures that even in the event of
threshold of recognition set for the page. failures the full suite of tests will be proved to be water resistant once
executed, instead of stopping while upon a time when our raft flipped
But, what happens if the action awaiting human intervention.
ultimately leads to a failure that results in over on the Shenandoah River,
the subsequent failures of the remainder
and it proved to be shock proof
of the test? Recoverability
upon taking many hits as we rode
The test will continue the long and
Recoverability is defined as: once on the Bizarro roller coaster
painful failure on each step until it gets
to the end of the test. “Attributes of software that bear at the theme park. The only thing
on the capability to re-establish
Recovery scenarios can be utilized in that ever broke on this watch was
its level of performance and
a keyword driven architecture offering
recover the data directly affected the band once when I caught it
a very effective way of implementing
in case of a failure and on the
fault tolerance. on a nail while building the kids
time and effort needed for it.”
Suppose you have a scenario file or (ISO 9126: 1991, A.2.2.3) playhouse. It takes a licking and
design document such as the one keeps on ticking.
The Open Group defines recoverability
illustrated in Figure 1. In this scenario,
as “The ability to restore a system to
the automation architecture performs
a working state after an interruption.”
each task or row and then moves to the
(www.opengroup.org/public/arch/p3/
next.
trm/tx/tx_quals.htm)
The test starts at row 3 by opening the
When we refer to recoverability in
application under test, and then verifies
a software automation architecture,
that Page_1 is available. Next, it enters a
we are referring to the ability to see
user name, and then continues executing
what the architecture was doing or has
each row until all rows have been
done before it failed so an automation

Figure 1: Test_1_Scenario_File.xls

March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 29


Reliability
Analogy (Part 4)
Recoverability
Ten more minutes and the
presenter just announced that
some of his data files were
corrupted so he will need to get
updates from the Internet. Looking
again at my watch I see that we’ve
Figure 2: Failure Screenshot
now lost 20 minutes and the
engineer can pick up the pieces. about building a reliable automation presentation still hasn’t started.
architecture is that you are building My mind once again begins to
In scenario discussed above, suppose software that should be based on test
a test failed, reported the issue and requirements and include a road map to wander and I remember the one
moved on to the next test. That’s nice, maturity. Maturity involves expecting time my watch did let me down. It
but why did the test or architecture the unexpected, being fault tolerant,
fail? Most automation tools come with failing gracefully and having the ability to was when I was comfortably half-
built in reports, but sometimes they effectively recover from a failure. asleep waiting for my wake up
do not give the detail that is necessary
for troubleshooting an issue. We might beep and pondered why it was
learn that Page_1 did not display, but Wait! There’s More! so unusually bright out for 5:00 in
what page did display?
Disclaimer: the morning. Then I looked at my
One other possible way to help recover The views expressed in this article do not
represent the views of the Social Security watch and the screen was blank.
a test may be to have the automation
architecture write detailed logs. Logs Administration or the United States The blank AM screen provided
Government. They are solely the views
such as the one shown in Figure 3 allow indicated that the battery was
of David Fern. There is no expressed
you to know what was happening when
or implied endorsement of views or dead. So I replaced the battery on
the test failed and it saves information
activities of David Fern by either the
that may make it easy to manually Social Security Administration or the my watch, reset the time and it
continue the test if necessary. United States.
recovered in no time.
The most important thing to remember

Figure 3: Sample Test Log

30 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011


Are You Contributing
Content Yet?
The Automated Testing Institute relies heavily on the automated testing
community in order to deliver up-to-date and relevant content. That’s why
we’ve made it even easier for you to contribute content directly to the ATI
Online Reference! Register and let your voice be heard today!

As a registered user you can submit content directly to the site, providing you with
content control and the ability to network with like minded individuals.
Community
Comments Box >> Community Comments Box - This comments box, available on the home page of
the site, provides an opportunity for users to post micro comments in real time.
>> Announcements & Blog Posts - If you have interesting tool announcements, or
you have a concept that you’d like to blog about, submit a post directly to the ATI
Announcements & Online Reference today. At ATI, you have a community of individuals that would love
Blog Posts to hear what you have to say. Your site profile will include a list of your submitted
articles.
>> Automation Events - Do you know about a cool automated testing meetup,
webinar or conference? Let the rest of us know about it by posting it on the ATI site.
Automation Add the date, time and venue so people will know where to go and when to be there.
Events
Learn more today at http//www.about.automatedtestinginstitute.com
AST Comics

AST Comics Art by Torsten Zelger

Negative Testing...Illogical

About The Artist


Torsten Zelger has worked as a tester, test automation engineer, test manager and developer. He draws inspiration for the cartoons from his daily
and past experiences. Find more of his cartoons at www.zelger.org.

32 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011


Training That’s Process Focused Yet Hands On
Software Test Automation Training
www . training . automatedtestinginstitute . com

Public Courses
Software Test Automation Foundations
Automated Test Development & Scripting
Designing an Automated Test Framework
Advanced Automated Test Framework Development

Virtual Courses
Automated Test Development & Scripting
Designing an Automated Test Framework
Advanced Automated Test Framework Development
Test Automation Framework Workshop: A Shortcut to Success

Come
participate in a set
of test automation courses Featured Course
that address both fundamental
and advanced concepts from a
theoretical and hands on perspective.
These courses focus on topics such
as test scripting concepts, automated
framework creation, ROI calculations
and more. In addition, these courses
may be used to prepare for the
TABOK Certification
exam.

March 2011
Public and Virtual Training Available
www.automatedtestinginstitute.com Automated Software Testing Magazine 33
34 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011
March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 35
Verify/ATI Con
Venture Beyond

www.ver
36 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011
nference 2011
d the Abstract

Save The Date


September 26 - 28, 2011
Crowne Plaza Hotel
Arlington, VA
The Test Automation Conference You’ve Been Waiting For

rifyati.com
March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 37
Go On A Retweet

Paying a Visit To T
Microblogging is a form of power that has definitely not been lost
communication based on the concept on the test automation community. Let’s
of blogging (also known as web retreat into the world of microblogs for
logging), that allows subscribers of the a moment and see how automators are
microblogging service to broadcast brief using their 140 characters.
messages to other subscribers of the
service. The main difference between
microblogging and blogging is in the fact
that microblog posts are much shorter,
with most services restricting messages
to about 140 to 200 characters.
Popularized by Twitter, there are
numerous other microblogging services,
including Plurk, Jaiku, Pownce and
Tumblr, and the list goes on-and-on.
Microblogging is a powerful tool for
relaying an assortment of information, a

Twitter Name: jasnow


Post Date/Time: Jan 29
Topic: Kids Coding
Some reasons to say no to the
cloud, from @BobWarfield -
http://monk.ly/gDp5yA

ruby4kids — Alberto Morales is


doing some cool stuff, teaching
kids to code.. http://bit.ly/fBzKqr

Twitter Name: cote


Post Date/Time: Jan 28
Topic: Cloud Computing

38 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011


The Microblogs
Twitter Name: ITPRO
The White House isn’t only us- Post Date/Time: Feb 10
ing #Drupal, it’s contributing to Topic: IT Spending
the #OpenSource community:
http:... http://bit.ly

Worldwide IT spending reached


$1.5 trillion in 2010 http://bit.ly/
idNJcw

Twitter Name: FOSSwiki


Post Date/Time: Feb 18
Topic: Open Source Contributions

Twitter Name: hugs


The Test Automation Body of Post Date/Time: Feb 2
Knowledge (TABOK) Guide- Topic: Complex System Philosophy
book is now available! Free pre-
view at http://www.tabok.auto-
matedtestinginstitute.com/

“A complex system that works is


invariably found to have evolved
from a spreadsheet.”

Twitter Name: automatedtest


Post Date/Time: Feb 1
Topic: Test Automation Body of Knowledge

March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 39


Hot Topics in Automation

Finishing the Internet


Performance Testing Implications of Moving From IPv4 to IPv6

seemed more than adequate, but there has been tremendous


growth in the use of the Internet. Not only have more people
begun using it, but more devices have been introduced to the
Internet such as smartphones, cars, appliances, etc., all that need
their own IP addresses. This explosion in growth as brought us
to a point where the last 5 blocks (16.7 million addresses per
block) of Internet addresses have been distributed as of February
2011 by the Internet Assigned Numbers Authority (IANA) to
the Regional Internet Registries (RIR) for distribution around
the world. So while there is still some time left in which these
addresses will be available for distribution, it is clear that these
addresses and the Internet as we know it is coming to an end (1, 2).

Why is there not widespread panic over this news? Where’s


the looting, rioting and gnashing of teeth? Well, while the average
For those of us that have grown up with it, the Internet has Internet user may not have been aware of or concerned about
seemed to be this indelible force ready to accept whatever we this, it didn’t exactly creep up on us. In the previous paragraph,
could give it. It was this infallible entity whose presence you a question was posed: “Who could’ve ever imagined that this
could count as sure as you could count on the sun rising, the was even possible?” Turns out a lot of people imagined it was
moon glowing and the earth revolving. In recent years, however,
this child-like reverence for the Internet has been severely
challenged. A couple of years ago a chink was put in the Internet’s
armor when word of Michael Jackson’s death nearly brought
large portion of the Internet to a screeching halt. And even more
recently, we’ve seen the Internet being shut down in countries
such as Egypt to quell protests and various legislators are even
trying to institute a “kill switch” for the Internet in the US.

But, while these things may have previously been thought


of as unlikely, none were particularly shocking, or did much
on their own to pierce the view that we had of the Internet as
an unstoppable and ubiquitous phenomenon; perhaps nothing
could. Nothing until the recent news that we’ve finished the
Internet!

Yes, the unimaginable has happened. We’ve finally finished


the Internet. Who could’ve ever imagined that this was even
possible? Well, it turns out that it was not only possible, it was
inevitable. Let’s take a step back for a moment and exam how
it works. The Internet works by transferring small packets of
data across a network using a set of rules or protocol known as
the Internet Protocol (IP). This protocol calls for every system
or device sending or receiving information from the Internet to
have a unique address known as an IP address. In this respect
the Internet works like a virtual postal service that uses the
addresses of the sender and receiver to determine how and where
to send a package. The currently used format for these addresses,
developed in 1970 (1) and used publicly since 1981 (2) is called IP
version 4 (IPv4). This format is a 32-bit format, which basically
means that it allows for 232 – 4,294,967,296 – addresses (2, 3).
Back in the 1970s and 1980s 4 billion addresses probably Figure 1: Internet Growth Trend

40 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011


Seems like everything has been
thought of, so the problem is solved,
right? Yes, but… whenever you introduce
a new level of abstraction to a system,
such as the transition mechanisms being
introduced to the Internet, it typically
doesn’t come without consequences.
One of the potential consequences is
the affect these transition mechanisms
have on performance and performance
testing. Whether or not this affect will
be considered negligible is yet to be
determined, but it is clear that there will
be an affect none-the-less.

Figure 2: IPv4 to IPv6 Transition (9) Some performance studies and testing
have been done on the matter mainly to
assess whether the various transition
possible, and began imagining it in the Generation Transition Working Group
mechanisms have different performance.
late 1980s. And in 1994, the Internet (NGtrans) which proposed several
A study published in the European
Engineering Task Force (IETF), a transition mechanisms to be used when
Journal of Scientific Research evaluated
“large open international community of deploying an IPv6 based service across an
the tunneling transition mechanisms
network designers, operators, vendors, IPv4 Internet, including the dual stack and
with respect to latency, throughput and
and researchers concerned with the tunneling transition mechanisms (7).
CPU utilization and found that the each
evolution of the Internet architecture and
Dual Stack involves a single system mechanism definitely impacted the
the smooth operation of the Internet” (4),
using both an IPv4 and IPv6 Stack that performance that was observed on the
began working on a plan to replace the 32-
operate in parallel. This mechanism can system. It found that the IPv6 to IPv4
bit addressing format with a new 128-bit
be used both in the end system and the Automatic Tunneling definitely performed
format known as IPv6 . IPv6 will allow
(5)
the best out of the three tunneling
for 2 – 340 undecillion or 340
128
mechanisms in the areas of latency,
,282,366,920,938,463,463,374,6
throughput and CPU utilization (9).
07,431,768,211,456 – addresses,
which for now seems like more Another study looking at
than enough for the foreseeable performance of the IPv6 protocol
future. But while IPv6 has already stacks as compared to IPv4
been implemented by some (6), it protocol stacks acknowledged
has yet to be largely embraced that, “It is also clear that when
by the network industry and considering tunneling transition
is expected to have a gradual mechanisms, they will contribute
transition period that will see the Figure 3: Dual Stack Illustration (9) to degraded performance for IPv6,
IPv6 protocol running along side since IPv6 packets have to be
of the IPv4 protocol (5) for a while. encapsulated in IPv4 packets and
Initially, there will be a small number of network node. When used in the end suffer the additional overhead.” (10)
IPv6 systems (IPv6 Island) that will need system, it becomes possible for IPv4 and
to communicate with a large IPv4 Internet IPv6 applications to operate at the same A third research study evaluated
(IPv4 Ocean). Eventually, however, the time. When used in the network node, both various transition mechanisms on various
Internet will be largely an IPv6 network IPv4 and IPv6 packets can be transmitted operating systems (11) and found evidence
(IPv6 Ocean) with a small IPv4 Island. by the system. to support the following three hypotheses
Then ultimately, the Internet will be fully that they began with:
Tunneling, involves encapsulating an
IPv6.
IPv4 address in an IPv6 packet, and is itself • Hypothesis 1: There are
The dilemma with this gradual broken into several distinct mechanisms. performance differences between
transition from IPv4 to IPv6 is that the One source identifies the following three different IP version 4 and IP
two versions are not interoperable, thus tunneling mechanisms (9): version 6 transition mechanisms.
essentially making them two separate
• IPv6 over IPv4 • Hypothesis 2: Different IP
Internets. Therefore when the two need
to communicate with each other some version 4 and IP version 6
• IPv6 to IPv4 Automatic Tunneling
sort of transition mechanism must be transition mechanisms perform
in place. Fortunately, the IETF foresaw • Tunneling Broker differently on various Operating
this as well and formed the IETF Next Systems.

March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 41


References
1. Rashid, Fahmida. “IPv4 Address Exhaustion Not Instant
Cause for Concern with IPv6 in Wings.” eWeek.com. 01
Feb. 2011. http://www.eweek.com/c/a/IT-Infrastructure/
IPv4-Address-Exhaustion-Not-Instant-Cause-for-
Concern-with-IPv6-in-Wings-287643/. 2 Feb. 2011
2. “IPv6.” Wikipedia. 01 Feb. 2011. http://en.wikipedia.
Figure 4: Tunneling Illustration (9) org/wiki/IPv6. 2 Feb. 2011
3. Aftab, Syed. “ IPv4 IPv4 Vs IPv6 (Advantages
• Hypothesis 3: There are factors that cause performance and Disadvantages).” Ezine Articles. http://
differences between IP version 4 and IP version 6 ezinearticles.com/?IPv4-Vs-IPv6-(Advantages-and-
transition mechanisms. Disadvantages)&id=5160096. 2 Feb. 2011
4. “About the IETF.” IETF. http://www.ietf.org/about/. 2
One final study examines the performance of two transition Feb. 2011
mechanisms – IPv6 over IPv4 and IPv6 in IPv4 Tunneling –
compared to the performance of an IPv6 protocol stack. The 5. Marsan, Carolyn. “IPv4 vs IPv6.” Network World.
study found that there were definite performance differences 26 Oct. 2007. http://www.networkworld.com/
between each transition mechanism and the IPv6 protocol stack. news/2007/102607-arguments-ipv4-ipv6.html. 2 Feb.
In certain measures such as CPU utilization, the differences were 2011
stark. 6. Kooser, Amanda. “Following Protocol.” Money &
Business.24 May 2007. http://www.usnews.com/
Conclusion usnews/biztech/articles/070524/24tech.proto.htm?s_
cid=rss:24tech.proto.htm. 2 Feb. 2011
We’ve come close to finishing the Internet, but that’s OK
7. “Basic Transition Mechanisms for IPv6 Hosts and
because we have the equivalent of a new Internet that has been
Routers RFC 4213.” datatracker.ieft.org. Oct. 2005.
set in motion to gradually replace it. The two Internets – the IPv4
http://datatracker.ietf.org/doc/rfc4213/. 2 Feb. 2011
Internet and the IPv6 Internet – are not compatible with each
other, but that’s OK too, because we have transition mechanisms 8. Rashid, Fahmida. “ IPv4 Address Exhaustion Not Instant
to allow the two to communicate with one another until IPv6 Cause for Concern with IPv6 in Wings.” eWeek.com. 01
fully replaces IPv4. So ultimately, life will continue as we know Feb. 2011. http://www.eweek.com/c/a/IT-Infrastructure/
it and the changes to how the Internet works will more than likely IPv4-Address-Exhaustion-Not-Instant-Cause-for-
be transparent to the common user. The people that will probably Concern-with-IPv6-in-Wings-287643/. 2 Feb. 2011
feel the brunt of these changes will be network engineers, system 9. Punithavathani, D.; Sankaranarayanan, K. “IPv4/IPv6
administrators and anyone else responsible for setting up the Transition Mechanisms.” European Journal of Scientific
networks and systems; and QA people need to also be aware that Research. Vol.34 No.1 (2009). http://www.eurojournals.
performance testers responsible for testing that system will also com/ejsr_34_1_12.pdf. 02 Feb. 2011.
be impacted.
10. Bouras, C.; Gkamas, A.; Primpas K. “Performance
During the transition from IPv4 to IPv6, transition Evaluation of an IPv6-capable H323 Application.” 6net.
mechanisms will likely become extremely common. This means http://www.6net.org/publications/papers/grnet-h323-
there is another level in the system that performance testers need performance.pdf. 02 Feb. 2011.
to evaluate for bottlenecks. Also, performance testers will likely 11. Tauch, Sotharith. “Performance evaluation of IP
be asked to help determine which transition mechanisms yield version 4 and IP version 6 transition mechanisms on
the best performance given inputs such as operating system, various systems.” Unitek Research Bank. http://unitec.
processing power, application type, etc. Finally, since IPv6 researchbank.ac.nz/bitstream/handle/10652/1456/
offers many new features not available in IPv4, performance Sotharith%20Tauch%20MComp.pdf?sequence=1. 2
testers may need to evaluate the impact on performance these Feb. 2011
new features may yield once implemented.
12. Raicu, Ioan; Zeadally, Sherali. “Evaluating IPv4 to IPv6
So, for some, the end of the Internet means the beginning of Transition Mechanisms.” CiteSeerX. http://citeseerx.ist.
a lot more work. psu.edu/viewdoc/download?doi=10.1.1.75.8970&rep=r
ep1&type=pdf. 2 Feb. 2011

42 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011


http://www.googleautomation.com
March 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 43
Verify/ATI Conference
2011 The Test Conference You’ve Been Waiting For

Save The Date


September 26 - 28, 2011
Crowne Plaza Hotel
Arlington, VA

Venture Beyond the Abstract


www.verifyati.com
44 Automated Software Testing Magazine www.automatedtestinginstitute.com March 2011