Software Testing

Assignment No. 3
Contents
Summary 2
1. Introduction 2
2. DOM-Based & Visual Web Testing 3
2.1. Web testing tools 3
2.2. The Page Object and Page Factory patterns 3
2.3. DOM-based Web testing 3
2.4. Visual Web testing 3
3. Challenging aspects of the Transformation from DOM to Visual 3
3.1. Generating visual locators for repeated Web elements 4
3.2. Web elements changing their state 4
3.3. Web elements with complex visual interaction 4
4. PESTO: The Page Object Transformation TOOL 4
4.1. Visual locators generator (module 1) 4
4.2. Test suite transformer (module 2) 5
5. PESTO Improvement Process 5
5.1. Considered Web applications 5
5.2. Training test suite development 5
5.3. PESTO Improvement 5
5.4. Summary of the results on the training test suites 5
6. PESTO Evaluation 6
6.1. Web Applications 7
6.2. Test suite development 7
6.3. Experimental procedure 7
6.4. Experimental results 7
6.5. Discussion 7
6.6. Generalizability of the results 8
7. Related Work 8
8. Conclusion and Future Work 8
Strengths 9
Weaknesses 9
PESTO: Automated migration of
DOM-based Web Tests towards the Visual
Approach
Summary

1. Introduction
Testing web applications is a continually evolving task due to fast pace development of web
applications. This is the reason E2E testing systems are widely popular which require scripts that
can auto-fill the data fields to test the assertions. E2E have undergone multiple generations over
the years.

First Generation: Test cases were produced on the basis of screen coordinates. It required
locating web elements by coordinates and is thus fragile.

Second Generation: Test cases are produced on the basis of DOM properties. It’s easier and
better approach than first generation testing approaches.

Third Generation: It is most popular in contemporary era due to its ability of handling complex
applications like Google Docs & Google Maps. In such cases, DOM-based tools do not suffice
because the DOM of the application may not include all user interface controls.

Nowadays, a combination of DOM and Visual based approach is popular. But it is unclear which
approach which overcome the other in the future because migration of test cases is a
cumbersome task.

The research paper deals with migration of Old test cases to New ones automatically which
proves to be complicated and complicated and error prone task.

PESTO, a component of NEONATE, is a testing application which can migrate Selenium

WebDriver tests (second generation) into Sikuli API (third generation) seamlessly. The paper
makes following contributions:

● Generate visual locators of web elements corresponding to DOM-based locators.

● Cater elements that have multiple appearances; elements with changing properties and
complex elements like drop-down lists.
● PESTO can be used for generating different visual test suites, tailored for different
platform-browser combinations.
 ● It is highly accurate and is open source.

2. DOM-Based & Visual Web Testing

E2E testing is a kind of black box testing that can generate multiple test cases of the same
element on the basis of underlying technology and testing tool.

2.1. Web testing tools

DOM-Based
These are second generation testing tools. Widely used example is Selenium. It relies on locating
web elements on the basis of their DOM properties like id, name etc. XPaths can be used too, if
DOM attributes cannot be used.

Visual
Such tools rely on image processing techniques to generate test cases of web elements.

2.2. The Page Object and Page Factory patterns

It is responsible for improving maintainability and reduction of duplication of code. It follows
the principle of Separation of Concerns for test must be decoupled from the implementation.
Usually page objects are initialized by a Page Factory which checks mapping and initialization of
web elements.

2.3. DOM-based Web testing

For this section, Selenium WebDriver is used which is a popular tool for testing web
applications. It locates web elements on the basis of their (1) Attributes (2) Tag Names (3) HREF
text (4) CSS identifiers and (5) XPath expressions. The WebDriver also provides various
methods to verify presence of authenticated username on the web page.

2.4. Visual Web testing

Sikuli API was used for this purpose which is a third-generation tool. It is a visual tool able to
automate and test graphical user interfaces using screenshot images. Sikuli allows testers to write
scripts based on images that define the GUI widgets to interact with and the assertions to be
checked. Some useful WebDriver methods are not natively available in Sikuli (eg, click() and
sendKeys()).

DOM–based tools allow users to specify fine-grained assertions while Visual tools check that an
image representing the visual appearance of an element is present in the web page as rendered at
runtime by the browser. DOM based approach can locate elements regardless of the position,
while Visual tools require positions of the elements.
 3. Challenging aspects of the Transformation from DOM to Visual
This process requires solution to various problems like generating test for repeated elements,
localizing the elements and managing complex elements.

3.1. Generating visual locators for repeated Web elements

When a web page contains multiple instances of the same kind of Web element (e.g., an input
box), creating a visual locator is more complex than creating a DOM-based locator. Elements
with same visuals can are tricky for this approach. This can be solved by (1) Expanding the
visual locator to span more elements (2) Another stable Web element is identified in the page,
close enough to the Web element of interest, to be used as an “anchor.

3.2. Web elements changing their state

Different localization approaches (eg, DOM vs visual) may require a different number of
locators to be created on the basis of their state.

3.3. Web elements with complex visual interaction

Web apps have complex elements like drop down list. A DOM-based tool, such as Selenium
WebDriver, provides commands to select directly a list item from the drop-down list. This is way
more complex in Visual approach since it deals with images so in this case visual approach
replicates the same steps that a human tester would do.

4. PESTO: The Page Object Transformation TOOL

Pesto aims to convert Selenium test suits into Visual web test suite as in Sikuli. A general
architectural procedure of PESTO is

Input: DOM-based Test Input

PESTO: Visual Locator Generator (Generates a set of visual locators for each WebElement
located by DOM-based locator). This maps DOM based to Visual Locator. Thus DOM based
suite is transformed into Visual Test Suite.

Output: Visual Test Suite

4.1. Visual locators generator (module 1)

To capture the images of the target Web elements, as rendered by the browser at run time, we
need to intercept the commands executed by the test cases. PESTO is able to intercept calls to the
WebDriver commands executed by the test cases and create the corresponding visual locators.
The process is as followed:

1. Invoke the WebDriver method getScreenshotAs(…), to create a screenshot of the entire

web page.
2. Get location and Size of web element by WebDriver methods.
 3. Invoke the visual locator creator, which generates a visual locator for the Web element of
interest.
4. Invoke the visual locator optimizer, which checks for duplicate locators associated with
the same Web element.
5. Save the visual locator, and keep the mapping between DOM-based locator and visual
locator.

For each Web element, DOM to Visual Locator Mapping consists of following triple:

(PO Name, Locator Type = Locator Value, Image Path)

4.1.1. Visual locator creator

Multiple matches would be found for the perfectly cropped image representing the Web element.
For this reason, in such cases, the visual locator creator expands the size of the rectangle image
until a unique locator is found. More specifically, the algorithm tries to expand the image by the
same amount of pixels in both directions until the match is unique.

4.1.2. Visual locator optimizer

The visual locator creator generates a correct visual locator but as we know during the execution
of a test suite, the same Web element could occur multiple times. PESTO analyses each created
visual locator and compares it with the previously created ones for the target Web element in
order to recognize and remove the clones (eg, multiple locators for the same Username field).

4.2. Test suite transformer (module 2)

Module 2 uses the information provided by module 1 to transform the DOM-based test suite into
a visual test suite. The Page Objects Transformation module transforms the DOM-based page
objects into new visual page objects.

5. PESTO Improvement Process

This phase tests PESTO against training and test sets. Different web applications were used to
train and test PESTO.

5.1. Considered Web applications

Following web applications considered for training purposes: PHP Password Manager (PPMA),
Claroline, PHP Address Book and Meeting Room Booking System (MRBS).

5.2. Training test suite development

The training test suites exhibit the following characteristics: (1) the test cases adopt the Page
Object and Page Factory patterns, (2) the test cases interact with a relevant portion of the Web
elements rendered on the web pages (3) the Web elements with multiple visual rendering are
stimulated in multiple states and (4) the test cases contain assertions on visible Web elements.
 5.3. PESTO Improvement
PESTO was able to migrate only a subset of the used WebDriver commands. So hybrid test cases
were used (both DOM based and Visual based test suits). Such hybrid test suites did not run
correctly, since the Web elements with different visual states were not managed properly. So
each time an error occurred, it was to be handled, improving PESTO.

5.4. Summary of the results on the training test suites

Following table 1 summarizes the training set at the end of improvement process

Table 2 provides a summary of WebDriver commands used in testing

Total 189 calls of 942 turned out to be fairly common. Overall, PESTO was able to manage
almost all the command calls used in the four training test suites. In remaining, it copied
commands from original test suits and thus used a hybrid test suit. The resulting test suites could
be compiled and executed without requiring any manual intervention. The migrated test suites
contain a total of about 99% of visual command calls and just about 1% untransformed
DOM-based command calls. When ran without intervention, we had the same results obtained
with the original DOM-based training test suites.
 6. PESTO Evaluation
To evaluate our tool without introducing any author bias, we recruited a professional tester
(referred from now on as ​Tester)​ . Following testing questions were answered in this phase:

1. Automation​: What’s the required amount of intervention that goes into Module 1 & 2?
2. Correctness: ​How many migrated visual test suites have issues when locating,
interacting with, and asserting on the Web elements under test?
3. Multistate​: How often are multiple visual locators, associated with a single Web
element, required?
4. Applicability: ​What percentage of the original DOM-based command calls is left
un-transformed by PESTO?

6.1. Web Applications

Tester tested two open source applications without knowledge of PESTO. Both applications,
regardless of their size in LOC, include quite complex web pages.

6.2. Test suite development

Systematic approach was used to develop test suit including required assertions.

6.3. Experimental procedure

Following experimental procedure was adopted

1. DOM-based test suite development:​ The tester developed a DOM-based test suite
2. DOM-based test suite transformation: PESTO was used for this. Problems were noted
and modified.
3. Visual test suite evaluation: The newly generated visual test suite was executed to
check for its correctness.

6.4. Experimental results

Questions asked at the start of this section are answered in the results

Automation: ​To answer this, we need to consider (1) What were the issues faced by PESTO’s
Visual Locator Generator. Elements at odd places like close to border were problematic as visual
locator couldn’t be created for them. Luckily, they were few and were tested at the end of test
suit. (2) Compilation and execution issues faced by Module 2. This phase required only minor
manual refactoring to transform its syntax into one supported by PESTO.

Correctness: ​Apart from a minor problems due to Sikuli’s search for web elements during page
transition, every test case worked as expected.

Multistate: ​For 192 Web elements, 449 visual locators were used. 51% of such locators
corresponded to single state elements, 20% dealt with double state and 29% were for triple state
elements. Thus, the more test cases are produced, the higher the probability that a test case may
interact with a given Web element.

Applicability: ​Many such commands were expected however, no such commands were used by
tester.

6.5. Discussion
Advantages: ​PESTO reduces time by at-least 50% as it generates visual locators automatically
and removes duplicated locators.

Machine Time required for Migration: ​Keeping in view the hardware specs, PESTO’s average
time for evaluating test suits is almost similar to typical process.

Execution time of the visual test cases created by PESTO: ​This time hinges on underlying
image processing algorithms & loading time of web pages. PESTO caters to the computational
time overhead by removing duplicates, so processing is quicker as fewer elements are there to
care for. PESTO also introduces small delays to cater loading problem.

Visual locators readability: ​The visual locators generator algorithm of PESTO has been
designed to mimic the visual selection strategy of a human tester and is thus easier to read.

WebDriver commands used in practice: ​We can conclude that the potential PESTO's coverage
of theWebDriver commands used in practice is high so the tool is easy to learn.

PESTO's limitations: ​Few commands are not covered by PESTO like accept() and
selectByValue(). The elements at the borders are difficult to be covered by PESTO. This
however can be solved in future.

Overall, advantages exceed limitations.

6.6. Generalizability of the results

Due to reasons described above, it is fair to say that the results produced here are general and
will be reproduced in industrial settings.

7. Related Work
The only work that faced similar challenges with respect to PESTO is the one by Alégroth et al.
They migrated existing automated component-based GUI test cases (tool GUITAR [57]) to the
visual approach (tool VGT GUITAR). Since PESTO is a tool for migrating the test code, while
keeping it functionally the same, it can be regarded as a refactoring tool.

Van Deursen et al explained bad smells of the code. In another work, the same authors study
interplay between testing and refactoring. Chu et al. proposed a plugin tool to guide test case
refactoring after having applied well-established pattern-based code refactoring. PESTO differs
from these works since it aims at refactoring the test code to make it executable by visual Web
testing tools, without modifying the application code.

8. Conclusion and Future Work

In the paper, we proposed and experimented a tool called PESTO which allows developers to
experiment with third-generation testing tools at a lower cost compared to a manual migration.
We tested it on two different open-source Web applications. PESTO generated the corresponding
visual test suites for the DOM-based validation test suites requiring only a small amount of
manual work. Moreover, PESTO can handle more than 95% of the used command calls making
the transition easier and lowering the learning curve.

In future, we intend to evaluate PESTO on some industrial case studies. Moreover, we need to
investigate different visual locator generation strategies and to study how they impact the locator
comprehensibility. Currently, PESTO does not generate a visual test suite portable across
different platform-browser combinations. A possible extension of PESTO could take into
account the possibility of merging two or more visual test suites generated by PESTO in order to
obtain a single, platform-browser portable visual test suite. Finally, we plan to investigate
whether starting from PESTO, it would be possible to derive a more general and tool
independent transformational framework.

Strengths
1. Procedure of transformation from DOM to 3​rd​ generation is well explained.
2. Logic behind transformation is clearly portrayed.
3. Tool experimentation results are well cited.
4. Aids a 2​nd​ generation tester to understand a 3​rd​ generation tests.
5. Sample codes are provided to add in to application of the tool in industry.
6. References are well cited.
7. Need of the tool is well explained.
8. Advantages are well explained.
9. Related work is well explained
10. Refers to the latest applications.

Weaknesses
1. Explicit implementation of function like send keys
2. Require more locators
3. More complex than DOM
4. Sometimes may not create a visual locator.
5. May require modification of test suite to match pesto syntax.
6. Test suit take more time to execute than DOM based.
7. Some test suite may not adapt to page object and page factory design patterns.
8. Can’t generate visual locator for accept().
9. Can’t migrate selectbyvalue().
10. Visual locator may not exist if target object is repeated on web page.
11. Only understandable by tester specifically
12. Requires alot of domain logic.