2013 IEEE Seventh International Symposium on Service-Oriented System Engineering

Testing as a Service (TaaS) on Clouds

1

Jerry Gao1,2, Xiaoying Bai2, and Wei-Tek Tsai2,3 San Jose State University, USA, 2Tsinghua University, China, and 3Arizona State University, USA Tadahiro Uehara Fujitsu Laboratories Ltd., Japan Contact email: jerry.gao@sjsu.edu,

ABSTRACT
Cloud computing leads an opportunity in offering testing as a service (TaaS) for SaaS, clouds, and cloud-based applications. This brings new business opportunities, challenges, and demands in innovative service models, testing techniques, QoS standards, and requirements. This paper provides a comprehensive tutorial on testing as a service in a cloud environment. It answers the common questions raised by engineers and managers, and provides clear conceptual discussions about testing as a service (TaaS), including its scope, objectives, motivations and values, distinct features, required techniques, as well as testing environments. It not only presents a classification of different types of testing services in TaaS, but also offers a clear comparative view and perspectives between conventional software testing service and cloud-based testing as a service. In addition, it examines underlying issues, challenges, and emergent needs. Keywords - cloud computing, cloud testing, testing as a service (TaaS), software testing, testing service and delivery.

o o o o

What are the major differences between conventional software testing service and TaaS in a cloud infrastructure? What are the special issues, challenges, and needs? What are the current TaaS practice, tools, and major players? What are the required TaaS infrastructures, frameworks, and tools?

This paper is written to provide a tutorial on Testing as a Service by covering its essential concepts and providing in-depth discussion in different perspectives. The paper introduces and clarifies testing services in a cloud environment, including its scope, objectives, distinct requirements, features, and benefits. In addition, the paper classifies various testing services for SaaS and cloudbased applications, and addresses new issues, challenges, and needs. Moreover, it presents a comparative view between conventional software testing services and TaaS in a cloud infrastructure. Furthermore, the paper also reviews several major players and their products using a comparative view. The rest of this paper is structured as follows. The next section presents our view of testing as a service and related concepts, including TaaS scope, objectives, motivations, and benefits, different testing forms and environments. Section 3 is devoted to the discussion of different types of testing services. It focuses on the new requirements and distinct features, and highlights the major differences between TaaS and conventional software testing services. Section 4 examines the primary issues, challenges, and needs, particularly in TaaS infrastructures. Section 5 reviews the existing research work and emergent technologies.. Finally, the conclusion remarks are given in Section 7.

1. INTRODUCTION
Cloud computing is changing the way of management and delivery of computing systems, networks, and software, but it also brings new business opportunities and challenges in IT industry and software industry. One of them is software testing. A new term, known as testing as a service (TaaS), is becoming a hot subject in different research communities, as well as cloud computing and IT businesses. TaaS in a cloud infrastructures is considered as a new business and service model, in which a TaaS provider undertakes software testing project activities and tasks for a under-test web-based software (or application system) in a cloud infrastructure, and delivers them as a service for customers. Although there are many published papers discussing cloud architectures, technologies, and models, design, and management, cloud testing and TaaS are still new research subjects in both cloud computing community and software engineering community. As more internet-based applications are moved into the cloud world, and more SaaS applications and cloud-based systems developed, there will be a strong testing service demand in cloud environments and infrastructures. This implies that many test engineers and quality assurance managers have to understand the new issues, needs, and challenges in testing as a service on clouds. Although there is a number of published papers addressing TaaS subject [ ][ ][ ][ ], we are lack of a comprehensive tutorial on TaaS in cloud infrastructures, and need more clear understanding and in-depth discussion. Here are some typical questions about TaaS. o What is testing as a service in the cloud world? What are its scope, motivations, requirements, and unique features? o What are the testing forms, environments, and provided diverse services?

2. UNDERSTANDING TESTING AS A SERVICE (TAAS)

2.1. What is Testing as a Service (TaaS)? The term of Testing as a Service (TaaS) was initially introduced and coined by Tieto (http://www.tieto.com/) in Denmark in 2009, and its TaaS solution was nominated by IBM for the Software Innovation Award 2009. Recently, TaaS has received lots of attention in both academic and industry communities due to its advantage in its scalable testing environment, cost reduction, utilitybased service models, and on-demand testing services. In the past few years, there are a number of definitions about TaaS in a cloud infrastructure. Here is one example. Testing-as-a-Service delivers application quality management solutions in a flexible service model that accelerates the implementation of your quality center of excellence. Testing-as-a-Service involves the on-demand test execution of well-defined suites of test material, generally on an outsourced basis. (by Qutesys at http://www.qutesys.com/). Other definitions emphasize different perspectives of TaaS. For example, George Candea et al. in [7] identified three kinds of testing service below.

978-0-7695-4944-6/12 $26.00 2012 IEEE DOI 10.1109/SOSE.2013.66

212

A public certification service, akin to Underwriters Labs, that independently assesses the reliability, safety, and security of software; A "home edition" on-demand testing service for consumers to verify the programs they are about to install on their PC or mobile device; and A "programmer's sidekick" enabling developers to thoroughly and promptly test their code with minimal upfront resource investment.

TaaS customers to meet dynamic business and service needs. 5. Multi-tenant based testing services [20] Since multi-tenancy is a special feature of SaaS applications, TaaS must support multi-tenant based testing services according to the given QoS requirements and service level agreements in functions, performance, and security. Quality certification by third parties Easily leverage scalable cloud system infrastructure to test and evaluate system (SaaS/Cloud/Application) performance and scalability.

As pointed out by Tauhida Parveen and Scott Tilley in [4], TaaS is automated software testing as a cloud-based service. This suggests that the focus of test automation is not only test automation tools and solutions, but automatic test services. In [14], the authors view Testing-as-a-Service as a new model to provide testing capabilities to end users. In [21], we defined testing as a service (TaaS) in a cloud infrastructure as follows. o TaaS provides an on-demand service model for software testing, in which on-demand test requests in software validation processes are processed in a cloud-based scalable testing environment based on pre-defined service-levelagreements (SLAs). o TaaS on clouds offers a new business model for software testing based on utility billing. This allows end users to obtain different types of testing services using the pay-as-you-test model to achieve cost-sharing and cost-reduction. 2.2. Why is TaaS Important? Many IT and software vendors and businesses like to find out why TaaS is important, and where and when they need TaaS. Here is a list of primary reasons. 1. Cost-reduction and resource-sharing in test processes TaaS in a cloud infrastructure enables IT and SaaS vendors to set up a virtual testing laboratory on a private (or public) cloud without purchasing computing resources and testing resources (i.e. test tool licenses). This reduces the front costs and increases resource-sharing and utilization. For a large IT company, the cloud-based TaaS provides a cost-effective way to support many production lines which require diverse computing resources and test tools. Scalable test environments with virtualization [18] Due to the elastic nature of clouds, cloud-based TaaS are able to provide a scalable test environment for both online applications and SaaS applications through automatic provision and de-provision based on virtual and physical computing resources. This feature addresses the strong demand of SaaS scalability measurement and performance/load testing by supporting large-scale test simulation and execution [19]. On-demand automated testing service in 365/7/24 [4][21] TaaS is able to provide on-demand automated testing services to respond diverse online test services from IT and SaaS vendors at anytime and anywhere. Pay as you testing at any time Unlike conventional software testing services, in TaaS, customers and clients are charged for their received services based on a pre-defined utility model and cost metrics. They will be charged in a pay-as-you-test approach. This approach provides more flexibility for

6.

According to [13], IBM reported the experience on cloud testing in small business division, where a flexible and cost-efficient cloud-based development and testing environment is implemented, and cloud testing has demonstrated the following major benefits. Reduce its capital and licensing expenses as much as 50% to 75% using virtualized resources. Reduce operating and labor costs as much as 30% to 50% by automating development and testing resource provisioning and configuration. Shorten its development and testing setup time from weeks to minutes. Improve product quality and reduce the detected defects by as much as 15% to 30%. Help to accelerate cloud computing initiatives with IBMCloudBurst implemented through QuickStart services.

TaaS over Clouds

Testing Services for Applications or SaaS Over Clouds

TaaS on Clouds

Testing Service for Cloud-Based Software (or SaaS) on Clouds

2.

TaaS for Web App.on Clouds

Testing WebBased Applications on Clouds

Figure 1. Different Levels of Testing as a Service (TaaS) 2.3 Testing Environments and Forms Supporting TaaS There are three different forms of testing as service (TaaS) in a cloud environment. Each of them has different focuses and objectives. o TaaS for web-based software on clouds In this form, webbased software is deployed on a cloud, and validated using the provided test services given by TaaS vendors. The major objective here is to take the advantage of large-scale test simulations and elastic computing resources on a cloud. TaaS on clouds In this form, cloud-based applications and SaaS systems are deployed, executed, and validated on a cloud for software validation on a cloud infrastructure or a cloud

3.

4.

213

platform. Unlike the previous form, TaaS on clouds must address and validate the scalability and multi-tenancy of SaaS systems. In addition, a SaaS system usually is developed based on a specific clouds technologies and its services, SaaS connectivity protocols and APIs must be validated too. o TaaS over clouds In this form, SaaS applications crossing hybrid clouds are deployed and validated based on different clouds (such as private clouds, and public clouds). In a hybrid cloud infrastructure, diverse on-demand test services are provided and delivered by a TaaS vendor. A typical SaaS system crossing multiple clouds usually is structured with components, service software, and servers deployed crossing over several clouds. These components and severs are communicated, connected, and interacted following different connectivity protocols.

Figure 2 Different Test Environments [18] As shown in Figure 2, there are three types of cloud-based test environments [18]. A private/public cloud test environment, in which vendors deploy SaaS applications SaaS in a private (or public) cloud to validate their quality. A cloud-based enterprise test environment, in which application vendors deploy web-based applications in a cloud to validate their quality in a cloud infrastrutcure. As shown in Figure 3(b), this cloud-based enterprise test environment is useful to test cloud-based enterprise service programs where A hybrid cloud test environment, in which vendors deploy cloud-based applications on a hybrid cloud infrastructure to check their quality. 2.4. The Scope and Classification of Testing as a Service Figure 3 shows the scope of testing as a service (TaaS), which includes the following eight types of testing services. Scalable test environment service This allows clients to configure and set up their desirable virtual test laboratory environment by requesting and selecting diverse computing resources and licensed tools for software testing. Multi-tenant test modeling and adequacy service This offers test modeling tools and adequate test coverage analysis techniques to support engineers to conduct SaaS testing based

on multi-tenanted SaaS requirements in functions, security, and performance. Digital test management service This supports test project management, test-ware management, and detected problem management. Here, test-ware includes test suites with test cases, test data and scripts, and test results. All these test-wares are managed and grouped based on the types of testing. In addition, they can be uploaded, downloaded, transferred, and migrated upon user requests. On-demand automated test and control service - This offers on-demand automated test execution and control services according to requested testing task schedules for a test project. Test solution integration and composition service This offers and delivers the diverse systematic test solutions based on well-defined test models, methods and algorithms to achieve well-known test criteria. For any selected test solution, TaaS vendors should provide solution adoption, selection, generation, integration, and composition services. A test solution here refers to test modeling tools, test methods and algorithms, test-ware generation and management technology. Test tracking and monitor service This allows test engineers to track and monitor diverse program behaviors at different levels in/on/over clouds during a test process. Large-scale test simulation service - This supports test engineers to set up the desirable a test simulation environment with selected facilitates (such as tools) to fulfill different types of simulation needs. Typical types of simulation needs include: communication traffic and load simulation, SaaS/cloud connectivity simulation, GUI-based user scenario simulation, and domain process-based simulation. TaaS contracting and billing service This provides end users with TaaS service management in accounts, contracts, and billing reports. With this service, TaaS vendors are able to offer customers with selectable testing service contracts based on pre-defined pricing models and billing metrics.
Test Tracking & Monitoring Scalable Test Environment Service Multi-tenant Test Modeling and Adequacy Service

Digital Test Management Service

Testing as a Service (TaaS)

On-Demand Automated Test Service

Large-Scale Test Simulation Service TaaS Contracting and Billing Service

Test Solution Integration & Composition

Figure 3 The Scope of Testing as a Service (TaaS) In short, testing as a service (TaaS) in a cloud infrastructure refers to the delivery of diverse on-demand automated test executions and control services, and test project management services to clients based on well-defined service level agreements (SLAs) and predefined QoS standards. by leveraging cloud technologies and solutions. It has four major objectives. Figure 4 shows a detailed classification of TaaS services.

214

Figure 4 A Classification of Different TaaS Services provided by test solution providers as commodes, and they are shared and utilized by end users with service charges. 2.5. What are New Features in TaaS? Testing SaaS actually is testing SaaS with its databases. Since SaaS programs commonly support multi-tenancy at Unlike conventional web-based software testing, TaaS on a cloud different level, this suggests that any SaaS application must be infrastructure has several unique features and distinct requirements. validated for various tenants with their supported data contexts. In other words, for each tenant, SaaS software must Several published papers have discussed a number of new features be tested with its own meta-database scheme, and its dynamic and requirements in different perspectives. Here are some data store with configured and customized features and data. examples: Testing for multi-tenancy of SaaS applications The multi Cloud-based testing environment [18][20] ], in which diverse tenancy of SaaS applications demands TaaS vendors to offer computing resources and test-wares can be selected, tenant-based validation services to support function testing, configured, and provisioned (or de-provisioned) dynamically. QoS validation, as well as security assurance based on tenant SLA-based software testing [18], in which well-defined QoS based requirements and service-level agreements. service-level agreements are the essential base for testing Intelligent on-demand test automation services On-demand services. testing is one of the major services in TaaS. Although this Utility billing and price models [18][20], in which the preprovides a distinct advantage to support dynamic test defined utility model, billing approach, and cost metrics are operations and executions to deal with large-scale auto-testing used for pay-as-you-test. and simulation, it requires more intelligent control in automatic On-demand testing service [18][20], in which various ontest scenario selection, test script control, and bug reporting demand service requests will be issued and processed in a since auto-testing must cope with testing failures, program testing service SaaS from a TaaS vendor. exceptions and bugs. Large-scale test simulation [18][19][20], in which large-scale Continuous validation and re-testing Supporting online em-level testing in test simulation is required to support syste access of SaaS applications with multi-tenancy in 365/24/7 system scalability, performance, load, reliability, and requires continuous validation and regression test techniques. availability. This suggests that system function validation and QoS testing need to be performed while SaaS operates. The detailed discussions can be found in the related references. m osition service An ideal Test solution integration and comp Here we discussed the other new characteristics in testing as a p allow engineers (or users) to TaaS rovider should be able to service (TaaS). composite and integrate the selected test solutions and tools ng resources are Both computing resources and testin from a TaaS environment to form a desirable test environment. commodes Here the testing resources refer to various wellHence, test solution integration and composition becomes a defined testing techniques, tools, and solutions which are very important service for TaaS.

215

3.

TAAS COMMUNITY, PROCESS, AND ROAD MAP

b) c)

Testing as a service (TaaS) refers to providing static/dynamic ondemand testing services in/on/over clouds for the third-party is at any time and all time (365/7/24). One of the primary objectives is to reduce the IT budget of businesses to focus on their core businesses by outsource software testing tasks to a third party using TaaS service model. TaaS involves the on-demand test execution of welldefined suites of test material, generally on an outsourced basis. The execution can be performed either on client site or remotely from the outsourced providers test lab/facilities. As shown in Figure 5, the work-flow of TaaS includes the following major TaaS service capabilities.

d)

Test solution vendors they provide diverse test tools, test project and process management solutions. Test-ware vendors they use in-house, free-lance, or thirdparty engineers to provide test-ware design and generation service for a contracted test project, including test scripts, test cases, documents. Testing service vendors - they provide, manage, and control TaaS infrastructures, platforms, and deliver testing services based on provided resources, test tools, and test-wares from other vendors.

QoS REQ Manageme TaaS Pricing and Billing TaaS Process Management Tracking & Monitor Service Test Solution Service Test Envir. Service

Vendors may set up three kinds of TaaS clouds. Private TaaS cloud, which provides and deliver diverse dedicated testing services on a private cloud to limited users for private businesses and enterprises, Public TaaS cloud, which provides and delivers various testing services on a public cloud to public users for software testing and quality engineering education, training, and certification purposes. Community TaaS cloud - which provides and delivers various testing services on a community cloud. A typical example will be a research TaaS community cloud supporting for TaaS research, result validation and case studies.
Multi-tenant Test Adequacy Service

On-Demand Test Service

Test Simulation

On-Demand Automated Testing Service

Figure 5 A TaaS Workflow Example [18]

Large-Scale Test Simulation Service

TaaS provider

Testing as a Service (TaaS)

Digital Test Management Service TaaS end-users (Test Engineer)

Scalable Test Environment Service

Test-Ware Provider

TaaS end-users (QoS Engineer) Test Solution TaaS end-users Provider (Test Manager)

Figure 7 A Road Map to Achieve Test as a Service on Clouds A Road Map for Testing as a Service (TaaS) As shown in Figure 7, a road map to achieve TaaS maturity includes the following steps: Step #1: Scalable test environment service The objective of the first step is to support the dynamic creation and deployment of a scalable test environment by leveraging computing resources on a selected cloud infrastructure. Here, TaaS users are offered with selection and configuration services of different computing platforms and resources to allow them to form desirable cloud-based test environments. Both licensed and free test tools and solutions are also provided as selective resources for clients.

Figure 6 User Perspectives of A TaaS Cloud for SaaS As shown in Figure 6, vendors can set a TaaS community as a cloud to deliver various testing services for different types of users. There are three types of TaaS client users. They are: (a) test managers, (b) test engineers or QA engineers, and (c) end users of SaaS. Each TaaS cloud involves three types of vendors. a) Cloud vendors they provide the required networks, such as computing and storage resources, and internet infrastructures.

216

TABLE 1 A Comparison between TaaS and Conventional Software Testing Services

Service Types Primary Objectives Testing Focuses Conventional Software Testing Services Labor intensive testing services to reduce engineering costs Validate software quality according to given requirements System functions and system QoS attributes, such as performances, reliability, availability, vertical scalability, security, and so on. Apply ad-hoc or enterprise-oriented QoS requirements, standards, and specifications Perform ad-hoc test modeling analysis A project-based business model with testing contracts Manual service delivery following a pre-defined contract Manually perform a pre-selected ad-hoc or enterpriseoriented test process with a selected test process tool Offline testing in a test lab. before product delivery Expected TaaS Services from Vendors Solution-based services to reduce resource and testing costs for test projects Assure SaaS quality against expected system function requirements and SLAs SaaS service functions, and special features, such as multi-tenancy, customization, and configurability SaaS QoS attributes, such as scalability and reliability SaaS connectivity, interoperability and portability Modeling QoS requirement with selectable test models with QoS modeling services A service-based transaction model based on a defined cost model and service-level-agreement On-demand dynamic test service delivery in 356/7/24 Select, apply, and perform a well-defined test process offered by a TaaS vendor and supported in a tool Offline testing in a private cloud-based test environment On-demand test execution in a cloud-based virtual test environment Continuous testing for SaaS in/on/over clouds Perform diverse test modeling & analysis using tools Semi-automatic test edition and generation using selective test generation tools based on desirable test models Systematic configure and set-up scalable test environment based on virtual and physical computing resources Dynamic provision and de-provision of computing resources On-demand selection, configuration and deployment of highly shared tools provided by TaaS vendors Semi-automatic test edition, integration, and composition using selective tools All existing testing techniques for software Required innovative continuous testing techniques New testing solutions to deal with SaaS multi-tenancy and elasticity Systematical test-ware service management for test scripts/test suites (test cases/data) using selective tools Test suite migration, download and upload services Selective, customizable and configurable tracking and monitoring services for SaaS, including end-to-end transactions, connectivity, data accesses, and user operations in a cloud infrastructure Perform and apply customizable and configurable largescale data load, traffic load, and user accesses with test simulation services Systematic integration and composition Based on a pre-defined service-level-agreement (SLA) Pay-as-you-test service costs Apply selective and desirable test adequacy criteria and receive the coverage analysis with pre-selected solutions

QoS requirement modeling and specification service Service business Service delivery Test process support Testing Execution

Test modeling Test edition and generation Test environment

Perform ad-hoc test modeling and analysis Manual and semi-automatic test edition and generation using licensed or in-house tools Manually configure and set-up a test laboratory and environment with physical computing resources

Test tool Test composition and integration Testing Techniques

Manual test tool configuration and deployment of limited licenses Ad-hoc test editing, integration, and composition using limited tools Apply selected well-known white-box and black-box testing techniques at the component level (or unit level) and the system level Manual management of test scripts and test suites (test cases/data) following enterprise-based templates using a selected test tool Ad-hoc or enterprise-oriented application-based program tracking and monitoring for application behaviors, messages, and API interactions Generate and perform limited test simulation in a prefixed test environment using selected simulation tools Ad-hoc manual integration and composition Computing hardware, tools, and engineering costs Apply selected test adequacy and measure test coverage analysis using a license (or open source)

Test-ware management Software tracking & monitoring

Test simulation

Solution integration and composition Test service cost Test adequacy analysis and measurement

217

Step #2: Digital test management service The second step focuses on offering essential test management services for clients, including test project management, test process management, and test-ware management solutions and services for TaaS clients. In addition to conventional test management solutions, TaaS vendors must provide test migration on/over clouds, test- ware uploading and downloading on clouds. Step #3: Large-scale test simulation service In this step, TaaS vendors must offer large-scale test simulation, management, and control service for SaaS clients and test engineers to support them in SaaS performance validation, QoS evaluation, and scalability analysis. Step #4: On-demand automated test service The fourth step needs to deal with test automation needs by offering on-demand automatic test execution, test control, test failure processing, and bug reporting. To achieve this, both pre-scheduled and dynamic test requests should be processed timely and responded with quality results. Step #5: Multi-tenanted test adequacy services The final step pays a special attention to deliver automatic test adequacy analysis for each SaaS tenant. With this service, TaaS is able to dynamically monitor and evaluate the quality of its underlying test services for each testing projects. CONVENTIONAL TESTING SERVICE V.S. TAAS

4.

Table 1 provides a comparative view from service perspectives between conventional software testing services and testing as a service on clouds. The distinct differences are summarized as follows. Primary objectives Conventional testing service is labor intensive service. Its primary objective is to reduce engineering costs in a test project. However, the major objective of TaaS in cloud is to reduce and share the cost of computing resources and speed-up the project cycle by leveraging a cloud infrastructure to provide a scalable test environment to support the need in testing SaaS scalability. Testing objectives and focuses - Similar to conventional software testing, software program structures, functions, behaviors, and interfaces, and QoS requirements must be validated. However, SaaS testing has new focuses on SaaS elasticity, multitenancy, connectivity, configurability, customization, and interoperability. Unlike conventional software testing, SaaS testing must be conducted to confirm the posted SLAs from its vendor. Testing environment and execution - Instead of using a fixed testing environment, SaaS testing requires a cloud-based scalable testing environment with provisioned computing, and diverse selectable and configurable test tools. As we known, most test executions in software testing are carried out in an offline mode. However, test executions in TaaS should be performed in both offline and online modes. Testing techniques - Although current testing techniques are applicable to TaaS, engineers require new testing methods to cope with special SaaS

features, such as multi-tenancy, horizontal scalability, customization, and connectivity. In addition, effective SaaS testing requires innovative continuous testing approaches and solutions to support SaaS frequent upgrading. Systematic tool composition and integration In current practice, engineers need to manually integrate various in-house tools with existing licensed tools together to form a desirable system test bed or/and environment. However, in TaaS, users expect TaaS vendors provide some easy integration and composition solutions and tools to support engineering operations for on-demand testing tasks as well as continuous testing operations. Large-scale test simulation on clouds Today, it is common practice to construct some test simulators to support software testing at the system level. Due to SaaS scalability its test simulation must be performed at the large-scale. This implies that a well-defined cloud-based test simulation platform and infrastructure with tools are required. Service delivery and business model Unlike current practice, test services are delivered as a labor-intensive engineering service based on a project contract between a client and a service provider. In TaaS, different on-demand service solutions are provided by vendors to support various engineering activities in SaaS testing. In TaaS, all services are delivered and charged as service transactions based on a pre-defined pay-as-you-test billing model and service-levelagreements (SLAs). Test supporting services Comparing with conventional testing service, TaaS could provide a rich set of testing services to support engineers. These services can be classified into the following classes: Class #1 Test environment services This includes test environment construction and configuration, tool selection, configuration and deployment, test solution composition and integration. Class #2 - Pre-testing services - This includes QoS modeling, test planning, modeling and analysis, and contracting. Class #3 Test management services - This includes test process support, test-ware management, test coverage analysis and measurement. Class #4 Test supporting services This includes test editing and generation, on-demand test execution, program tracking and monitoring, and test simulation.

5.

ISSUES, CHALLENGES AND NEEDS

Due to the new features of SaaS, there are a number of emergent needs in testing as a service (TaaS). In [18][20], we have identified a number of special issues, needs and challenges in cloud testing. Here we highlight some primary issues and challenges, as well as essential needs.

218

shows that different test projects may be supported by a virtual test laboratory and accessed by different user groups, including test and Q&A engineers, as well as test project managers, and users. Comparing with a conventional test laboratory, a virtual test laboratory has several distinct advantages: On-demand establishment and deployment Dynamic provision of computing resources and tools Highly scalable in horizontal and vertical configured Highly resource sharing and tool license sharing However, with our experience in [21], we found a couple of challenges and issues in computing resource allocation and management. Issue #1: What are the cost-effective resource allocations and management strategies that support on-demanded virtual test laboratory construction and deployment in a TaaS infrastructure? Figure 8. A Cloud-Based Infrastructure for SaaS Virtual Test Laboratory 5.1. Challenges and Needs in Building Cloud-Based TaaS clouds There are a number of issues, challenges, and needs in building cloud-based TaaS clouds. This section discusses and summaries some important ones. (a) Issues and needs for scalable test environments The complexity of this issue is caused by dynamic provision requirements for creating and deploying diverse virtual test laboratories with dynamic computing resource needs and tool license demands. Here are some typical factors to be concerned. Both static and dynamic test project schedules must be considered If third-party clouds are used, then the usage cost and business model becomes a critical factor in resource allocation and management Both computing resources and licensed tools need to be managed and allocated Dynamic fulfillment of computing resource needs for each virtual test laboratory Market-driven policy and cost-based approach must be adopted and balanced. This suggests that there is a need for dynamic cost-effective methods for virtual test laboratory construction, deployment and resource allocation and management in TaaS. (b) Issues and needs infrastructure for well-defined TaaS

CP Memor Virtual Machine

EC Instance

System Admin.

Storag

Test Lab. Project Manager

Softwar

Virtual Test Lab.

Virtual Test Lab. User Test Project

Issue #2: What is a well-defined cloud-based TaaS infrastructure for SaaS validation and evaluation? Clearly, there are different ways to design cloud-based TaaS infrastructure. The key requirements and design concerns for a TaaS infrastructure can be listed here. Elasticity and scalability Standard interfaces and connections to test tools and third-party solutions Cloud technology independency and standards Easy connectivity to other clouds Easy SaaS deployment, execution and validation Cost-driven design in TaaS infrastructure, for example, design for energy saving A typical TaaS infrastructure which includes three parts: A SaaS test environment (or SaaS test platform), which is used to support the deployment, execution, and validation of an under-test SaaS system on a selected cloud. In addition, it consists of a set of

Test Tools

Figure 9. Different Users in SaaS Virtual Test Labortary As indicated in [21], there is a strong demand to provide a cloud-based TaaS infrastructure by leveraging cloud technologies to provide resource provision and de-provision for virtual test laboratory in order to support SaaS QoS validation. As shown in Figure 8, a virtual test laboratory can be created or deleted with a dynamic client request to obtain necessary computing resources and tools to form a desirable test laboratory based on a cloud infrastructure. Figure 9
219

supporting test frameworks, such as program tracking framework, and test framework, and so on. TaaS agent layer, which enables large-scale test simulation, tracking, and on-demand test automation. A cloud-based TaaS server which includes the following components. a) Business engine it supports TaaS service contracting, billing, and user account and management. b) Tool manager - it configures and controls a set of tools supporting test and problem management as well as test process management for each test project. c) Simulation engine it supports large-scale test simulation by working with different test simulation agents, such as GUI test simulation agent, and load test simulation agent. d) Test tracking engine it supports and controls different test tracking capability for under-test SaaS program behaviors and performance. e) On-demand test engine it support on-demand test automation, including test selection, scheduling, execution, reporting as well as test failure processing. f) Test repostory manager it supports mult-tenant based test-ware management, migration, and deployment.

How to deal with test failures or detected bugs during large-scale test execution in an automatic manner? During large-scale test execution, any test failure usually causes the stop of automatic test execution because an incorrect operation outcome (or an inappropriate program update) always leads to incorrect program states and invalidated system conditions.
Performance/Load Test Agent GUI Test Simulation Agent

Connectivity Simulation Agent

SaaS

SaaS Test Simulation

Third-Party System

Unit Test Multi-Tenant Test DB

TaaSSimulation Engine

As the advance of cloud testing and TaaS technology, more TaaS infrastructures are needed to support diverse needs from different vendors, such as cloud vendors, SaaS vendors, and TaaS vendors. Test Simulation Needs in a TaaS cloud To support effectively support SaaS testing on TaaS clouds, engineers need several types of test simulations. These include: a) load simulation for interaction traffic messages; b) user access simulation; c) connectivity simulation; and d) third-party system simulation. Figure 10 shows an example of test simulation infrastructure in a TaaS infrastructure. Test Tracking Needs in TaaS Similar to conventional software testing, supporting SaaS validation in TaaS must provide well-defined SaaS program tracking capability to support: a) error tracking, b) system and user interaction tracking, c) system performance and QoS tracking, and d) SaaS connectivity tracking. Unlike conventional program tracking, software tracking in TaaS must address SaaS multi-tenancy by supporting tenantoriented program tracking. Hence, a systematic multi-tenant based program tracking solution should be provided in a TaaS infrastructure. Figure 11 shows an example of software tracking structure in a TaaS infrastructure. On-demand automated testing issues and needs in TaaS To support on-demand automated testing services, TaaS vendors must provide both pre-scheduled and on-demand automatic testing execution services. Since SaaS test automation must deal with large-scale test simulation and test execution, hence engineers must encountered two critical issues:

Figure 10. Test Simulation Infrastructure in TaaS GUI Test Tracking Agent Performance/Load Test Tracking Agent Connectivity Tracking

SaaS

Test Tracking

Third-Party System

TaaS -Tracking Engine Multi-Tenant Tracking Repository Figure 11 A Software Tracking in a TaaS Infrastructure Hence, engineers need intelligent test execution solutions and innovative auto-test infrastructure to support and process any test failures during automatic test execution.

220

This implies that automatic test scripts must be designed and developed to make sure that automatic test scripts are auto re-settable, auto-recoverable, and auto-clearable. In addition, the auto test runner must provide some intelligent test engine which controls and manages large-scale auto-test execution. How to provide automatic test solutions to address SaaS elastic scalability? W.T. Tsai, et al. in [10] points out the importance of scalability testing for SaaS systems on clouds. Supporting test automation for SaaS program, TaaS vendors must provide test automation solutions with elastic scalability. This implies a TaaS infrastructure must be designed to support scalable test automation, control, and reporting. For example, an auto-test engine must be able to deal with one or many SaaS instances at the same time to process a large-scale auto tests. How to manage tenant-based test repository and control tenant-oriented test execution to address SaaS multi-tenancy? Due to SaaS multi-tenancy, test design and management must be done according to tenant-based QoS requirements in functions, behaviors, performance, and security. This demands TaaS vendors to provide tenant-based test repository management service, and support tenant-based test execution and control.

tools, languages and APIs, it is important for engineers to consider how to avoid technology lock-in when building TaaS infrastructures. 5.3. Business and Standard Challenges Lack of connectivity standards Although many existing SaaS systems and cloud technologies have provided well-defined connectivity APIs, there is a lack of well-defined standards in SaaS connectivity. This causes more difficult to build a cloud-based TaaS infrastructure for SaaS to adapt to diverse SaaS connectivity interfaces and interactions. Lack of test tool standards Although there are many existing software test tools, there is a lack of tool connectivity standards in software test automation. This becomes a huddle in supporting tool integration and composition in TaaS infrastructures. Lack of well-defined cost models and billing metrics Although a few TaaS of vendors have used certain simple cost model and billing metrics for their provided testing services, they are very primitive and lack of well-defined billing metrics and theoretic foundation. Hence, more well-defined cost models and rational billing metrics are needed to support TaaS.

7.

RELATED WORK

5.2 Technical Challenges in TaaS #1: Continuous validation and regression testing - The multi-tenancy feature of SaaS brings new challenges and requirements for SaaS testing on clouds. As indicated in [20], engineers need to test the multi-tenancy of SaaS. In [18], we pointed out the challenge of continuous validation and regression testing for cloud-based applications. Whenever software changes for SaaS due to bug-fixing, feature enhancement, and adding new tenants, engineers must re-test SaaS to confirm its quality. However, according to [18] it is still an open question about how to conduct and validate SaaS during software service time using a dynamic approach. #2: Testing Solution Integration and Composition - In a TaaS infrastructure, engineers must deal with the integration of different testing tools and solutions during runtime. Clearly, current software research results did not provide adequate automatic approaches to supporting test solution integration and composition as a service due to the following facts pointed in [18]. There is a lack of well-defined quality standards for SaaS connectivity protocols and interaction interfaces to facilitate SaaS integration and connectivity validation. There is a lack of cost-effective integration and composition service solutions and frameworks to facilitate test tool integration, assembly, and composition in clouds, on clouds, and even over clouds. #3: Technology lock-in This is a typical concern and issue in building a cloud-based TaaS environment and infrastructure for SaaS programs. Since many existing cloud technologies used some enterprise-specific or proprietary
221

Since 2010, researchers began to pay attention to testing as a service in a cloud environment. A number of published papers have addressed this subject in different perspectives. Several papers covered some concepts about testing services. For example, Ashfague Ahmed [1] focuses some basic concepts about software testing as service. In this book, he explains how the normal functions of manufacturing can be applied to commoditize the software testing service to achieve consistent quality across all software projects. Leo Vander Aalst in [8] presents his definition, process, infrastructure and some experience results on software testing as a service. In 2009, Y. Yang at al. [11] discusses that software testing can be conceptualized as a service rather than being viewed as a sequential line of responsibility in software development. In their view, TaaS has two key aspects: (1) a service to developers, and (2) a service to end users. Leah Muthoni Riungu, et al. in [3] reports their qualitative study to show an increasing demand for software testing as an online service on clouds. In this paper, authors present their survey results based on the interviews with managers from eleven organizations. In 2010, G. Candea and his coauthors in [7] discuss three application forms for TaaS. They are: a) a programmers sidekick enabling developers to thoroughly and promptly test their code with minimal upfront resource investment; b) a home edition on-demand testing service for consumers. In 2011, a TaaS workflow process is introduced in [18]. In addition, the paper also reviews and presents the major players in testing as a service and production lines. There are a number of publications discussing TaaS tools and solutions. For example, Yulian Liu at al. in [14][15] presents a TaaS reference architecture based on ontology,

process automation and SOA techniques. The proposed cloud-based TaaS framework consists of 5-layers: a) test service tenant and contributor layer, b) test task management layer, c) testing resource management layer, d) test layer, and testing database layer. Their research results focus on TaaS task management, scheduling, and processing. In addition, Xiaoying Bai, et al. in [20] presents a review on test service tools. The paper provides a comprehensive review about recent research results on testing service and tools. It covers several important issues in TaaS. One of them is how design TaaS infrastructure, test platforms, and environment. Moreover, it also covers some testing service tools on clouds. For example, R. Buyya, et al. in [5] introduces an extensible simulation toolkit (known as CloudSim), which enables modeling and simulation of Cloud computing environments for computing resources in data centers. To address test execution issues in cloud testing, L. Ciortea et al. in [17] presents Cloud9, which is the first symbolic execution engine that scales to large clusters of machines to support automated testing of real software. Its major focus is how to use symbolic execution to support on-demand software testing service. In addition, several published papers addressing performance testing and scalability evaluation on clouds. For instance, W. T. Tsai et al. in [10] presents their insightful discussion on SaaS scalability testing including its importance and issues. Jerry Gao, et al. in [19] discuss SaaS performance testing and scalability on clouds. In this paper, they present a graph-based model to define a set of performance and scalability evaluation metrics for SaaS systems on clouds. Moreover, a systematic cloud-based SaaS performance validation approach is presented based on Amazon EC2 cloud technology with a case study report. In [21], Yong Chen et al. in [16] propose their scalability testing and analysis system (called STAS), in which they introduced a scalability metric (known as isospeed-e) to assist system scalability measure and analysis..

ACKNOWLEDGMENT
This project is sponsored U.S. National Science Foundation project DUE 0942453 and National Science Foundation China (No. 61073003), National Basic Research Program of China (No. 2011CB302505), the Open Fund of the State Key Laboratory of Software Development Environment (No. SKLSDE-2009KF-2-0X), and Research Grant from Fujitsu Laboratory.

REFERENCES
1. 2. Ashfague Ahmed, Software Testing as a Service, Auerbach Publications, 1 edition, September, 2009. T. Vengattaraman, P. Dhavachelvan, R. Baskaran, Model of Cloud Based Application Environment for Software Testing, (IJCSIS) International Journal of Computer Science and Information Security, Vol. 7, No. 3, 2010. Leah Muthoni Riungu, Ossi Taipale, Kari Smolander, "Software Testing as an Online Service: Observations from Practice," Third International Conference on Software Testing, Verification, and Validation Workshops (ICSTW), pp.418-423, 2010. Tauhida Parveen, Scott Tilley, "When to Migrate Software Testing to the Cloud?" Third International Conference on Software Testing, Verification, and Validation Workshops (ICSTW), pp.424-427, 2010. R. Buyya, et al., Modeling and simulation of scalable Cloud computing environments and the CloudSim toolkit: Challenges and opportunities, Proceedings of International Conference on the High Performance Computing & Simulation (HPCS 09). Liu Gu and Shing-Chi Chenug, Constructing and testing privacy-aware services in a cloud computing environment: challenges and opportunities Proceedings of the First Asia-Pacific Symposium on Internetware, Beijing, China, 2009. George Candea, Stefan Bucur, and Zamfir Cristian, Automated Software Testing as a Service (TaaS), 2010. Proceedings of the 1st ACM Symposium on Cloud Computing, 2010. Leo.van der Aalst, Software testing as a service (STaaS), Internet: http://www.tmap.net/Images/Paper%20STaaS_tcm847910.pdf, [May 06, 2009]. G. Goth, Googling test practices? Web giants culture encourages process improvement, IEEE Software, 25, 2, pp. 92-94, 2008. W.T. Tsai, Yu Hang, and Qihong Shao, Testing the Scalability of SaaS Applications, Proceedings of IEEE International Conference on Service-Oriented Computing and Applications (SOCA), 2011. Y. Yang, C. Onita, J. Dhaliwal, X. Zhang, TESTQUAL: conceptualizing software testing as a service, Proc. 15th Americas conf. on information systems, 6-9.08, San Francisco, California, USA, paper 608, 2009. N. Yigitbasi, et al., C-Meter: A framework for Performance Analysis for Computing clouds , Proceedings of 9th IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGRID '09). IBM Smart Business Development and Test Cloud, IBM Global Technology Services, 2010. URL: http://www935.ibm.com/services/us/index.wss/offerin g/midware/a1030965 Yulian Liu, Testing as a Service over Cloud, Proceedings of the Fifth IEEE International

3.

4.

5.

6.

7.

8.

8.

CONCLOUSION AND FUTURE WORK

9. 10.

Testing as a service (TaaS) is becoming a hot research topic in both cloud computing and software engineering research communities. As the advance of cloud technology and testing as services, more research results are needed to address the open issues and challenges on TaaS infrastructures, techniques, and automation solutions. More innovative testing techniques and solutions, and QoS standards are needed to support on-demand testing services in a scalable cloud infrastructure, for example, SaaS testing adequacy and standards for multi-tenancy. This paper provides a comprehensive review and in-depth tutorial on cloud-based TaaS for SaaS applications. It offers essential tutorial concepts on TaaS definitions, scope, motivations and benefits, as well as classified testing services and test environments. In addition, it discusses in details about TaaS requirements, distinct features, issues, challenges, and needs. Moreover, it highlights the major differences between conventional software testing and cloud-based TaaS.
222

11.

12.

13.

14.

15.

16.

17. 18.

19.

20.

Symposium on Service Oriented System Engineering, 2010. Lian Yu, Shuang Su, Jing Zhao, et al, Performing Unit Testing Based on Testing as a Service (TaaS) Approach, Proceedings of International Conference on Service Science (ICSS) 2008. Chen Y, Sun X. STAS: A Scalability Testing and Analysis System. 2006 IEEE International Conference on Cluster Computing. 2006:1-10. Available at:http://ieeexplore.ieee.org/ Liviu Ciortea, et al, Cloud9: a software testing service, ACM SIGOPS Operating Systems Review, Volume 43 Issue 4, January, 2010. Jerry Gao, Xiaoying Bai, W. T. Tsai, Cloud-Testing Issues, Challenges, Needs and Practice, published by Software Engineering: An International Journal (SEIJ), Vol. 1, September, 2011. Jerry Gao, Xiaoying Bai and W. T. Tsai, SaaS Performance and Scalability Evaluation in Clouds, Proceedings of the Sixth IEEE International Symposium on Service-Oriented System Engineering (SOSE2011), Irvine, California, USA, 2011. Xiaoying Bai, Muyang Li, Bin Chen, Jerry Gao and Wei-Tek Tsai, A Survey of Cloud Testing Tools , Proceedings of The Sixth IEEE International Symposium on Service-Oriented System Engineering (SOSE2011), Irvine, California, USA, 2011.

223