Cloud-based Testing: Opportunities and Challenges

Abstract

Recent years, we are encountering a blast of distributed computing as another age of computing resources, software services and platforms are relocating to cloud, which likewise drives an extraordinary open door in giving progressively powerful and adaptable testing strategies and apparatuses just as testing services (TaaS).

Be that as it may, testing has its own attributes contrasting from different pieces of programming designing, which requires new diagram and systems for offloading. This paper offers a reasonable review of ideas, highlights, and difficulties in cloud testing.

Moreover, this paper condenses and surveys distinctive items and arrangements of cloud testing and testing condition as a services (TEaaS), and gives some conceivable research bearings of cloud-based testing

Introduction

Recently, cloud computing has been one of the freshest topics in IT enterprise when you consider that its miles converting the way of supplying Internet services and computation assets. Today, we're experiencing an explosion of cloud computing as a new generation of Internet carrier. More and extra organizations, together with a few leading ones along with Google, IBM and Amazon, are imparting service or infrastructure based totally on cloud techniques.

Get quality help now

Doctor Jennifer

Verified writer

Proficient in: Engineering

5 (893)

“ Thank you so much for accepting my assignment the night before it was due. I look forward to working with you moving forward ”

+84 relevant experts are online

Hire writer

Software testing, as one of the maximum vital however exertions- in depth parts of software improvement, can also benefit from cloud computing. Since cloud computing gives on-call for assets and services with high elasticity and compatibility, studies specializing in cloud testing, or every now and then referred to as as Testing as a Service (TaaS), gets more and more attention lately. Cloud testing is a new generation of software program trying out system.

The software program assessments in this system are executed and analyses inside the cloud-based environment.

One of the primary advantages of cloud testing is its fee- performance in coping with and retaining testing environment. Users can make bigger their checking out resources on call for. For instance, users do now not need to buy a couple of gadgets for huge- scale reliability exams. They can run their exams in the cloud with gadgets supplied with the aid of providers. In addition, it's miles simpler to simulate extraordinary hardware failures in a cloud environment. However, testing within the cloud isn't always free.

For service companies, testing within the cloud is done in a web manner in which virtualization and dynamical configuration of trying out assets are critical, however difficult to enforce. For the customers, properly and efficaciously migrating check cases or scripts to the cloud additionally calls for unique competencies. Cloud testing brings demanding situations with opportunities and blessings collectively. Although some organizations and companies have already provided cloud-based testing services such as cloud load testing and web application testing, cloud testing or TaaS is still a new topic for software engineering.

Understanding Cloud Testing

What is Cloud Testing?

Cloud testing refers back to the techniques of executing and handling software testing on a cloud-based totally environment [5]. There are basic kinds of cloud-primarily based software program trying out: testing in a cloud, and testing to a cloud [1]. The predominant distinction between them is whether or not the checking out makes a specialty of validation of the high-quality of a SaaS application inside the cloud. Testing in a cloud highlights getting take a look at resources from cloud to check special software program software no matter whether it's miles a service primarily based on the cloud, at the same time as testing toward a cloud specially runs software testing to assure the exceptional of a SaaS software inside the cloud.

Unlike conventional software testing, cloud testing has some unique testing features and requirements:

1. On-demand Service: This is one of the crucial features and requirements for maximum of the cloud computing strategies. Cloud testing affords consumers with more flexibility of the usage of computing aid. Users can select exactly what and what kind of they need [1], and that they handiest need to pay for what they use on every occasion.
2. Cloud-based Testing Environment: All the computing resources and check beds in addition to testing structures are furnished and allotted robotically primarily based on cloud infrastructure. The test-cases are executed within the cloud-based totally environment. The companies of cloud testing want to control the testing aid by means of virtualization.
3. Service Level Agreements (SLAs): The cloud testing offerings are supplied to unique customers with various however nicely- described provider-stage agreements [5]. These agreements are always designed by using exclusive companies and considered as a thing of the testing provider, such as information confidentiality, machine reliability and user privacy.

Multi-tenant: The cloud testing service is usually seasoned- vided to a couple of clients [7]. Different from traditional testing systems, the cloud provider is designed for sharing by using many one of a kind user.

Benefits

There are some wonderful blessings while cloud-based testing is used. First of all, the cloud-based testing can reduce expenses. Customers of cloud testing use virtualized computing resources within the cloud and percentage testing infrastructure on demand without buying required hardware and software. In addition, vendors of cloud testing provide testing systems (networks, operating structures and hardware) for exclusive take a look at obligations. Users can without difficulty execute checks without paying a great deal interest on configuration and installation of check beds. The test environment can be set up quickly.

In addition to the value efficiency, cloud testing can provide better load scalability and testing overall performance. It is easy for cloud testing machine to generate and bring scalable check load using parallel and concurrent computing as required to check and compare the performance of large structures. Improving overall performance by the usage of concurrent computing is likewise an essential feature of cloud-based testing strategies..

Furthermore, it may be less complicated to installation the cloud testing platforms for very huge distributed systems. The cloud testing structures offer interfaces for routinely environment construction, which enables customers to set up and rebuild large- scale check structures effectively. By the usage of techniques of virtualization, the configuration time for putting take a look at beds for massive structures may also be shorten.

When to Migrate Test to Cloud?

Before beginning to apply cloud testing, we first don't want to forget while it's miles more appropriate to use, or whether or not cloud testing is suitable for the venture. Though the advantages of cloud testing are vast, it isn't appropriate for all of the conditions. Users need to examine the advantages and trade-offs among a test environment with a cloud system before testing.

The first factor customers need to don't forget is the testing efficiency. Generally, speaking, an internal testing is smaller and smooth to accumulate, and there may be no extra value for test uploading and control. This testing surroundings is very appropriate for checking out a few small programs. In comparison, a cloud-based totally testing device may have higher performance for testing huge dispensed system. It is likewise more suitable to simulate complex user load of the real world..

The test case independence is also an critical thing for selecting test environment. Cloud testing may also shorten the executing time of a huge range of test cases by way of concurrent testing. However, this is possible most effective if the check cases or check tasks uploaded to the cloud are unbiased from every other. It is more appropriate to apply cloud testing for exceptionally independent test instances [6].

In addition to the independence of test cases, the environment requirement impacts the testing execution. Typically, the cloud environment is hardware-standardized. The cloud testing companies constantly offer the common hardware resources and software surroundings to the customers. Applications requiring special devices or particular hardware architectures may be now not appropriate for cloud testing. Complex test cases which depend upon a few unique tools and libraries also are no longer desirable candidates for TaaS. However, a few Web software checks are clean and suitable to migrate to the cloud nowadays because of their low environmental dependency on particular hardware or software program.

Moreover, users need pay attention to privacy and safety. By now, privacy and safety are still open questions for cloud- based techniques, and customers have to consider these ques- tions and their risks before testing.

Practical Solutions of Cloud Testing System

In this section, we introduce some typical structures and schemes for implementing cloud testing. First, we demonstrate the basic workflow of cloud testing.

Workflow

Sometimes cloud testing is also called as testing as a service (TaaS), which receives much attention recently because of its cost-efficiency and load scalability. The concept of TaaS was introduced by Tieto in Denmark in 2009 [5]. A typical workflow of TaaS is displayed in Figure 1.

The typical workflow of TaaS consists of the following steps and service modules:

1. TaaS Process Management: This is the main logical controller of TaaS systems. It provides process control and the management of test tasks.
2. QoS Requirement Management: This thing manages the predefinition of software testing QoS requirements. There are modules which might be used to guarantee testing at satisfactory on this thing.
3. Test Environment Service: This module is used for creating virtual testing environment by way of dynamically allocating cloud computing resources on call for. This part may be very critical for cloud-based testing techniques, which impacts the performance of the whole testing system.
4. Test Solution Service: TaaS system can generate test cases and suites routinely via the usage of this module. In addition, it's miles accountable for scheduling the test cases for the machine.
5. Test Simulation Service: The module simulates testing inputs in the virtual system and use these simulations to reflect varieties of real conditions. For example, specific faults can be generated in the cloud testing system to simulate real fault scenarios. This component can not only simulate specific conditions, but also establish necessary environment with user data.
6. On-demand Test Service: By using this module, the system executes test cases for the customers on demand as the requirements and schedules.
7. Other Components: There can be some assistant mod- ules in a TaaS system. For example, tracking and monitor module allows system to record the behaviors and results of the executed tests, and the pricing and billing module enables the customers to pay based on their testing workload.

D-Cloud

D-Cloud is developed by ToshihiroHanawaetal. [3] mainly for testing parallel and distributed systems at first. It can be considered as a typical cloud testing system which accelerates software testing by executing test cases simulta- neously using virtual computing system. D-Cloud consists of three different parts: virtual machine nodes providing comput- ing resource, resource controller which manages and allocates testing resource for different tasks, and testing configuration controller, which interprets test scenarios to detailed hardware and software requirements. The test configuration of D-Cloud is written in a specific format of XML file to describe the test scenario and resource requirement, and the test cases can be provided in separated files written in scripting language. The structure of D-Cloud is shown in Figure 2.

D-Cloud constructs its virtual computing system through the usage of QEMU (brief for Quick EMUlator), which is an open-supply software supplying hardware virtualization. The on-demand software testing is executed inside the QEMU virtual machines with precise emulated hardware and distinctive running systems. In addition, D-Cloud can emulate hardware fault with the aid of the use of QEMU.

In order to allocate testing resources dynamically and efficiently for performing a number of test cases simultaneously, D-Cloud uses Eucalyptus [8] to manage virtual machine resources flexibly. The system can transfer and allocate various operating system images to different virtual machine on demand as well as initial and close these virtual machines. The resource allocation is transparent for the test customers.

Before allocating testing assets dynamically, D-Cloud gadget first translates person-described testing configuration report to specific hardware and software requirements, and then uses these requirements for aid management. The configuration is written in XML and can be used to generate diverse resource descriptions. Table I lists a few gadget definition which may be considered as a few primary hardware configuration of the checking out platform. Table II is the descriptions of software program surroundings in D-Cloud machine.

The basic workflow of D-Cloud is similar to what we describe in component A of section IV. The device first gets the test configuration and take a look at scripts from the customers, and parses the configuration record to allocate trying out resources and set up testing environments. Then the test instances are done simultaneously. In addition, D-Cloud can simulate some sorts of gadget faults by the usage of virtual machines.

Cloud Testing System with Task Clustering

As noted before, cloud testing system is cost-effective for the clients because the clients can use digital assets and testing environment as asked without sincerely buying them. However, getting ready separated however similar testing resources for several customers in a brief time is still highly-priced for the providers. If the testing environment can be reused by using several customers who have comparable requests, the machine may be greater cost-efficient. LianYuetal. [2] expand an optimized cloud TaaS machine with assignment clustering. The architecture of this gadget is shown in Figure 3. It includes 5 layers: check provider frontend, check project supervisor, checking out useful resource manager, trying out layer and testing database.

1. Test Service Frontend: This is the interactive layer between users and the TaaS system where users can use clustered. This manager consists of several major components such as the module of checking test task capability and the module of clustering test tasks.
2. Testing Resource Manager: This layer is responsible for allocating test resources and creating test environment accord- ing to the requests processed by test task manager. It monitors hardware resources and manages all the virtual machines used for testing. This layer also contains some different modules, such as virtual machine controller and resource monitor.
3. Testing Layer: This layer is responsible for executing test cases on virtual machines.
4. Testing Database: This is the layer where test tasks, test cases, and testing result are stored.

The test task manager is the unique component in this cloud testing system, which performs test task aggregation. Every test task defined by users has its test requirements on the executing environment and platform. The Table III is an example of the test requirements on software different tools and methods to interact with the remote cloud testing system, give commands on what to test and transfer test cases as well as collect testing results.

Test Task Manager: In order to optimize the cloud testing system, LianYuetal. [2] develop a test task manager. This layer checks whether different test tasks can be clustered and executed together in one testing environment, and how these tasks can be scheduled in an appropriate order if they are

Main Features of Cloud Testing Systems

Though the two cloud testing systems we illustrated in our paper use special systems and have one-of-a-kind traits, they have one basic feature in not unusual which also can be taken into consideration because the wonderful characteristic of TaaS structures: Both of them have a resource supervisor for dynamically allocating testing for a couple of customers concurrently as re- quested. Different from conventional computerized testing machine, TaaS or cloud trying out machine ought to provide testing environment and structures on demand. A cloud testing gadget may be considered because the combination of PaaS and SaaS: it gives no longer best the trying out sources and surroundings as the platform, however the computerized testing as a service. Some research best consciousness on the platform element, and they call this sort of provider as test environment as a service (TEaaS)

Challenges

Challenges in Cloud Testing

1. Testing Security: Security is one major concern in all kinds of cloud systems, including current SaaS, PaaS and cloud testing system. How to upload data to the cloud and use the remote computing resources safely and privately has already become a hot topic in the field of software engineering. For cloud testing system, designers need to pay more attention to the test data management, which is one of the biggest and critical open questions in TaaS [9] The source code of test cases and applications can be confidential by either executing the test in some private clusters or providing specific and strong protections for the customers.
2. Environment Construction: One of the biggest differ- ences between conventional testing systems and cloud testing systems is on-demand environment construction. The cloud testing system should allocate testing resources and build up the testbed automatically as required. Though current TaaS systems we mentioned in Section IV have the ability to construct the testing environment on demand, they can not guarantee high cost-efficiency for the system. To implement setting up test environment on demand in a cost-effective way, the cloud testing system needs to understand more about test tasks with specific techniques. Some PaaS techniques can be used here for enhancing the efficiency of construction.
3. Scalability: The cloud testing system could be more cost-effective when concurrent and parallel computing techniques are applied. However, the first thing to use concurrent computing for cloud testing is to find appropriate ways to scale automated test systems [10]. When the system is very large, new challenges arise. The TaaS systems we introduce in Section IV can execute automated tests on demand, but if there are thousands of computers to manage and a large number of users together, the resource allocation and test monitor will become difficult. New techniques are needed for management the testing resources dynamically in a large scale.

Conclusion

Cloud computing is a popular research field which not only brings opportunities to software testing, but also raises challenges. With the development of networks and software engineering, more researchers and companies start to test software in the cloud.

However, cloud testing has no clear meaning or definition although it has been proposed for several years. In this paper, we presents some major features and benefits as well as the trade-offs and challenges of cloud software testing aiming to obtain better understanding about it. In addition, we study two types of practical schemes for cloud testing and analyse their major characteristics and structures. Moreover, this paper also discuss some basic requirements of effective cloud testing systems.

We hope that this paper can answer some basic questions about the nature of cloud testing systems, and provide some hints for efficient cloud testing and TaaS system design. computing for cloud testing is to find appropriate ways to scale automated test systems [10]. When the system is very large, new challenges arise.

The TaaS systems we introduce in Section IV can execute automated tests on demand, but if there are thousands of computers to manage and a large number of users together, the resource allocation and test monitor will become difficult. New techniques are needed for management the testing resources dynamically in a large scale.

References

1. J. Wu, C. Wang, Y. Liu, and L. Zhang, “Agaric – a hybrid cloud based testing platform,” in Proceedings of the 2011 International Conference on Cloud and Service Computing, ser. CSC ’11. Washington, DC, USA: IEEE Computer Society, 2011, pp. 87–94
2. L. Yu, W.-T. Tsai, X. Chen, L. Liu, Y. Zhao, L. Tang, and W. Zhao, “Testing as a service over cloud,” in Proceedings of the 2010 Fifth IEEE International Symposium on Service Oriented System Engineering, ser. SOSE ’10. Washington, DC, USA: IEEE Computer Society, 2010, pp. 181–188.
3. T. Hanawa, T. Banzai, H. Koizumi, R. Kanbayashi, T. Imada, and
4. M. Sato, “Large-scale software testing environment using cloud com- puting technology for dependable parallel and distributed systems,” in Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops, ser. ICSTW ’10. Wash- ington, DC, USA: IEEE Computer Society, 2010, pp. 428–433.
5. L. Yu, X. Li, and Z. Li, “Testing tasks management in testing cloud environment,” in Proceedings of the 2011 IEEE 35th Annual Computer Software and Applications Conference, ser. COMPSAC ’11. Washing- ton, DC, USA: IEEE Computer Society, 2011, pp. 76–85.
6. X. B. Jerry Gao and W.-T. Tsai, “Cloud testing - issues, challenges, needs and practice,” Software engineering: an international Journal (SeiJ), vol. 1, no. 1, p. 923, 2011.
7. T. Parveen and S. Tilley, “When to migrate software testing to the cloud?” in Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops, ser. ICSTW ’10. Washington, DC, USA: IEEE Computer Society, 2010, pp. 424– 427.
8. K. Incki, I. Ari, and H. Sozer, “A survey of software testing in the cloud,” in Proceedings of the 2012 IEEE Sixth International Conference on Software Security and Reliability Companion, ser. SERE-C ’12. Washington, DC, USA: IEEE Computer Society, 2012, pp. 18–23.
9. D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman, L. Yous- eff, and D. Zagorodnov, “The eucalyptus open-source cloud-computing system,” in Proceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, ser. CCGRID ’09. Washington, DC, USA: IEEE Computer Society, 2009, pp. 124–131.
10. L. Riungu-Kalliosaari, O. Taipale, and K. Smolander, “Testing in the cloud: Exploring the practice,” IEEE Softw., vol. 29, no. 2, pp. 46–51, Mar. 2012.
11. G. Candea, S. Bucur, and C. Zamfir, “Automated software testing as a service,” in Proceedings of the 1st ACM symposium on Cloud computing, ser. SoCC ’10. New York, NY, USA: ACM, 2010, pp. 155–160.