Browse Prior Art Database

Method and System to Automate the Selection of Test Suites and Test Environment Setup in Continuous Testing Environments

IP.com Disclosure Number: IPCOM000243293D
Publication Date: 2015-Sep-18
Document File: 4 page(s) / 128K

Publishing Venue

The IP.com Prior Art Database

Abstract

Disclosed is a system and method to determine an optimized set of test scenarios or a test suite and an optimized set of test conditions for the execution of the automated testing in a continuous Testing Environment based on different criteria. The system considers different events and data from customers and development systems to select a set of test cases for execution in the continuous testing environment given an automatically determined set of conditions.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 40% of the total text.

Page 01 of 4

Method and System to Automate the Selection of Test Suites and Test Environment Setup in Continuous Testing Environments

In any software delivery process, testing is needed to ensure the software runs well. It is a challenging task to begin with, but in a DevOps environment where software is continuously delivered the challenge of testing the software is even greater. Risks of not identifying all the critical defects, with the limited time given to testing, are high.

In this kind of environment, the focus typically falls on automated testing as much as possible. However, given the current technologies and tools, the kind of automated testing being done in a DevOps environment is more units testing than functional testing or system testing. These tests do not take into account how the customer actually uses the software. The defects found are not those that a customer might find in actual use of the software. Defects related to specific conditions in the customer environments are found when the customer uses the software. The cost of fixing these defects at that point is much higher than it is pre-release of the software. The greatest cost is in the loss of customer satisfaction with the software.

The novel contribution is a system and method to determine an optimized set of test scenarios or a test suite and an optimized set of test conditions for the execution of the automated testing in a continuous Testing Environment based on different criteria. The system considers different events and data from customers and development systems to select a set of test cases for execution in the continuous testing environment given an automatically determined set of conditions.

Including actual customer data during the continuous testing allows the suite to find defects related to specific conditions in a customer environment before delivering the code to the clients. It also helps prioritize testing based on real customer scenarios and configurations.

The proposed system defines two sets of criteria: selection criteria and execution criteria. The selection criteria uses software development data and customers' provided data (in logs or social media) to identify the components that need to be tested

with more focus, i.e. establishing the priorities for testing. The execution criteria uses test conditions to set up the test environment for a more customer-related kind of environment test run.

Figure: System diagram including Change and Configuration Management System 100, Log Analyzer 200, Social Network 300, and External Defects Repository components

1


Page 02 of 4

Selection criteria

Within the Change and Configuration Management System 100, the Test Selection Processor 160 creates a variable for each component: component1, component2, etc., componentN, where N is the number of components, and updates this variable by invoking the following modules, described at a high level:

Components Usage Retriever 110

Most used components of the system...