Browse Prior Art Database

Automated Feedback for Automated Test Generation

IP.com Disclosure Number: IPCOM000028261D
Original Publication Date: 2004-May-06
Included in the Prior Art Database: 2004-May-06
Document File: 4 page(s) / 80K

Publishing Venue

IBM

Abstract

The use of automated test generators is well established in the fields of telecommunications, protocol verification, and hardware testing, but less so in software testing. These automated test generators are guided in their selection of test cases by either specific test purposes or general coverage criteria. One of the main drawbacks to the use of test suites automatically generated from general coverage criteria (see US Patent Application 09/847309, "Technique Using Persistent Foci For Finite State Machine Based Software Test Generation") is the large volume of test cases generated. The extensive testing produces large volumes of trace data that cannot be manually inspected. Our invention proposes the use of automation in the analysis of the trace data, combined with automated generation of new test purposes to create a smaller, more focussed set of test cases automatically. This feedback from trace analysis to test generation is a process carried out manually and inefficiently today with manually generated test suites. For large, automatically generated test suites it is impossible with current technology.

This text was extracted from a PDF file.
At least one non-text object (such as an image or picture) has been suppressed.
This is the abbreviated version, containing approximately 37% of the total text.

Page 1 of 4

Automated Feedback for Automated Test Generation

The invention revolves around the analysis of large volumes of test case trace data. We distinguish two types of automated feedback:
defect analysis and feedback sub-sequence coverage analysis and feedback

    When a large test suite is executed against a defective implementation, the number of failing test cases may be very large, even though the number of defects causing the failures may be quite small. We are proposing a clustering analysis of failed test cases to group test cases which fail for similar reasons or after similar sequences of test stimuli.

    We further analyse the clusters of similar failing test cases and extract a set of "common features" from the test cases in a cluster. These common features are then used to create a new test purpose, which can be fed back into an automated test generator. The test generator will generate a few tests (possibly only one) which contain these features and thus has a high likelihood of causing the failure to recur. A single test case and a list of the common features can be used as a guide to the developers of the software in diagnosing and repairing the defective software.

    Finally we would rerun the entire test suite after fixing the problems diagnosed for each of the clusters in order to verify that the problems had been fixed and to expose the next set of defects.

    A method of deriving feedback from coverage analysis was described in the patent application "A METHOD, SYSTEM, AND COMPUTER-PROGRAM PRODUCT FOR INTEGRATING TEST COVERAGE MEASUREMENTS WITH MODEL BASED TEST GENERATION" (US Patent Application 09/946237). This invention is a further development of the idea, with implementation details for a particular kind of coverage tasks which are sub-sequences of the test case trace.

    The key elements of the system are the clustering engine, the feature extractor, and the test purpose synthesizer. These three elements can be used for the defect analysis feedback as well as to implement the coverage analysis feedback in an efficient way, as illustrated in the diagram below.

Page 2 of 4

Test G enerator

Test S uite Test trace records

Coverage Failing tests

Test P urpose S ynthesizer

analysis

Feedback Loop

The clustering engine works by defining a metric on the objects to be clustered, then by using common, well-known clustering heuristics to group objects that are similar into clusters. The key to creating a good clustering mechanism is the definition of an appropriate metric. We now give the details of: the metric used to define the distance (dissimilarity) between two defect traces,

and the metric used to define the distance (dissimilarity) between two coverage tasks.

E xtractor

                          Clustering Feature engine


1.


2.

    The format of the trace record of a failing test case consists of a sequence of stimuli to the system under test (SUT) mixed with a sequence of observations of the SUT, and culminating either in an observation that conflicts with the exp...