Mechanism And Algorithm For Optimizing Test Case Execution Using Historical Data
Publication Date: 2010-Aug-04
The IP.com Prior Art Database
The disclosed describes a mechanism for optimally prioritizing test cases for execution. The mechanism collects and stores statistics from the execution of test cases and employs a ranking model using a specific algorithm. test case rank indicates priority level.
Mechanism And Algorithm For Optimizing Test Case Execution Using Historical
A test case validates a specific feature or set of features and is typically composed of
which are the most granular level of testing.
The development of large software products involves the creation of a set of test cases.
The set of test cases associated with a product grows as the product evolves and
engineers incorporate new features. Complex and mature software products such as
IBM DB2TM often require hundreds of thousands of test cases for validation.
Managing, executing and maintaining a large set of test cases presents a number of
challenges, such as:
1. Prioritizing the execution of test cases.
2. Selecting optimal sets to execute with limited resources (i.e., time, machine).
3. Identifying test cases which require significant effort to maintain. (Engineers use
such sets to focus refactoring and reengineering efforts). Long-running test
cases without a history of identifying quality regressions are candidates for
4. Efficiently distributing work to a set of machines for execution.
5. Efficiently tolerating non-robust test cases until they can be reengineered or
Current solutions present methods that are based on code coverage of test cases. The
disclosed algorithm and method are unique because they are based on historical data
from existing test cases independent of coverage.
This invention uses data from the Test Case Aggregate Results and Classification
Model (TARCM) to prioritize the order in which test cases are dispatched for the
purposes of gaining an initial indication of product quality and ensuring the results of
historically problematic test cases are available to testers first. This optimally prioritizes
test cases for execution.
Another application of the model involves using it to classify
test cases and to optimize test case execution based on a given criteria. The key
attributes include ordering of test case execution so that:
[circle6]The process assigns test cases a ranking from a number of criteria. For
example, a test case is valued more favorably if it has a history of discovering
whereas a test case is valued less favorably if it requires a
long time to execute. The mechanism executes the most valuable test cases
one or more test units,
[circle6]The process is adaptive and automatic. If the characteristics of one of the test
cases or the product change, then the ranking system adjusts and re-ranks the
[circle6]Developers can use the ranking and classification system to focus
reengineering efforts or identify candidates for deprecation.
The invention uses TARCM to assign a test case rank based on its execution duration
and its history of finding code defects.
1. Rank each test case using data from the TARCM. Rank = (
A and B constants can be used to control the importance of number of code
regressions found and the average runtime of the te...