Browse Prior Art Database

Maintaining Test Suite Vitality - Dynamic Baseline Refresh Disclosure Number: IPCOM000250902D
Publication Date: 2017-Sep-13
Document File: 5 page(s) / 181K

Publishing Venue

The Prior Art Database

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 37% of the total text.

Maintaining Test Suite Vitality - Dynamic Baseline Refresh

Delivery of enterprise software (and associated maintenance) typically includes extensive QA processes to ensure that each delivery meets end user expectations of quality and does not cause new issues that impact the customer's critical production environments. Functional and system test cases will be written and then automated such that they can be run repeatedly to ensure new capabilities are working and existing capabilities are not regressed. For a mature enterprise software product, such test suites can grow to include many thousands of test cases. It is not practical in many cases to run the entire test suite for every delivery due to the machine and people resources required to execute all the tests and perform analysis of any failures (which may be related to product, test material or environmental problems), particularly for fixes/maintenance which tend to be released on a regular cadence.

Targeted testing is a known approach that uses code coverage analysis to create an index identifying which test cases exercise particular areas of product code. The index is then used to select and execute appropriate test cases for given code changes. The index itself can be created using a variety of techniques. On other platforms, entry and exit trace output can be used to identify the functions or methods that each test invokes during execution. However such an index (or "baseline") is constructed, it's purpose is to enable a software engineer to retrieve a list of test cases that relate specifically to the code they have changed, for example in order to test a defect fix in a particular component. The relevant tests can be run quickly and unnecessary further testing is avoided, enabling a faster delivery of the fix to the end user without compromising the QA process.

This approach relies upon the baseline being up to date. Changes to product code and test material as a result of ongoing development can degrade the baseline, causing test cases to become relevant to new code branches, or conversely no longer relevant to code branches which formerly they were. Ensuring the baseline is completely up-to-date requires that code coverage analysis be refreshed to capture all such changes throughout the product code base. For products under active development, this presents just as significant a resource challenge as running the whole test suite. Failure to keep the baseline updated may result in relevant tests not being identified for particular code changes, which in the worst case can lead to expensive field regressions where an existing test that could have caught the issue was not executed.

The idea presents a method of maintaining an acceptable level of vitality to baseline test coverage for software development and maintenance. This method does not sacrifice quality of delivery below what an organisation deems to be an acceptable level whilst at the same time enabling a corresponding amo...