Method for Determination of Product Quality by Measurement of Positive/Negative Testing
Original Publication Date: 2009-Mar-16
Included in the Prior Art Database: 2009-Mar-16
Software testing covers a broad range of character - from positive testing for proof of basic functionality, to negative testing for proof of adequate error handling. This article describes a mechanism by which the character of existing tests can be measured and understood automatically to (i) provide a perspective on current test efficacy and (ii) assist with quality-based software release decisions.
Method for Determination of Product Quality by Measurement of Positive /Negative Testing
Software tests generally comprise components of "positive" or "negative" character.
Positive tests generally seek to demonstrate "golden path" functionality works in a product.
Negative (or error) tests generally seek to ensure that the software copes under non-ideal and error conditions. For example, faults are deliberately injected into the system (such as invalid data fields, dropped network connections or hardware failures) to ensure the software adequately recovers. Such resilience under error conditions is often a fundamental competitive advantage in enterprise software.
The bias between execution of positive and negative tests generally shifts as the development cycle progresses. Although positive testing continues to demonstrate no regressions are introduced, negative testing increasingly builds on these foundations to ensure the software copes with anticipated error conditions.
Consider all test cases that have been written for a product. In general, each individual test case will have a mixture of positive and negative character. For example, a test case is never truly negative: some positive aspects of code must be exercised to bring the system ready to test a particular error case. Understanding this mix of positive and negative character will help address two important problems facing project and release managers today:
To provide a quantitative measurement of negative, error or recovery
testing, and hence assure this testing is adequate for the product's target market. For example, (i) to drive increased product resilience through a demonstrable improvement in recovery testing from release X to release X+1 and
(ii) to better understand the level of actual recovery code driven by a test - a "simple" recovery test may in fact drive the product code very hard.
To better understand the impact of test failures and hence assist with
product ship decisions. For example, a defect in a negative test may represent
a rarely reached error case - and hence have a low immediate impact on customers. In contrast, a failure in a positive test case could demonstrate that some aspect of basic functionality does not work where customer impact is markedly higher.
There is currently no solution for understanding this mixture of positive and negative character in a test case - apart from crude estimation - and hence no rigorous mechanism for addressing the two problems outlined above.
Positive" and "negative" attributes are assigned to the underlying code of the product under test, either through specific input by developers or by automatic analysis. These characterisation are then used to provide additional information for code coverage analysis: test cases ar...