Browse Prior Art Database

CRITERIA FOR THE ADEQUACY OF TEST DATA

IP.com Disclosure Number: IPCOM000128202D
Original Publication Date: 1980-Dec-31
Included in the Prior Art Database: 2005-Sep-15
Document File: 13 page(s) / 42K

Publishing Venue

Software Patent Institute

Related People

Elaine Weyuker: AUTHOR [+3]

Abstract

Goodenough and Gerhart [7j defined an ideal set of tests to have properties which would imply that the tests are capable of exposing all errors in a program. Thus, if a program produces correct results on a set of ideal tests, it must be correct. However, these properties are non-constructive in the sense that they do not tell us how to produce ideal tests for a given program. In addition, it is generally impossible to determine whether a given set of tests for a program is ideal. Lacking a guaranteed way to create tests which can conclusively demonstrate correctness, software test developers need a method of detemining when sufficient testing has been done. Such an adeduacy criterion for test data should reflect the test's ability to expose errors in the program. Many of the adequacy criteria which have been proposed and are in use today approach this natural goal only indirectly. It is common, for example, to require that every statement, branch, or path fulfilling some condition be traversed in order that test data be deemed adequate [14, 24], even though it has been pointed out [3, 7, 23] that these notions of adequacy suffer from deficiencies. In particular it is easy to devise simple programs and test data such that, even though the program contains errors, the requirements of each of these criteria are fulfilled. Since the goal of testing is to detect the presence of errors, and these notions of adequacy measure code traversal, it is not too surprising that they are not really satisfactory indicators of how thoroughly the program has been tested. Furthermore, these criteria are themselves untestable in general in the sense that there can be no algorithm to decide of an arbitrary program whether there exist test data which will cause a given statement, branch, or path to be traversed, or whether every statement, branch, or path can be traversed [21]. Several other criteria for test data adequacy have beer. proposed and discussed. Error seeding [16] consists of the deliberate implantation of bugs in the program being tested. The buggy version of the program is then run on the set of test data which has been proposed as adequate to see how many of the implanted bugs are exposed. If k$ of the implanted bugs are located, it is then assumed that k% of the original bugs have been found. This technique assumes that the types and distribution of bugs which occur unintentionally are the same as those implanted, a convenient, but usually inaccurate, assumption.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 9% of the total text.

Page 1 of 13

THIS DOCUMENT IS AN APPROXIMATE REPRESENTATION OF THE ORIGINAL.

CRITERIA FOR THE ADEQUACY OF TEST DATA

By

Elaine Weyuker

May 1980

Report #22 ~i :::-,;,::::v;.:- , ,. "rh: T

2. EXISTING NOTIONS OF TEST DATA ADEQUACY

Goodenough and Gerhart [7j defined an ideal set of tests to have properties which would imply that the tests are capable of exposing all errors in a program. Thus, if a program produces correct results on a set of ideal tests, it must be correct. However, these properties are non- constructive in the sense that they do not tell us how to produce ideal tests for a given program. In addition, it is generally impossible to determine whether a given set of tests for a program is ideal. Lacking a guaranteed way to create tests which can conclusively demonstrate correctness, software test developers need a method of detemining when sufficient testing has been done. Such an adeduacy criterion for test data should reflect the test's ability to expose errors in the program. Many of the adequacy criteria which have been proposed and are in use today approach this natural goal only indirectly. It is common, for example, to require that every statement, branch, or path fulfilling some condition be traversed in order that test data be deemed adequate [14, 24], even though it has been pointed out [3, 7, 23] that these notions of adequacy suffer from deficiencies. In particular it is easy to devise simple programs and test data such that, even though the program contains errors, the requirements of each of these criteria are fulfilled. Since the goal of testing is to detect the presence of errors, and these notions of adequacy measure code traversal, it is not too surprising that they are not really satisfactory indicators of how thoroughly the program has been tested. Furthermore, these criteria are themselves untestable in general in the sense that there can be no algorithm to decide of an arbitrary program whether there exist test data which will cause a given statement, branch, or path to be traversed, or whether every statement, branch, or path can be traversed
[21]. Several other criteria for test data adequacy have beer. proposed and discussed. Error seeding [16] consists of the deliberate implantation of bugs in the program being tested. The buggy version of the program is then run on the set of test data which has been proposed as adequate to see how many of the implanted bugs are exposed. If k$ of the implanted bugs are located, it is then assumed that k% of the original bugs have been found. This technique assumes that the types and distribution of bugs which occur unintentionally are the same as those implanted, a convenient, but usually inaccurate, assumption. Another proposed adequacy criterion is known as the program mutation method [1, 3]. This system makes a series of minor changes to the program being tested, creating a set of programs known as mutations. Some of these modifications cause program errors, while other...