Comparison and analysis of test-results across test-dimensions
Publication Date: 2014-Jun-25
The IP.com Prior Art Database
This publication describes a system that automatically runs suites of tests for a number of combinations of configuration and system variations (dimensions). The system reports the test results, tagged with the combination of dimensions, and allows visual organisation and comparison of the results for sub-sets of the dimension combinations. Where test failures occur only in certain combinations of dimensions, the system analyses results to identify the causes of test failures, and allows the tests to be re-run with the identified causes in combination with alternate dimensions.
Page 01 of 10
Comparison and analysis of test
Comparison and analysis of test-
results across test-
Disclosed is a system that:
automatically runs a suite or suites of tests for a number of combinations of configuration and system variations (dimensions),
reports the test results to a repository, tagged with the combination of dimensions, and
allows visual organisation and comparison of the results for sub-sets of the dimension combinations, and
--results across test
results across test
where test failures occur only in certain combinations of dimensions, analyses results to identify the causes of test failures, and
allows the tests to be re-run with the identified causes in combination with alternate dimensions.
Software products can often be run on a range of platforms . Large enterprise applications are typically supported on a number of operating systems, and will often run in combination with a number of third party applications, such as databases, and LDAP (Lightweight Directory Access Protocol) servers. On top of this, these applications may be configured in a number of ways. They may also be accessed through a number of client applications (e.g. one of the popular web browsers for web applications).
During development and support, software products are automatically tested, using automated tests, to verify correct operation of new and existing function.
Test results may however vary across dimensions, i.e. product defects may only happen on certain platforms or configurations, or combinations thereof.
Existing tools such as Jenkins®* CI  can be configured to run tests across combinations of parameters (referred to as matrix builds, equivalent to this disclosure's dimensions).
This disclosure aims to improve the ease of comparison and analysis of these results .
The system consists of a number of modules, each with specific function, but working in combination.
A Test-run is the process of running all or a sub-set of the available Test-cases against one or more systems configured in a common way (i.e. with the same Dimensions).
A Dimension is a key-value pair parameter to a Test-run, which may take discreet or continuous values. For example OPERATING_SYSTEM may equal one of Windows, Linux; or SCREEN_WIDTH may be between 800 and 1900.
Page 02 of 10
A System is the computer or cluster where the tests are run.
Dimensions may apply to a System, (e.g. OPERATING_SYSTEM, MEMORY) or to Configuration (e.g. USERNAME, CONTEXT_ROOT).
Referring to Figure 1. The modules:
The Test-run orchestrator module triggers Test-runs for specified combinations of Dimensions, and can match these to an appropriate System (based on System Dimensions).
A Test-runner module runs on a System and is responsible for applying any Configuration Dimensions, and running the tests.
The tests may be unit or integrated tests.
A Test-reporter module reports test results from the Test-runner module to the Result-storage module, tagged...