Dismiss
InnovationQ will be updated on Sunday, Oct. 22, from 10am ET - noon. You may experience brief service interruptions during that time.
Browse Prior Art Database

Comparison and analysis of test-results across test-dimensions

IP.com Disclosure Number: IPCOM000237572D
Publication Date: 2014-Jun-25

Publishing Venue

The IP.com Prior Art Database

Abstract

This publication describes a system that automatically runs suites of tests for a number of combinations of configuration and system variations (dimensions). The system reports the test results, tagged with the combination of dimensions, and allows visual organisation and comparison of the results for sub-sets of the dimension combinations. Where test failures occur only in certain combinations of dimensions, the system analyses results to identify the causes of test failures, and allows the tests to be re-run with the identified causes in combination with alternate dimensions.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 36% of the total text.

Page 01 of 10

Comparison and analysis of test

Comparison and analysis of test-

results across test-

--dimensions

dimensions

Disclosed is a system that:


automatically runs a suite or suites of tests for a number of combinations of configuration and system variations (dimensions),

reports the test results to a repository, tagged with the combination of dimensions, and


allows visual organisation and comparison of the results for sub-sets of the dimension combinations, and

--results across test

results across test

where test failures occur only in certain combinations of dimensions, analyses results to identify the causes of test failures, and

allows the tests to be re-run with the identified causes in combination with alternate dimensions.

Software products can often be run on a range of platforms . Large enterprise applications are typically supported on a number of operating systems, and will often run in combination with a number of third party applications, such as databases, and LDAP (Lightweight Directory Access Protocol) servers. On top of this, these applications may be configured in a number of ways. They may also be accessed through a number of client applications (e.g. one of the popular web browsers for web applications).

During development and support, software products are automatically tested, using automated tests, to verify correct operation of new and existing function.

Test results may however vary across dimensions, i.e. product defects may only happen on certain platforms or configurations, or combinations thereof.

Existing tools such as JenkinsĀ®* CI [1] can be configured to run tests across combinations of parameters (referred to as matrix builds, equivalent to this disclosure's dimensions).

This disclosure aims to improve the ease of comparison and analysis of these results .

The system consists of a number of modules, each with specific function, but working in combination.

Definitions:

A Test-run is the process of running all or a sub-set of the available Test-cases against one or more systems configured in a common way (i.e. with the same Dimensions).

A Dimension is a key-value pair parameter to a Test-run, which may take discreet or continuous values. For example OPERATING_SYSTEM may equal one of Windows, Linux; or SCREEN_WIDTH may be between 800 and 1900.

1


Page 02 of 10

A System is the computer or cluster where the tests are run.

Dimensions may apply to a System, (e.g. OPERATING_SYSTEM, MEMORY) or to Configuration (e.g. USERNAME, CONTEXT_ROOT).

Referring to Figure 1. The modules:


The Test-run orchestrator module triggers Test-runs for specified combinations of Dimensions, and can match these to an appropriate System (based on System Dimensions).

A Test-runner module runs on a System and is responsible for applying any Configuration Dimensions, and running the tests.

The tests may be unit or integrated tests.

A Test-reporter module reports test results from the Test-runner module to the Result-storage module, tagged...