Browse Prior Art Database

Automated verification and ranking of tests results using SVM

IP.com Disclosure Number: IPCOM000227868D
Publication Date: 2013-May-23
Document File: 2 page(s) / 40K

Publishing Venue

The IP.com Prior Art Database

Abstract

Significant human resources may be needed to analyse results of regression tests. We propose classification of the test results based on Support Vector Machine (SVM) algorithm, in order to reduce the required efforts - especially to reduce time and resources spent on analysing false positive results.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 52% of the total text.

Page 01 of 2

Automated verification and ranking of tests results using SVM

Having 1000 regression tests running each night requires significant human resources to analyze execution results. Let's have 3 templates with correct behaviour per one testcase that gives us 3000 templates checked each night. If 10% of tests fail it give us 300 differences between template and current state to manual analysis. From our experience around 60% of all differences are caused by environmental changes, positive development changes (new features), and human mistakes. Less than 40% are actually valid regression defects. Let's imagine how much the manual work can decreased if all failures are categorized to 2 classes: false positive regression, real regression and within each class the confidence factor is assigned to each failure. It will allow the test team to focus in first place on items from set 2 (real regression) and re-use confidence factor for work prioritization.

Let's adopt Support Vector Machine (SVM) algorithm to classify failures to two classes:

1 - false positive regression,


2 - real regression.

In addition, assign to each failure in each class confidence level which is the distance to created optimized hyperplane. The most important part is selecting significant features for SVM algorithm - basing on our experience and historical data (RTC, past execution results) we have created the features list (for details please see section below). The main advantages of proposed method are:

reduction of automated tests maintenance costs


"smart" prioritization of queued failures

faster reaction and defect reporting for real regression - less time spent on false positive differences analysis


selection of the "unstab...