Browse Prior Art Database

Statistical Methods for Categorizing the Performance and Variability of Tests of Dynamic Binary Translators

IP.com Disclosure Number: IPCOM000199538D
Publication Date: 2010-Sep-08
Document File: 5 page(s) / 174K

Publishing Venue

The IP.com Prior Art Database

Abstract

In measuring computer performance it is highly desirable to be able to test with benchmarks that have minimal variability i.e. that give repeatable and consistent results. This allows experimentation at a greater resolution than if the benchmarks varied in an unpredictable way. This paper develops the mathematical framework for the measurement and reporting of variability in benchmarks.

This text was extracted from a PDF file.
At least one non-text object (such as an image or picture) has been suppressed.
This is the abbreviated version, containing approximately 50% of the total text.

Page 1 of 5

Statistical Methods for Categorizing the Performance and Variability of Tests of Dynamic Binary Translators

1 Introduction

In measuring computer performance it is highly desirable to be able to test with benchmarks that have minimal variability i.e. that give repeatable and consistent results. This allows experimentation at a greater resolution than if the benchmarks varied in an unpredictable way.

This paper develops the mathematical framework for the measurement and reporting of variability in benchmarks.

2 Confidence Intervals

A Confidence Interval is a commonly-used method of indicating how likely it is that an estimated parameter lies within a given range in relation to the true value of the parameter that is estimated; how likely the estimate is, is determined by the confidence coefficient. The more likely it is for the interval to contain the parameter, the wider the interval will be.

Consider a normally distributed population with mean

µ and variance

2. Suppose

xn } are an independent sample from this population. The sample

arithmetic mean and the square of the sample standard deviation s2are estimators for

{

x,xx3 …,

1

2

,

µ and

,

2and are given by

[This page contains 3 pictures or other non-text objects]

Page 2 of 5

Now the Student's t-distribution is a probability distribution that arises in the problem of estimating the mean of a normally distributed population when the sample size is small. It is a special case of the generalised hyperbolic distribution. It can be shown that the transformed value T,

(3)

has a Student's t-distribution with (

N

- 1) degrees of freedom. Now it is possible to

find numbers

-t

and t

,

independent of

,

µ where T lies in between such that

and this is equivalent to

so t is the " 95th percentile" of this probability distribution and can be more generally written as t1 -α ⁄ 2;

N-

1. Then

which is equivalent to

Therefore the interval

is a stochastic 90-percent confidence interval for

µ . s and

N

                                         can be calculated from the data, and is usually looked up in a table because calculating it is complicated.

What does this mean? It is tempting to think this means:

2

[This page contains 7 pictures or other non-text objects]

Page 3 of 5

There is a 90% chance a result will fall within the confidence interval.

However, this is incorrect. Adopting a frequentist method, the correct explanation of a confidence interval is:

The confidence interval represents values for the population parameter for which the difference between the parameter and the observed estimate is not statistically significant at the 10% level.

3 Using Confidence Intervals to Calculate a Variability Metric

One standard variability metric is the sample standard deviation, which is

In order to facilitate comparisons it is normal to standardise this value by expressing
it as a proportion of the measured mean . Thus the coefficient of variation (COV) is defined to be

and so provides a dimensionless value that compares the relative size...