As an example, consider a program that executes on two computers, A and B.

Measuring the execution time of 3 executions each shows the following results:

System A: 10s, 10s, 4s

System B: 8s, 8s, 2s

With these values, we can calculate the mean and the confidence interval for both systems (considering Student's t distribution for a confidence of 90%):

A: 8s +- 5.84

B: 6s +- 5.84

We can now calculate an average speedup between System A and B as 8s/6s =

1.33.

**My question is**: Is it possible to calculate an error or confidence interval for this speedup? If yes, how?

**Contents**hide

#### Best Answer

You should take a look at the theory of 'ratio estimators' , there are references to find via google (e.g. http://www.math.montana.edu/~parker/PattersonStats/Ratio.pdf). The idea is that you can compute the mean and the variance of a ratio of random variables and then use the mean and variance to define confidence intervals.

But I think you will need larger samples.

**@Matthias Diener**

So for your example: $y = (10, 10, 4), bar{y}=8$, $x=(8,8,2), bar{x}=6$

the ratio estimator for your speedup is $r=frac{bar{y}}{bar{x}}=frac{8}{6}=1.333$ and the variance of the estimator is $sigma_r^2=frac{1}{bar{x}^2}frac{s_r^2}{3}$, with $s_r^2=frac{1}{3-1}sum_{i=1}^3 (y_i – rx_i)^2$.

The R-code looks like:

`y<-c(10, 10, 4) x<-c(8, 8 , 2) m.x<-mean(x) m.y<-mean(y) m.x m.y r<-m.y/m.x r s.r.2<-1/(length(x)-1) * sum(( y - r * x ) * ( y - r * x)) variance.r<- 1/m.x^2 * s.r.2/length(x) stand.deviation.r<-sqrt(variance.r) cat(paste("estimate for speedup:", round(r, digits=3), "with stand. deviation:", round(stand.deviation.r, digits=3))) `

### Similar Posts:

- Solved – Confidence interval for parameter in uniform distribution using MOM estimator
- Solved – Confidence interval for parameter in uniform distribution using MOM estimator
- Solved – Confidence Interval Widths as Measure for Variability
- Solved – How to calculate expected value and standard deviation if I have 100 values divided into 15 groups (normal distribution)
- Solved – How many samples to get the accuracy to the Kth digits of Pi