I'm having trouble figuring out the appropriate way to place a confidence interval on a ratio I'm estimating. It's for a social science application, but I'll present the problem via coin-flips as I think its clearer.
Imagine you have $N=100$ coins, and you are going to flip each coin 3 times. Your null hypothesis H0 is that all the coins are fair, $p=q=.5$. But you suspect (H1) that actually some of the coins are fake such that they always return heads (which we will set equal to 1). You flip each coin three times, such that your results are N binomial variates of size=3.
I can observe the proportion of 'all heads' trials out of the N coins, and I can also calculate the predicted number of 'all heads' trials under the null hypothesis, which is $p^3N$. Say I observe 16 'all heads' trials. Under the null I expected to observe $(.5times.5times.5)N=12.5$ 'all heads' trials – how can I tell if these numbers are meaningfully different?
I know that the variance is $Npq$ for the proportion of successes $n$ in $N$ coin flips, but that doesn't seem the right way to go. In particular, the expected value of my result is $p^3N$, should I be treating an 'all heads' outcome like a success in a bernoulli trial of size $N$, and thus calculate variance as $Np^3(1-p^3)$?
Off the top of my head:
H0 : all coins are fair H1 : at least one gives only heads
I see it as a binomial where the experiment is "throwing a coin 3 times", with N=100, just like you suggested.
Hence the number of "3 heads" $X$ follows a binomial distribution. $$ X sim B(n=100,; p=1/8) $$
This can be approximated by a normal law $$ N(mu=pn,; sigma² = p(1-p)n) $$
So if you want say a 95% confidence interval (z=1.645), you will reject H0 if $$ X > mu + z sigma = 17.94 approx 18$$
So if when you get 18 or more "3 heads" you conclude "something is fishy!!", you'll be wrong only about 5% of the time.
- Solved – Calculating confidence interval for whether some of N coins are unfair
- Solved – How to combine two beta-binomial distributions
- Solved – Using Bayes Theorem to calculate the probability of a coin being biased given a specific test outcome
- Solved – Can confidence intervals be added
- Solved – How to combine likelihoods from multiple binomial trials. Can we also determine the overall FDR