Solved – How to calculate the confidence interval of an ICC

A straightforward way to calculate a confidence interval would be to create a bootstrapped distribution, then obtain the relevant quantiles from that distribution. This can be done as a parametric or nonparametric bootstrap, depending on what assumptions you're comfortable with. Generally, if you accept a Normal distribution for the random effects, a parametric bootstrap is also appropriate.

Since you're using R, the lme4 package offers a very nice parametric bootstrap function, bootMer() to calculate a bootstrapped distribution of any statistic of interest derived from a random effects model. The parametric method is fully implemented, which simulates the model of interest using new values of the random effects with each iteration. New values are drawn from a Normal distribution using parameters derived from the mixed model. See the documentation in bootMer() for some detail on control over what effects get bootstrapped. Set your own number of iterations, but 1000+ is a good rule of thumb.

To calculate this in R, you need to fit the random effects model and pass a function to bootMer() that calculates the statistic of interest. An example implementation is below:

Example Code using R

#Make some mocked data library(lme4) library(reshape2) set.seed(2024)  #For the Bell Riots id <- factor(seq(1, 15)) id.mu <- rnorm(15, 10, 5) mydat <- NULL for (a in 1:length(id)){   score <- rnorm(2, id.mu[a], 3)   id.fr <- data.frame(id=id[a], score1=score[1], score2=score[2])   mydat <- rbind(mydat, id.fr) } mydat <- melt(mydat, id.vars='id', value.name='score')  #Create function to calculate ICC from fitted model calc.icc <- function(y) {   sumy <- summary(y)   (sumy$varcor$id[1]) / (sumy$varcor$id[1] + sumy$sigma^2) }  #Fit the random effects model and calculate the ICC mymod <- lmer(score ~ 1 + (1|id), data=mydat) summary(mymod) calc.icc(mymod)  #Calculate the bootstrap distribution boot.icc <- bootMer(mymod, calc.icc, nsim=1000)  #Draw from the bootstrap distribution the usual 95% upper and lower confidence limits quantile(boot.icc$t, c(0.025, 0.975))