Solved – How to test equivalence of two models using MSE

I want to compare two generalized linear mixed models (GLMM), model A and model B, which differ from their link function. More specifically, I want to test wheter these models are equivalent or model A is better than model B.
The dependant variable is an always positive and continuous variable. Model A is the linear mixed model and model B is the GLMM with the log link function. I want to proove that using the additive model or multiplicative model is equivalent$^*$. I have no assumption about whether the model is additive or multiplicative.
I choose the Mean Squared Error (MSE i.e. $displaystyle frac{1}{N}sum^N_{i=1} left(widehat{y}_i – y_iright)^2$) as criterion for comparing the two models.

Thus, the hypotheses are $H_0 : MSE_A = MSE_B $ and $H_1 : MSE_A < MSE_B$

The test statistic is $T = MSE_A – MSE_B $

As the T-distribution is unknow, I thought to :

  1. create P datasets randomly permuted between the predictions of the two models (i.e. for each $i$, $widehat{y}_{A,i}$ randomly assigne to the A or B model and $widehat{y}_{B,i}$ to the other one)
  2. for each permuted dataset $p$, compute the $T^*_p$ statistic
  3. compare the observed $T$ value to the $T^*$ distribution to conclude on the test

Is it a valid process? If not, any suggestion on how to implement this test?

$^*$ My boss want me to use the additive model. I want to proove him (or not) that is equivalent to use the multiplicative model,regarding model performance. Thus, if two models are equivalent, it is more accurate to use the GLMM with log link function because the dependant variable is always positive.

If you just want to compare the link functions and find which one is the "best", I think you can use a likelihood-ratio test, where you compare the ratio of likelihoods with the quantile of a $chi^2$ distribution (Wilks theorem). If the test rejects H0:"the likelihoods are equal" then it means that one of the models gets a significantly better fit to the data. So, you can select the one with the highest likelihood. For GLMM, you have numerical methods for evaluating the likelihood, take a look here for instance : Or inside the R package "glmm".

If the variables of your models are not the same, it seems to me that it is more adequate to use criteria like Akaike Information Criteria (AIC) or Bayesian Information Criteria (BIC). This kind of criterion allow to select the model with the best fit to the data, while avoiding overfitting. To compute these criteria, you also have to compute the likelihood of each model. Next, you choose the model with the smaller criterion.

Similar Posts:

Rate this post

Leave a Comment