I am looking to compare the fit of a zero-inflated mixture model and a Poisson mixture model. The random effects in both models are different.

Comparing the fitted values of both models ignores the complexity of the models, and using model selection methods such as DIC, AIC, etc., is not a straightforward exercise due to the difference in mixture distributions, so I am wondering if there is a better way?

**Contents**hide

#### Best Answer

Probably what you will need to use is the *Parametric Bootstrap Cross-fitting Method*. Here is the basic procedure:

- Fit each model to the data. Estimate the models' parameters and extract your favorite measure of goodness of fit. We will call the model with the higher value for this GoF measure $A$ and the other model $B$. Calculate the difference $d$ between the two measures of GoF, and store that value. (Be sure you are clear about whether higher or lower numbers of your GoF measure indicate a better fit–i.e., whether $d>0$ implies $A$ is better or worse.)
- Using the fitted parameters for $A$ from step 1, generate a large number of synthetic datasets (say 1000). With each of these datasets, fit both of your models, extract their GoF measures, compute $d$ and store it.
- Using the fitted parameters for $B$ from step 1, generate another set of (1000) synthetic datasets. With these datasets, again fit your models, compute the $d$s and store them.
- You now know what the sampling distribution of $d$ looks like when the true model is $A$ and when the true model is $B$. Determine the cutpoint, $d_text{cut}$, that optimally differentiates between the models. If you want, you can bring prior knowledge to bear by differentially weighting the alternatives.
- Compare your found $d$ from step 1 to $d_text{cut}$ and select the corresponding model.

I demonstrate this approach here. (There is another description of PBCM in this answer: Measures of model complexity.) Here is the reference:

- Wagonmakers, E.J., Ratcliff, R., Gomez, P., & Iverson, G.J. (2004). Assessing model mimicry using the parametric bootstrap,
*Journal of Mathematical Psychology, 48*, pp. 28-50. (pdf)

### Similar Posts:

- Solved – Using AIC to distinguish between models using multiple datasets
- Solved – Choosing best fit regression model from root MSE and adjusted R squared
- Solved – Best metric for evaluation of mixture-of-Gaussian clusters on big-data
- Solved – Number of free parameters in Gaussian mixture models
- Solved – Number of free parameters in Gaussian mixture models