I am very new to statistics and trying to implement a Gibbs sampler. However, according to wikipedia https://en.wikipedia.org/wiki/Gibbs_sampling and this discussion thread http://metaoptimize.com/qa/questions/7831/disadvatage-of-gibbs-sampler, it is possible for Gibbs sampling to get stuck in a local mode. When this is the case, how do people tend to resolve this issue?
I am attempting to sample a distribution in the context of bioinformatics, where the joint distribution is difficult to compute analytically. The reference paper is here http://bioinformatics.oxfordjournals.org/content/31/12/i97.long, though it might be too wordy to go through.
Any general directions will be appreciated.
Best Answer
This is certainly possible. It often happens when variables are strongly correlated.
For simplicity, consider a two-parameter model. Because Gibbs sampling alters only one variable at a time, it can only move vertically or horizontally on the Cartesian plane. It will be unable to reach regions of high posterior probability that lie diagonally from one another.
You may be able to resolve this problem by reparameterizing your model so the posterior is roughly spherical or by jointly updated blocks of correlated parameters. For an example of how to reparameterize, you could take a regression model and standardize the covariates. This makes is so that the slope won't have much effect on the fit of the intercept and vice versa. Without standardizing, data may sit far from the origin, and a small change in slope could demand a huge change in intercept for the regression line to fall across the data.
MCMC implementation is difficult and error prone, so you may want to consider tools such as STAN. STAN provides already-implemented state-of-the-art MCMC methods via a language similar in spirit to R's formula syntax. It has interfaces with many common scientific computing languages. (I am not affiliated with STAN — merely a believer.)