For many of us, Bayesian statistics is voodoo magic at best, or completely subjective nonsense at worst. Among the trademarks of the Bayesian approach, Markov chain Monte Carlo methods are especially mysterious. They’re math-heavy and computationally expensive procedures for sure, but the basic reasoning behind them, like so much else in data science, can be made intuitive. That is my goal here.
So, what are Markov chain Monte Carlo (MCMC) methods? The short answer is:
MCMC methods are used to approximate the posterior distribution of a parameter of interest by random sampling in a probabilistic space.
In this article, I will explain that short answer, without any math.