Home

Bayesian Model Selection

Model selection is the practice of choosing the model that best fits the data. That task may sound deceptively simple—a reasonable first guess is that the best model is the one with the highest likelihood? This idea seems to make sense because it is the natural extension of the frequentist maximum likelihood estimator (MLE), which derives best-fit parameters by maximizing the likelihood, and in fact we showed in a later post that the MLE is equivalent to a Bayesian analysis with uniform priors. So this idea finds roots in both frequentist and Bayesian analyses.

But simply picking the model with the best likelihood doesn’t work when the models have different numbers of parameters. The more parameters a model has, the better it can match the data simply by tweaking those parameters to maximize the likelihood. Thus, a high-parameter, incorrect model can have a higher likelihood than a correct, low-parameter model. We therefore need some kind of penalty for models with more parameters. Since free parameters are also called degrees of freedom (DoF), this penalty is referred to as a degree of freedom penalty.

To calculate the DoF, we should first quantitatively show that models with more parameters deliver better likelihoods.e

A Bayesian solves this problem by using not the maximum likelihood, but the probability of observing the given data \(P(D)\). \(P(D)\) is also called the evidence. Recalling that the likelihood is defined as the probability of the data given the parameters \(L(\theta) = P(D|\theta)\), by one of the basic theorems of conditional probability,

\[P(D) = \int d\theta\, \pi(\theta) L(\theta).\]
(1)
So this evidence is not the maximum of \(L(\theta)\), but instead the prior-weighted average of \(L(\theta)\). The Bayesian then defines the Bayes factor as the ratio of the evidence for the two models: \(B = P_{M_1}(D) / P_{M_0}(D)\), where \(M_1\) and \(M_0\) are the two models we wish to compare. When the Bayes factor is large, that means \(M_1\) matches the data better than \(M_0\).

The rest of this post will prove that using the evidence builds in the degree of freedom penalty we needed. In the next post, we’ll address the other important question of how to interpret a Bayes’ factor—how high should we expect a Bayes factor to be before we say \(M_1\) is proven correct accepted.

This section shows why the likelihood average, or evidence \(P(D)\) automatically includes a penalty for models with large numbers of parameters, but first we should quantitatively understand why models with more parameters deliver higher likelihoods.