Home
Finishing Parameter Fitting: Uncertainty on the Prediction
We concluded the previous post by stating that \(\sigma_a\) and \(\sigma_b\) don’t fully express the uncertainty of the fit, because of the nonzero covariance. But this merely raises the new question of how does one fully express uncertainty? Listing an entire covariance matrix is certainly an option, but it’s clunky if the number of parameters is large, and it’s not very intuitive.
A graphical option is to translate the uncertainty on fit parameters into uncertainty on the best fit line. These can be represented as “error bands” on a plot, which enclose all the lines which could have been drawn consistent with the posterior distribution. An example is shown in the interactive applet below.
How do we compute which lines are consistent with the posterior? In the previous post, we found that the posterior is a bivariate-Gaussian with mean given by the best fit parameters \(\hat a\) and \(\hat b\) and covariance matrix
The next task is to convert this region of consistent best fit parameters to a band of consistent lines by finding what are the maximum and minimum values of \(y\) that can be achieved by these parameters. This can be done with the method of Lagrange multipliers. That is, we set
This equation is saying that \(y\)-values consistent with the posterior are the best fit line \(\hat y\) plus the square root of a hyperbola which grows for large \(|x|\). As we would expect, the uncertainty on the best fit line increases far from the data. One can confirm this intuition by calculating that the band is narrowest at \(x = \langle x \rangle\).
Far from the data, the \(x^2\) term dominates so that \(y\) looks line a line. The slope of this line is \(\hat a \pm \sqrt{2.3\Sigma_{11}} = \hat a \pm \sqrt{2.3}\sigma / (\sigma_x\sqrt{N}).\) This is proportional to our conclusion in the previous section that the uncertainty on the slope is \(\sigma_a = \sigma/ (\sigma_x\sqrt{N})\).2 Likewise, the value of the line at the \(y\)-intercept is given by \(b = \hat b \pm \sqrt{2.3\Sigma_{22}} = \hat b \pm \sqrt{2.3}\sigma \sqrt{\langle x^2 \rangle} / (\sigma_x\sqrt{N})\), again proportional to our uncertainty for \(b\).
This exercise of finding uncertainties on the best fit line therefore reproduces the information we found in the previous post, but it does so in a more intuitive and graphical way which also takes into account the covariance between the best fit parameters. An applet combining all the techniques we’ve applied so far to linear fits is provided at the beginning of the post. By clicking the randomize button and moving the sliders, you can confirm that our best fit solution is valid.
In these posts on linear fitting, we assumed a Gaussian likelihood with a linear model and equal uncertainty \(\sigma\) across the data. If any of these assumptions are violated, our results do not hold and an analytical solution may not even be possible. In particular, most scientific models are nonlinear, and even those that are likely have uneven error bars. But even in cases where these assumptions are violated, the fundamental Bayesian principles still hold. In the next post, we discuss the general case of a Gaussian likelihood and make no assumptions on the model or the uncertainties.