Mathematics and Statistics Vol. 9(4), pp. 535 - 551
DOI: 10.13189/ms.2021.090413
Reprint (PDF) (409Kb)

On the Gaussian Approximation to Bayesian Posterior Distributions

Christoph Fuhrmann ¹, Hanns-Ludwig Harney ², Klaus Harney ³, Andreas M¨uller ^4,*
1 School of Education, Institute for Educational Research, University of Wuppertal, D-42097 Wuppertal, Germany
² Max-Planck-Institute for Nuclear Physics, Postfach 103980, D-69029 Heidelberg, Germany
³ Faculty of Philosophy and Educational Science, Institute for Educational Research, Ruhr-Universitat, D-44801 Bochum, Germany
⁴ Faculty of Sciences/Physics Department and Institut Universitaire de Formation des Enseignants, University of Geneva, CH - 1211 Genève 4, Switzerland

ABSTRACT

The present article derives the minimal number N of observations needed to approximate a Bayesian posterior distribution by a Gaussian. The derivation is based on an invariance requirement for the likelihood . This requirement is defined by a Lie group that leaves the unchanged, when applied both to the observation(s) and to the parameter to be estimated. It leads, in turn, to a class of specific priors. In general, the criterion for the Gaussian approximation is found to depend on (i) the Fisher information related to the likelihood , and (ii) on the lowest non-vanishing order in the Taylor expansion of the Kullback-Leibler distance between and , where is the maximum-likelihood estimator of , given by the observations . Two examples are presented, widespread in various statistical analyses. In the first one, a chi-squared distribution, both the observations and the parameter are defined all over the real axis. In the other one, the binomial distribution, the observation is a binary number, while the parameter is defined on a finite interval of the real axis. Analytic expressions for the required minimal N are given in both cases. The necessary N is an order of magnitude larger for the chi-squared model (continuous ) than for the binomial model (binary ). The difference is traced back to symmetry properties of the likelihood function . We see considerable practical interest in our results since the normal distribution is the basis of parametric methods of applied statistics widely used in diverse areas of research (education, medicine, physics, astronomy etc.). To have an analytical criterion whether the normal distribution is applicable or not, appears relevant for practitioners in these fields.

KEYWORDS
Bayesian Posterior, Gaussian Approximation, Chi-squared and Binomial Distributions

Cite This Paper in IEEE or APA Citation Styles
(a). IEEE Format:
[1] Christoph Fuhrmann , Hanns-Ludwig Harney , Klaus Harney , Andreas M¨uller , "On the Gaussian Approximation to Bayesian Posterior Distributions," Mathematics and Statistics, Vol. 9, No. 4, pp. 535 - 551, 2021. DOI: 10.13189/ms.2021.090413.

(b). APA Format:
Christoph Fuhrmann , Hanns-Ludwig Harney , Klaus Harney , Andreas M¨uller (2021). On the Gaussian Approximation to Bayesian Posterior Distributions. Mathematics and Statistics, 9(4), 535 - 551. DOI: 10.13189/ms.2021.090413.