Saturday, May 27, 2017

Using prior knowledge in frequentist tests

... or Use of Bayesian posteriors with informative priors as optimal frequentist tests

Making decisions under uncertainty based on limited data is important and challenging. Decision theory provides a framework to reduce risks of decisions under uncertainty with typical frequentist test statistics being examples for controlling errors, e.g., Dudley (2003) or Rüschendorf (2014). This strong theoretical framework is mainly applicable to comparatively simple problems. For more complex problems and/or if there is only limited data, it is often not clear how to apply the strong framework to the actual problem at hand (e.g., Altonji, 1996)


Using prior knowledge in frequentist tests, figshare, doi.org/10.6084/m9.figshare.4819597.v3 proposes a new approach to apply decision theory in a frequentist setting to these more challenging cases.

Key elements are   
  • An efficient integration method for repeated calculation of statistical integrals for a set of hypotheses (e.g., p-values, confidence intervals) using importance sampling
  • Introduction of pointwise mutual as an efficient test statistics that was shown to be optimal under certain conditions.
  • Eliminating the need for complex minimax optimizations by demonstrating that priors may be used to derive loss functions, rather than searching for optimal priors given the loss function.

The proposed approach has a few fringe benefits:
  • The proposed test statistics (point wise mutual information) can be used for frequentist and Bayesian inference and results in consistent confidence intervals and credible intervals.
  • The proposed approach is entirely based on the Likelihood function  
  • It includes a generic proposal on how to use prior information for frequentist tests.

Still missing is an efficient implementation of marginalization.


Here, is a collection of related work:

Starting with a frequentist question ending up in Bayesian predictive checks
  • Gelman, A., Meng, X. L., & Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica sinica, 733-760.

Frequentist accuracy of Bayesian estimates
  • Efron, B. (2015). Frequentist accuracy of Bayesian estimates. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 77(3), 617-646.

Using Bayesian prior information to optimize the power of  frequentist tests
  • Dobriban, E., Fortney, K., Kim, S. K., & Owen, A. B. (2015). Optimal multiple testing under a Gaussian prior on the effect sizes. Biometrika, 102(4), 753-766.
  • Simon, N., & Simon, R. (2017). Using Bayesian modeling in frequentist adaptive enrichment designs. Biostatistics.

Minimax optimization to construct confidence regions of optimal expected size
  • Schafer, C. M., & Stark, P. B. (2009). Constructing confidence regions of optimal expected size. Journal of the American Statistical Association, 104(487), 1080-1089.

Establish pointwise mutual information as optimal criterion for Bayesian set selection
  • Evans, M. (2016). Measuring statistical evidence using relative belief. Computational and structural biotechnology journal, 14, 91-96.

Limitations of frequentist methods:
  • Altonji, J.G. and Segal, L.M., 1996. Small-sample bias in GMM estimation of covariance structures. Journal of Business & Economic Statistics, 14(3), pp.353-366.

Hypothesis testing with guaranteed p-Value for complex models in the limit of small sample sizes:
  • Vexler, A., Wu, C., & Yu, K. F. (2010). Optimal hypothesis testing: from semi to fully Bayes factors. Metrika, 71(2), 125-138.

No comments:

Post a Comment