Saturday, May 27, 2017

Using prior knowledge in frequentist tests

... or Use of Bayesian posteriors with informative priors as optimal frequentist tests

Making decisions under uncertainty based on limited data is important and challenging. Decision theory provides a framework to reduce risks of decisions under uncertainty with typical frequentist test statistics being examples for controlling errors, e.g., Dudley (2003) or Rüschendorf (2014). This strong theoretical framework is mainly applicable to comparatively simple problems. For more complex problems and/or if there is only limited data, it is often not clear how to apply the strong framework to the actual problem at hand (e.g., Altonji, 1996)


Using prior knowledge in frequentist tests, figshare, doi.org/10.6084/m9.figshare.4819597.v3 proposes a new approach to apply decision theory in a frequentist setting to these more challenging cases.

Key elements are   
  • An efficient integration method for repeated calculation of statistical integrals for a set of hypotheses (e.g., p-values, confidence intervals) using importance sampling
  • Introduction of pointwise mutual as an efficient test statistics that was shown to be optimal under certain conditions.
  • Eliminating the need for complex minimax optimizations by demonstrating that priors may be used to derive loss functions, rather than searching for optimal priors given the loss function.

The proposed approach has a few fringe benefits:
  • The proposed test statistics (point wise mutual information) can be used for frequentist and Bayesian inference and results in consistent confidence intervals and credible intervals.
  • The proposed approach is entirely based on the Likelihood function  
  • It includes a generic proposal on how to use prior information for frequentist tests.

Still missing is an efficient implementation of marginalization.


Here, is a collection of related work:

Starting with a frequentist question ending up in Bayesian predictive checks
  • Gelman, A., Meng, X. L., & Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica sinica, 733-760.

Frequentist accuracy of Bayesian estimates
  • Efron, B. (2015). Frequentist accuracy of Bayesian estimates. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 77(3), 617-646.

Using Bayesian prior information to optimize the power of  frequentist tests
  • Dobriban, E., Fortney, K., Kim, S. K., & Owen, A. B. (2015). Optimal multiple testing under a Gaussian prior on the effect sizes. Biometrika, 102(4), 753-766.
  • Simon, N., & Simon, R. (2017). Using Bayesian modeling in frequentist adaptive enrichment designs. Biostatistics.

Minimax optimization to construct confidence regions of optimal expected size
  • Schafer, C. M., & Stark, P. B. (2009). Constructing confidence regions of optimal expected size. Journal of the American Statistical Association, 104(487), 1080-1089.

Establish pointwise mutual information as optimal criterion for Bayesian set selection
  • Evans, M. (2016). Measuring statistical evidence using relative belief. Computational and structural biotechnology journal, 14, 91-96.

Limitations of frequentist methods:
  • Altonji, J.G. and Segal, L.M., 1996. Small-sample bias in GMM estimation of covariance structures. Journal of Business & Economic Statistics, 14(3), pp.353-366.

Hypothesis testing with guaranteed p-Value for complex models in the limit of small sample sizes:
  • Vexler, A., Wu, C., & Yu, K. F. (2010). Optimal hypothesis testing: from semi to fully Bayes factors. Metrika, 71(2), 125-138.

Sunday, July 10, 2016

Expanding the reach of decision theoretic, frequentist methods

Making good decisions under uncertainty based on limited data is highly important but remains challenging.

We have decision theory that provides a framework to reduce risks of decisions under uncertainty with typical frequentist test statistics being examples for controlling errors in absence of prior knowledge. This strong theoretical framework is mainly applicable to comparatively simple problems. For non-trivial models and/or if there is only limited data, it is often not clear how to use the decision theoretical framework.
In practice, careful iterative model building and checking seems to be the best what can be done - be it using Bayesian methods or applying “frequentist” approaches (here, in this particular context, “frequentist” is often used as implying “based on minimization”).

Trying to expand the armory for decision making under uncertainty with complex models, I’m focusing on trying to expand the reach of decision theoretic, frequentist methods. Perhaps at one point in the future, it will be become possible to bridge the existing, good pragmatic approaches into the decision theoretical framework.
So far,
  • Evaluation of an efficient integration method for repeated calculation of statistical integrals (e.g., p-values) for a set of of hypotheses. Key to the method was the use of importance sampling. More details at: https://dx.doi.org/10.6084/m9.figshare.1054694.v5
  • Proposal of pointwise mutual information as an efficient test statistics that is optimal under certain considerations. The commonly used alternative would be the likelihood-ratio test, which, in the limit where asymptotics are not valid, is annoyingly inefficient since it requires repeated minimizations based on randomly generated data. More details at: https://dx.doi.org/10.6084/m9.figshare.1528163.v5
More work is required, in particular,
  • Dealing with nuisance parameters
  • Including prior information

Working on these aspects, I would appreciate feedback on what exists so far, in general, and on the proposal of using the pointwise mutual information as test statistics, in particular.

Saturday, January 16, 2016


Reproducibility of statistical test results

This is a short, simple exercise to assess the reproducibility of decisions based on statistical testing.



Consider a null hypothesis H0 with a set of alternative hypotheses containing H1 and H2. Setup the statistical hypothesis test procedure at a significance level of 0.05 to have a power of 0.8, if H1 is true. Further assume that the power for H2 is 0.5. To assess reproducibility of test result, consider the experiment of executing the test procedure two times.

Starting with the situation, where H0 is true, the probabilities for the outcomes of the joint experiment are displayed in Table 1. The probability of not being able to reproduce decisions is 0.095.

Table 1. Frequencies, if Ho is true


Frequency of decision
Reject H0
Retain H0
Reject H0
0.0025
0.0475
Retain H0
0.0475
0.9025
The frequencies change as the true state of nature changes. Assuming H1 is true, Ho can be rejected as designed with a power of 0.8. The resulting frequencies for the different outcomes of the joint experiment are displayed in Table 2. The probability of not being able to reproduce decisions is 0.32.

Table 2. Frequencies, if H1 is true

Frequency of decision
Reject H0
Retain H0
Reject H0
0.64
0.16
Retain H0
0.16
0.04
Assuming H2 is true, H0 will be rejected with a probability of 0.5. The resulting frequencies for the different outcomes of the joint experiment are displayed in Table 3. The probability of not being able to reproduce decisions is 0.5.

Table 3. Frequencies, if H2 is true

Frequency of decision
Reject H0
Retain H0
Reject H0
0.25
0.25
Retain H0
0.25
0.25
The test procedure was designed to control type I errors (the rejection of the null hypothesis even though it is true) with a probability of 0.05 and limit type II errors (no rejection of the null hypothesis even though it is wrong and H1 is true) to 0.2. For both cases, with either H0 orH1 assumed to be true, this leads to non-negligible frequencies, 0.095 and 0.32, respectively, of “non-reproducible”, “contradictory” decisions, if the same experiment is repeated twice. The situation gets worse with a frequency up to 0.5 for “non-reproducible”, “contradictory” decisions, if the true state of nature is between the null- and the alternative hypothesis used to design the experiment. The situation can also get better - if type I errors are controlled more strictly, or if the true state of the nature is far away from the null, such that the power to reject the null is close to 1.

Tuesday, September 1, 2015

Generic and consistent confidence and credible regions

Generic and consistent confidence and credible regions. 

http://dx.doi.org/10.6084/m9.figshare.1528163

A generic, consistent, efficient and exact method is proposed for set selection. The method is generic in that its definition and implementation uses only the likelihood function. The method is consistent in that the same criterion is used to select confidence and credible sets making the two kinds of sets consistent even though the two sets may differ since they answer different questions. The method is exact in that no approximations are used except numerical integration which can be made as exact as needed by investing computational resources. The method is efficient to the point that it makes possible confidence set determinations by numerical integration which are otherwise impractical. The method requires computational resources comparable to what is needed for a Bayesian analysis and may be more efficient than bootstrap of maximum likelihood estimates as it avoids repeated minimizations of randomly perturbed data.
Central to the proposed approach are the use of (1) reference priors (e.g., Bernardo, 2005), (2) pointwise mutual information as test statistics and (3) importance sampling to efficiently evaluate series of related statistical integrals (e.g., Schafer, 2009). These central pieces are expected to be useful to address statistical questions beyond set selection.

Saturday, January 10, 2015

Historical autocorrelations in stock prices

Autocorrelations in stock prices were revisisted using public accessible stock data from quandl and modelled with the R package gamlss. The model confirmed that autocorrelations in daily stock prices clearly existed and continue to exist, and showed that the autocorrelation patterns change over time. Implications with respect to the efficient market hypothesis are discussed.

Bartels, Christian (2015): Historical autocorrelations in stock prices. figshare.
http://dx.doi.org/10.6084/m9.figshare.1287224



Friday, December 5, 2014

Beyond Monte Carlo: Solve high dimensional integrals efficiently

Adaptive umbrella sampling and incremental umbrella sampling are stochastic sampling methods to efficiently solve high dimensional integrals.
  • Directed: Focus on sampling of important states, i.e., states with large contributions to the intrgral that have so far been sampled comparatively little.
  • Incremental: Additional samples of states are combined with previous samples to form an augmented set that provides an improved estimate of the integral.
  • High dimensional integrals: Enhancement of stochastic integration method such as Monte Carlo or Metropolis Monte Carlo integration methods, which are particularly suited to solve high dimensional integrals.
  • Example applications: value at risk in finance; normalizing constants in Bayesian statistics; free energy differences in statistical physics, ...
For more information contact incremental.sampling@gmail.com 

Sunday, June 15, 2014

Judicious Bayesian Analysis to Get Frequentist Confidence Intervals


An algorithm has been proposed(1) to do an analysis of observed data which may be characterized as doing a judicious Bayesian analysis of the data resulting in the determination of exact frequentist p-values and confidence intervals. The judicious Bayesian analysis comprises the steps which one would or should do anyway:
  • Bayesian sampling of parameters given the data, e.g., using Stan
  • Simulation of new data given the sampled parameters
  • Comparison of the simulations with actually observed data
Using frequentist concepts to do the comparison of simulations with observations, one obtains frequentist p-values and confidence intervals. The frequentist p-values and confidence intervals are exact in the limit of investing sufficient computational time. This holds true independent of the probability model used, and independent of whether the observed data consists of a few or many observations. As such the algorithm is a valid if not superior alternative to bootstrap sampling of frequentis parameter estimates.
In the evaluation of the proposed algorithm, it has also been investigated in how far Bayesian estimates may be used as a frequentist test procedure. It has been shown that this is feasible, simple and results are comparable to those obtained with likelihood-ratio tests.

(1) Bartels, Christian (2014): Efficient generic integration algorithm to determine confidence intervals and p-values for hypothesis testing. figshare.

http://dx.doi.org/10.6084/m9.figshare.1054694