Sunday, July 10, 2016

Expanding the reach of decision theoretic, frequentist methods

Making good decisions under uncertainty based on limited data is highly important but remains challenging.

We have decision theory that provides a framework to reduce risks of decisions under uncertainty with typical frequentist test statistics being examples for controlling errors in absence of prior knowledge. This strong theoretical framework is mainly applicable to comparatively simple problems. For non-trivial models and/or if there is only limited data, it is often not clear how to use the decision theoretical framework.
In practice, careful iterative model building and checking seems to be the best what can be done - be it using Bayesian methods or applying “frequentist” approaches (here, in this particular context, “frequentist” is often used as implying “based on minimization”).

Trying to expand the armory for decision making under uncertainty with complex models, I’m focusing on trying to expand the reach of decision theoretic, frequentist methods. Perhaps at one point in the future, it will be become possible to bridge the existing, good pragmatic approaches into the decision theoretical framework.
So far,
  • Evaluation of an efficient integration method for repeated calculation of statistical integrals (e.g., p-values) for a set of of hypotheses. Key to the method was the use of importance sampling. More details at: https://dx.doi.org/10.6084/m9.figshare.1054694.v5
  • Proposal of pointwise mutual information as an efficient test statistics that is optimal under certain considerations. The commonly used alternative would be the likelihood-ratio test, which, in the limit where asymptotics are not valid, is annoyingly inefficient since it requires repeated minimizations based on randomly generated data. More details at: https://dx.doi.org/10.6084/m9.figshare.1528163.v5
More work is required, in particular,
  • Dealing with nuisance parameters
  • Including prior information

Working on these aspects, I would appreciate feedback on what exists so far, in general, and on the proposal of using the pointwise mutual information as test statistics, in particular.

No comments:

Post a Comment