Physicist Amok

Here’s a reasonable definition of the mathematical field of analysis: the methods and tools for solving problems by breaking them into infinitesimal subproblems, then reassembling the resulting subsolutions. It suffers from the same problem as Pound describes of a biography of Stravinsky: if you know analysis it is clarifying. If you don’t, you are as much in the dark as before. The definition tells you nothing about the path from the construction of the real numbers to the complex numbers, convergence of sequences, through to Lebesgue integration, and then off into the depths of probability.

I offer a similar definition of statistics, which I shall probably rue in years to come: statistics is the methods and tools for taking experimental data and drawing from it logically defensible and experimentally repeatable conclusions.

Early in his Data Analysis and Regression, Tukey offers three levels of statistical analysis which form the same role as the algorithm of decompose, solve, and compose in analysis.

First comes pure indication, consisting only of things we summarize from the data: the values from a repeated measurement are centered around 5; there is a kink in the curve about there; the proportion of relapses under drug regimen B is 8% lower than under regimen A.

Next we make some determination of the strength of the indications. For instance, we might supplement of the central value of a set of measurements by a measure of its spread, or give the range of values in which the curve kinks over several repetitions of the experiment.

Finally we make a formal inference from the data. Given our data and some mathematical model for how it was generated, choose one of a set of decisions based upon the data. The classic example is Student’s -test, which maps a set of data to a distribution over possible means of a Gaussian which generated that data.

Tukey points out that we often can’t get all the way to inference. The experiments may be prohibitively difficult. The mathematical techniques may not exist yet. Indications or determinations without inference are a fact of life, and should be handled with care and respect. All too often I see papers with inappropriate attempts at inference from what is only indication. Almost any use of microarrays in biology gives an example of this.

For indication, the problem seems deceptively simple: how do we find indications that may be deeply buried in a data set? This is the focus of the field of exploratory data analysis. In determination, we seek tools to tell us how strong the indications are. If we treat the suggestion of a mathematical model for inference as an indication, then these same tools give sanity checks on our models. The ideas of robustness and stability also enter here. The ecological equivalent in analysis is the toolbox of convergence tests and tricks making up old style "advanced calculus" courses. There is no general theory, no perfect approach. There are conventions and desirable qualities such as robustness to departures from the ideal and stability under perturbations to the data set.

Inference occupies a similar place in statistics to probability’s place in analysis: an axiomatic construction, which provides a framework, but must be tuned and specialized for each particular case. Rather like a matroishka doll, it is also formulated in terms of probability.