I was asked to get a conversation started between pharmacometricians and biostatisticians on multiplicity. Before I get started, this is one of those subjects that has subtlety, which can cause reasonable people to disagree.
Assumption: Whether from an experiment or an observational study, data analysis is performed in order to infer something about the universe from the summarization of data. What does it mean to infer? I looked up some synonyms of infer on Thesaurus.com and was given the following: ascertain, assume, construe, deduce, derive, figure out, glean, interpret, presume, presuppose, reckon, speculate, surmise, believe, collect, conjecture, draw, figure, gather, induce, intuit, judge, reason, suppose, think, understand, arrive at. If you notice that most of these words imply a little bit of uncertainty. To infer means to make a guess. To infer well is to make a good guess.
Statistical inference uses probability statements such as frequentist p-values and confidence intervals or Bayesian posterior probabilities, credible intervals, or Bayes factors to try and make a good guess. Sometimes the question of interest is something like “Do A and B differ?” But it does not have to be. It could be something like “Is there a strong relationship between C and D?” Or it could be, “Will this model make a good prediction about an untested patient’s drug concentration?” Whether or not we are using statistical inference techniques or not, if we are looking at data and make some sort of conclusion then there is always uncertainty with this conclusion.
It makes sense then, that the more questions I try to answer, the higher the likelihood that at least one of my answers is incorrect. This is multiplicity.
That is, if we are in the business of taking data and inferring something about the universe, then we are going to make incorrect inferences. We can try to control this (either formally or informally) by using appropriate statistical inference techniques. (What I mean by formally is through multiplicity adjustment techniques.)
How should a statistician, pharmacometrician, and/or clinician think about multiplicity? I think it depends on the situation, but the two most important things is 1) to be aware that it exists and 2) truly understand what your chosen statistical inference techniques actually mean. There are too many otherwise brilliant scientists that do not truly understand the inference technique that is being used and thus cannot properly weigh the evidence being presented. After you take stock of these two things, then what you do about multiplicity depends on the situation. Sometimes one can and should control the overall type 1 error for the multiple endpoints that one is interested. This, I would think, would be most useful when you are in the “confirm” phase (stealing from Sheiner). If you are in the “learn” phase there is less need for this, but do not assume that you still do not have to be concerned about multiplicity. It is everywhere.
I am truly interested in any of your thoughts on the subject. Feel free to shred me (I can take it), or expand on this if you would like.