Andrew Gelman wrote an extensive article on why Bayesian AB testing doesn’t require multiple hypothesis correction: Why We (Usually) Don’t Have to Worry
About Multiple Comparisons, 2012.
I don’t quite understand: why don’t Bayesian methods require multiple testing corrections?
A ~ Distribution1 + Common Distribution B ~ Distribution2 + Common Distribution C ~ Distribution3 + Common Distribution Common Distribution ~ Normal
My understanding is that the Bayesian approach shown above accounts for the shared underlying distribution by all the hypothesis (unlike in a frequentist Bonferroni correction). Is my reasoning correct?
One odd way to answer the question is to note that the Bayesian method provides no way to do this because Bayesian methods are consistent with accepted rules of evidence and frequentist methods are often at odds with them. Examples:
- With frequentist statistics, comparing treatment A to B must penalize for comparing treatments C and D because of family-wise type I error considerations; with Bayesian the A-B comparison stands on its own.
- For sequential frequentist testing, penalties are usually required for multiple looks at the data. In a group sequential setting, an early comparison for A vs B must be penalized for a later comparison that has not been made yet, and a later comparison must be penalized for an earlier comparison even if the earlier comparison did not alter the course of the study.
The problem stems from the frequentist’s reversal of the flow of time and information, making frequentists have to consider what could have happened instead of what did happen. In contrast, Bayesian assessments anchor all assessment to the prior distribution, which calibrates evidence. For example, the prior distribution for the A-B difference calibrates all future assessments of A-B and does not have to consider C-D.
With sequential testing, there is great confusion about how to adjust point estimates when an experiment is terminated early using frequentist inference. In the Bayesian world, the prior “pulls back” on any point estimates, and the updated posterior distribution applies to inference at any time and requires no complex sample space considerations.