What does it mean for a study to be over-powered?

What does it mean for a study to be over-powered?

My impression is that it means that your sample sizes are so large that you have the power to detect minuscule effect sizes. These effect sizes are perhaps so small that they are more likely to result from slight biases in the sampling process than a (not necessarily direct) causal connection between the variables.

Is this the correct intuition? If so, I don’t see what the big deal is, as long as the results are interpreted in that light and you manually check and see whether the estimated effect size is large enough to be “meaningful” or not.

Am I missing something? Is there a better recommendation as to what to do in this scenario?

Answer

I think that your interpretation is incorrect.

You say “These effect sizes are perhaps so small as are more likely result from slight biases in the sampling process than a (not necessarily direct) causal connection between the variables” which seems to imply that the P value in an ‘over-powered’ study is not the same sort of thing as a P value from a ‘properly’ powered study. That is wrong. In both cases the P value is the probability of obtaining data as extreme as those observed, or more extreme, if the null hypothesis is true.

If you prefer the Neyman-Pearson approach, the rate of false positive errors obtained from the ‘over-powered’ study is the same as that of a ‘properly’ powered study if the same alpha value is used for both.

The difference in interpretation that is needed is that there is a different relationship between statistical significance and scientific significance for over-powered studies. In effect, the over-powered study will give a large probability of obtaining significance even though the effect is, as you say, miniscule, and therefore of questionable importance.

As long as results from an ‘over-powered’ study are appropriately interpreted (and confidence intervals for the effect size help such an interpretation) there is no statistical problem with an ‘over-powered’ study. In that light, the only criteria by which a study can actually be over-powered are the ethical and resource allocation issues raised in other answers.

Attribution
Source : Link , Question Author : Frank Barry , Answer Author : Michael Lew

Leave a Comment