Is it wrong to refer to results as being “highly significant”?

Why do statisticians discourage us from referring to results as “highly significant” when the p-value is well below the conventional α-level of 0.05?

Is it really wrong to trust a result that has 99.9% chance of not being a Type I error (p=0.001) more than a result that only gives you that chance at 99% (p=0.01)?

Answer

I think there is not much wrong in saying that the results are “highly significant” (even though yes, it is a bit sloppy).

It means that if you had set a much smaller significance level α, you would still have judged the results as significant. Or, equivalently, if some of your readers have a much smaller α in mind, then they can still judge your results as significant.

Note that the significance level α is in the eye of the beholder, whereas the p-value is (with some caveats) a property of the data.

Observing p=1010 is just not the same as observing p=0.04, even though both might be called “significant” by standard conventions of your field (α=0.05). Tiny p-value means stronger evidence against the null (for those who like Fisher’s framework of hypothesis testing); it means that the confidence interval around the effect size will exclude the null value with a larger margin (for those who prefer CIs to p-values); it means that the posterior probability of the null will be smaller (for Bayesians with some prior); this is all equivalent and simply means that the findings are more convincing. See Are smaller p-values more convincing? for more discussion.

The term “highly significant” is not precise and does not need to be. It is a subjective expert judgment, similar to observing a surprisingly large effect size and calling it “huge” (or perhaps simply “very large”). There is nothing wrong with using qualitative, subjective descriptions of your data, even in the scientific writing; provided of course, that the objective quantitative analysis is presented as well.


See also some excellent comments above, +1 to @whuber, @Glen_b, and @COOLSerdash.

Attribution
Source : Link , Question Author : z8080 , Answer Author : Community

Leave a Comment