In a recent paper, I fitted a three-way fixed effects model. Since one of the factors wasn’t significant (p > 0.1), I removed it and refitted the model with two fixed effects and an interaction.
I’ve just had referees comments back, to quote:
That time was not a significant factor
in the 3-way ANOVA is not of itself a
sufficient criterion for pooling the
time factor: the standard text on this
issue, Underwood 1997, argues that the
p-value for a non-significant effect
must be greater than 0.25 before
treatment levels of a factor can be
pooled. The authors should give the
relevant p-value here, and justify
their pooling with reference to
My questions are:
- I’ve never heard of the 0.25 rule. Has anyone else? I can understand not removing the factor if the p-value was close to the cut-off, but to have a “rule” seems a bit extreme.
- This referee states that Underwood 1997 is the standard text. Is it really? I’ve never heard of it. What would be the standard text (does such a thing exist)? Unfortunately, I don’t have access to this Underwood, 1997.
- Any advice when responding to the referees.
Background: this paper was submitted to a non-statistical journal. When fitting the three-way model I checked for interaction effects.
I’m guessing the Underwood in question is Experiments in Ecology (Cambridge Press 1991). Its a more-or-less standard reference in the ecological sciences, perhaps third behind Zar and Sohkol and Rohlf (and in my opinion the most ‘readable’ of the three)
If you can find a copy, the relevant section your referee is citing is in 9.7 on p.273. There Underwood suggests a recommended pooling procedure (so not a ‘rule’ per se) for non-significant factors. Its a 2-step procedure that frankly I don’t quite understand, but the upshot is the p = 0.25 is suggested to reduce the probability of Type I error when pooling the non-significant factor (so nothing to do with ‘time’ in your example, it could be any non-sig factor).
The procedure doesn’t actually appear to be Underwood’s, he himself cites Winer et al 1991 (Statistical Procedures in Experimental Design McGraw-Hill). You might try there if you can’t find a copy of Underwood.