In a recent paper, I fitted a three-way fixed effects model. Since one of the factors wasn’t significant (p > 0.1), I removed it and refitted the model with two fixed effects and an interaction.

I’ve just had referees comments back, to quote:

That time was not a significant factor

in the 3-way ANOVA is not of itself a

sufficient criterion for pooling the

time factor: the standard text on this

issue, Underwood 1997, argues that the

p-value for a non-significant effect

must be greater than 0.25 before

treatment levels of a factor can be

pooled. The authors should give the

relevant p-value here, and justify

their pooling with reference to

Underwood 1997.My questions are:

- I’ve never heard of the 0.25 rule. Has anyone else? I can understand not removing the factor if the p-value was close to the cut-off, but to have a “rule” seems a bit extreme.
- This referee states that Underwood 1997 is the standard text. Is it really? I’ve never heard of it. What would be the standard text (does such a thing exist)? Unfortunately, I don’t have access to this Underwood, 1997.
- Any advice when responding to the referees.

Background: this paper was submitted to a non-statistical journal. When fitting the three-way model I checked for interaction effects.

**Answer**

I’m guessing the Underwood in question is *Experiments in Ecology* (Cambridge Press 1991). Its a more-or-less standard reference in the ecological sciences, perhaps third behind Zar and Sohkol and Rohlf (and in my opinion the most ‘readable’ of the three)

If you can find a copy, the relevant section your referee is citing is in 9.7 on p.273. There Underwood suggests a recommended pooling procedure (so not a ‘rule’ *per se*) for non-significant factors. Its a 2-step procedure that frankly I don’t quite understand, but the upshot is the p = 0.25 is suggested to reduce the probability of Type I error when pooling the non-significant factor (so nothing to do with ‘time’ in your example, it could be any non-sig factor).

The procedure doesn’t actually appear to be Underwood’s, he himself cites Winer et al 1991 (*Statistical Procedures in Experimental Design* McGraw-Hill). You might try there if you can’t find a copy of Underwood.

**Attribution***Source : Link , Question Author : csgillespie , Answer Author : Chris*