Why is backward elimination justified when doing multiple regression?

Does it not result in over-fitting? Would my results be more reliable if I added a jack-knife or bootstrap procedure as a part of the analysis?

Answer

I think building a model and testing it are different things. The backward elimination is part of model building. Jack knife and bootstrap are more used to test it.

You can certainly have more reliable estimates with bootstrap and jack knife than the simple backward eleimination. But if you really want to test overfitting, the ultimate test is a split-sample, train on some, test on others. Leave-one-out is too unstable/unreliable for this purpose:
http://www.russpoldrack.org/2012/12/the-perils-of-leave-one-out.html

I think at least 10% of subjects need to be out to get more stable estimates of robustness of the model. And if you have 20 subjects, 2 subjects are still very few. But then the question becomes whether you have a large enough sample to build a model that can be applied to the rest of the population.

Hope it answered your question at least in part.

Attribution
Source : Link , Question Author : sim , Answer Author : Dorian P

Leave a Comment