# Is there any theoretical problem with averaging regression coefficients to build a model?

I want to build a regression model that is an average of multiple OLS models, each based on a subset of the full data. The idea behind this is based on this paper. I create k folds and build k OLS models, each on data without one of the folds. I then average the regression coefficients to get the final model.

This strikes me as similar to something like random forest regression, in which multiple regression trees are built and averaged. However, performance of the averaged OLS model seems worse than simply building one OLS model on the entire data. My question is: is there a theoretical reason why averaging multiple OLS models is wrong or undesirable? Can we expect averaging multiple OLS models to reduce overfitting? Below is an R example.

``````#Load and prepare data
library(MASS)
data(Boston)
trn <- Boston[1:400,]
tst <- Boston[401:nrow(Boston),]

#Create function to build k averaging OLS model
lmave <- function(formula, data, k, ...){
lmall <- lm(formula, data, ...)
folds <- cut(seq(1, nrow(data)), breaks=k, labels=FALSE)
for(i in 1:k){
tstIdx <- which(folds==i, arr.ind = TRUE)
tst <- data[tstIdx, ]
trn <- data[-tstIdx, ]
assign(paste0('lm', i), lm(formula, data = trn, ...))
}

coefs <- data.frame(lm1=numeric(length(lm1\$coefficients)))
for(i in 1:k){
coefs[, paste0('lm', i)] <- get(paste0('lm', i))\$coefficients
}
lmnames <- names(lmall\$coefficients)
lmall\$coefficients <- rowMeans(coefs)
names(lmall\$coefficients) <- lmnames
lmall\$fitted.values <- predict(lmall, data)
target <- trimws(gsub('~.*\$', '', formula))
lmall\$residuals <- data[, target] - lmall\$fitted.values

return(lmall)
}

#Build OLS model on all trn data
olsfit <- lm(medv ~ ., data=trn)

#Build model averaging five OLS
olsavefit <- lmave('medv ~ .', data=trn, k=5)

#Build random forest model
library(randomForest)
set.seed(10)
rffit <- randomForest(medv ~ ., data=trn)

#Get RMSE of predicted fits on tst
library(Metrics)
rmse(tst\$medv, predict(olsfit, tst))
 6.155792
rmse(tst\$medv, predict(olsavefit, tst))
 7.661 ##Performs worse than olsfit and rffit
rmse(tst\$medv, predict(rffit, tst))
 4.259403
``````