Are Random Forest and Boosting parametric or non-parametric?

By reading the excellent Statistical modeling: The two cultures (Breiman 2001), we can seize all the difference between traditional statistical models (e.g., linear regression) and machine learning algorithms (e.g., Bagging, Random Forest, Boosted trees…).

Breiman criticizes data models (parametric) because they are based on the assumption that the observations are generated by a known, formal model prescribed by the statistician, which may poorly emulate Nature. On the other hand, ML algos do not assume any formal model and directly learn the associations between input and output variables from the data.

I realized that Bagging/RF and Boosting, are also sort of parametric: for instance, ntree, mtry in RF, learning rate, bag fraction, tree complexity in Stochastic Gradient Boosted trees are all tuning parameters. We are also sort of estimating these parameters from the data since we’re using the data to find optimal values of these parameters.

So what’s the difference? Are RF and Boosted Trees parametric models?

Answer

Parametrical models have parameters (infering them)or assumptions regarding the data distribution, whereas RF ,neural nets or boosting trees have parameters related with the algorithm itself, but they don’t need assumptions about your data distribution or classify your data into a theoretical distribution. In fact almost all algorithms have parameters such as iterations or margin values related with optimization.

Attribution
Source : Link , Question Author : Antoine , Answer Author : D.Castro

Leave a Comment