How important is basis expansion for deep nets?

If deep neural nets are considered to be universal function approximators, is basis expansion really necessary? Or would this be case-specific? For example, if one has three quantitative X variables, would there be any advantage in expanding the number of variables by introducing interactions, polynomials, etc.? This seems to have good utility in e.g. RFs and SVM, but I’m unsure of whether this would be a good strategy for neural nets.

If this is perhaps too broad or vague, could someone point me to some pertinent information on basis expansion and feature engineering in the context of deep nets?

Answer

The idea of deep neural network is it can do the feature engineering automatically for us. (See the first chapter of the deep learning book.) I would strongly recommend you to read the first chapter.

Doing basis expansion is not really necessary and uncommonly used. Keep in mind that, the deep net usually takes raw features as inputs, for images that have (at least) thousands of pixels, it is also not possible to do the basis expansion (say higher order polynomial expansion) effectively before feeding to the neural network.


In fact, there are some operations in deep neural network can be viewed as basis expansion.

Attribution
Source : Link , Question Author : srhoades10 , Answer Author : Haitao Du

Leave a Comment