Why don’t people use deeper RBFs or RBF in combination with MLP?

So when looking at Radial Basis Function Neural Networks, I’ve noticed that people only ever recommend the usage of 1 hidden layer, whereas with multilayer perceptron neural networks more layers is considered better.

Given that RBF networks can be trained with version of back propagation is there any reasons why deeper RBF networks wouldn’t work, or that an RBF layer couldn’t be used as the penultimate or first layer in a deep MLP network? (I was thinking the penultimate layer so it could essentially be trained on the features learned by the previous MLP layers)

Answer

The fundamental problem is that RBFs are
a) too nonlinear,
b) do not do dimension reduction.

because of a) RBFs were always trained by k-means rather than gradient descent.

I would claim that the main success in Deep NNs is conv nets, where one of the key parts is dimension reduction: although working with say 128x128x3=50,000 inputs, each neuron has a restricted receptive field, and there are much fewer neurons in each layer.In a given layer in an MLP- each neuron represents a feature/dimension) so you are constantly reducing dimensionality (in going from layer to layer).

Although one could make the RBF covariance matrix adaptive and so do dimension reduction, this makes it even harder to train.

Attribution
Source : Link , Question Author : user1646196 , Answer Author : seanv507

Leave a Comment