What’s the typical range of possible values for the shrinkage parameter in penalized regression?

In lasso or ridge regression, one has to specify a shrinkage parameter, often called by \lambda or \alpha. This value is often chosen via cross validation by checking a bunch of different values on training data and seeing which yields the best e.g. R^2 on test data. What is the range of values one should check? Is it (0,1)?

Answer

You don’t really need to bother. In most packages (like glmnet) if you do not specify \lambda, the software package generates its own sequence (which is often recommended). The reason I stress this answer is that during the running of the LASSO the solver generates a sequence of \lambda, so while it may counterintuitive providing a single \lambda value may actually slow the solver down considerably (When you provide an exact parameter the solver resorts to solving a semi definite program which can be slow for reasonably ‘simple’ cases.)

As for the exact value of \lambda you can potentially chose whatever you want from [0,\infty[. Note that if your \lambda value is too large the penalty will be too large and hence none of the coefficients can be non-zero. If the penalty is too small you will overfit the model and this will not be the best cross validated solution

Attribution
Source : Link , Question Author : rhombidodecahedron , Answer Author : Sid

Leave a Comment