Choice of penalty parameter for lasso with logit link function?

I need to estimate a logistic regression that is penalized by the L1-norm. Choosing the penalty parameter by cross-validation is too computationally expensive for my application. I was wondering whether there was a simple formula I can use? I am aware that simple formulas exist in the linear case but do not know of one … Read more

Decorrelating Systems of Random Variables

Suppose we have a dataset that includes dozens of attributes, that are all correlated with each-other. I would like to better understand which variables affect which other variables in a causal way, preferably as a system of regressions. The classical approach would be to setup a structured model, identifying manually which attributes are endogenous and … Read more

Lasso as feature selection on its own, e.g. before random forest classification

Could lasso be used by itself as a feature selection method without being tied to a regression loss function? And if so, what’s a good package for doing so? Also, could lasso be performed as a step before using a classifier like random forest? Answer AttributionSource : Link , Question Author : casualprogrammer , Answer … Read more

small data set and number of independent variables is larger than number of observations . What to do ? Should I apply Lasso?

In my data set I have only 6 observations & independent variables are 9. Can I use Lasso regression or multiple regression in this situation? when independent variables are > observations? Data : y x1 x2 x3 x4 x5 x6 x7 x8 x9 6142.8 90.25 164.19 15 0.91 0.88 2.99 0.5 7.255 15 8174.2 126.9 … Read more

is it possible to run elastic net (lasso) on as few as 8 samples? R

I have 8 samples with outcome 0 or 1 (4 samples of each) and I have 3100 variables, is it possible to run elastic net on these using an alpha of 1 (I just want to use lasso)? And is it possible to do in R? Answer AttributionSource : Link , Question Author : user3324491 … Read more

Poor regression results with LASSO which improves after variable selection – could someone shed some light on the observations?

This is my first post here – I’m not a statistician by training though I have a machine learning background, so please correct any erroneous usage of statistical terminology if you see any. Currently, I have a (linear) regression problem I’m trying to solve – ~5000 predictors and ~200 examples which is an extremely ill-posed … Read more

What is the relation between coefficients selected by an Oracle-procedure and significant coefficients?

As proved by Zou (2006), the adaptive LASSO enjoys the Oracle Property in the sense that it does variable selection as if it knew the true (sparse) model. However, in the early days, the standard hypothesis testing in e.g. OLS should also point in the direction of which variables to leave out in the model … Read more

Can someone explain how to solve L1 regularized logistic regression step-by-step?

All those online courses that mention logistic regression with regularization explain the L2 regularization. This makes sense because L2 has a derivation-based analytical solution, which leads to, in updating the coefficients, just reducing them by a factor (proportional to the penalty parameter), and it is very intuitive. However, I need to figure out how to … Read more

Request for references: Best way to estimate directed graphs *with* cycles

I have $n$ observations of $p$ random variables $\{Y_1,\dots,Y_p \}$. I know that each observation has been generated from a linear network model of the form $$ Y = WY+\epsilon $$ and that the graph described by $W$ is likely to be sparse and to have cycles. What is the current state of the art … Read more

lambda.min in lasso for correlated variable selection

Lasso uses cross-validation to determine both the number of included predictors and the degree of shrinkage to avoid over-fitting. I have used the glmnet package to do this. fit=glmnet(x,y) cvob1= cv.glmnet(x,y) lambda1=cvob1$lambda.min cvob2=cv.glmnet(x,y) lambda2=cvob2$lambda.min There is a high possibility that lambda1 and lambda2 are different. I need to get a proper lambda.min for the next … Read more