Problems with one-hot encoding vs. dummy encoding

I am aware of the fact that categorical variables with k levels should be encoded with k-1 variables in dummy encoding (similarly for multi-valued categorical variables). I was wondering how much of a problem does a one-hot encoding (i.e. using k variables instead) over dummy encoding for different regression methods, mainly linear regression, penalized linear regression (Lasso, Ridge, ElasticNet), tree-based (random forests, gradient boosting machines).

I know that in linear regression, multi-collinearity problems occur (even though in practice I have fitted linear regression using OHE without any issues).

However, does dummy encoding need to be used in all of them and how wrong would the results be if one uses one-hot encoding?

My focus is on prediction in regression models with multiple (high-cardinality) categorical variables, so I am not interested in confidence intervals.

The issue with representing a categorical variable that has $k$ levels with $k$ variables in regression is that, if the model also has a constant term, then the terms will be linearly dependent and hence the model will be unidentifiable. For example, if the model is $μ = a_0 + a_1X_1 + a_2X_2$ and $X_2 = 1 - X_1$, then any choice $(β_0, β_1, β_2)$ of the parameter vector is indistinguishable from $(β_0 + β_2,\; β_1 - β_2,\; 0)$. So although software may be willing to give you estimates for these parameters, they aren’t uniquely determined and hence probably won’t be very useful.