Variance-covariance structure for random-effects in lme4

The default variance-covariance structure is unstructured — that is, the only constraint on the variance-covariance matrix for an vector random effect with $n$ levels is that is positive definite. Separate random effects terms are considered independent, however, so if you want to fit (e.g.) a model with random intercept and slope where the intercept and slope are uncorrelated (not necessarily a good idea), you can use the formula (1|g) + (0+x|g), where g is the grouping factor; the 0 in the second term suppresses the intercept. If you want to fit independent parameters of a categorical variable (again, possibly questionable), you probably need to construct numeric dummy variables by hand. You can, sort of, construct a compound-symmetric variance-covariance structure (although with non-negative covariances only) by treating the factor as a nested grouping variable. For example, if f is a factor, then (1|g/f) will assume equal correlations among the levels of f.

For other/more complex variance-covariance structures, your choices (in R) are to (1) use nlme (which has the pdMatrix constructors to allow more flexibility); (2) use MCMCglmm (which offers a variety of structures including unstructured, compound symmetric, identity with different variances, or identity with homogeneous variances); (3) use a special-purpose package such as pedigreemm that constructs a special structured matrix. There is a flexLambda branch on github that eventually hopes to provide more capabilities in this direction.