What are some of the main pitfalls of using linear mixed-effects models? What are the most important things to test/watch out for in assessing the appropriateness of your model? When comparing models of the same dataset, what are the most important things to look for?
This is a good question.
Here are some common pitfalls:
Using standard likelihood theory, we may derive a test to compare two nested
hypotheses, $H_0$ and $H_1$, by computing the likelihood ratio test statistic. The null distribution of this test statistic is approximately chi-squared with degrees of freedom equal to difference in the dimensions of the two parameters spaces. Unfortunately, this test is only approximate and requires several assumptions. One crucial assumption is that the parameters under the null are not on the boundary of the parameter space. Since we are often interested in testing hypotheses about the random effects that take the form:
This a real concern. The way to get around this problem is using REML. But still, the p-values will tend to be larger than they should be. This means that if you observe a significant effect using the χ2 approximation, you can be fairly confident that it is actually significant. Small, but not significant, p-values might spur one to use more accurate, but time-consuming, bootstrap methods.
Comparing fixed effects: If you plan to use the likelihood ratio test to compare two nested models that differ only in their fixed effects, you cannot use the REML estimation method. The reason is that REML estimates the random effects by considering linear combinations of the data that remove the fixed effects. If these fixed effects are changed, the likelihoods of the two models will not be directly comparable.
P-values: The p-values generated by the likelihood ratio test for fixed effects are approximate and unfortunately tend to be too small, thereby sometimes overstating the importance of some effects. We may use nonparametric bootstrap methods to find more accurate p-values for the likelihood ratio test.
There are other concerns about p-values for the fixed effects test which are highlighted by Dr. Doug Bates [here].
I am sure other members of the forum will have better answers.
Source: Extending linear models with R — Dr. Julain Faraway.