What is an example of perfect multicollinearity?

What is an example of perfect collinearity in terms of the design matrix X?

I would like an example where ˆβ=(XX)1XY can’t be estimated because (XX) is not invertible.

Answer

Here is an example with 3 variables, y, x1 and x2, related by the equation

y=x1+x2+ε

where εN(0,1)

The particular data are

         y x1 x2
1 4.520866  1  2
2 6.849811  2  4
3 6.539804  3  6

So it is evident that x2 is a multiple of x1 hence we have perfect collinearity.

We can write the model as

Y=Xβ+ε

where:

Y=[4.526.856.54]

X=[112124136]

So we have

XX=[112124136][111123246]=[61116112131163146]

Now we calculate the determinant of XX :

det

In R we can show this as follows:

> x1 <- c(1,2,3)

create x2, a multiple of x1

> x2 <- x1*2

create y, a linear combination of x1, x2 and some randomness

> y <- x1 + x2 + rnorm(3,0,1)

observe that

> summary(m0 <- lm(y~x1+x2))

fails to estimate a value for the x2 coefficient:

Coefficients: (1 not defined because of singularities)
            Estimate Std. Error t value Pr(>|t|)
(Intercept)   3.9512     1.6457   2.401    0.251
x1            1.0095     0.7618   1.325    0.412
x2                NA         NA      NA       NA

Residual standard error: 0.02583 on 1 degrees of freedom
Multiple R-squared:      1,     Adjusted R-squared:  0.9999 
F-statistic: 2.981e+04 on 1 and 1 DF,  p-value: 0.003687

The model matrix X is:

> (X <- model.matrix(m0))

(Intercept) x1 x2
1           1  1  2
2           1  2  4
3           1  3  6

So XX’ is

> (XXdash <- X %*% t(X))
   1  2  3
1  6 11 16
2 11 21 31
3 16 31 46

which is not invertible, as shown by

> solve(XXdash)
Error in solve.default(XXdash) : 
  Lapack routine dgesv: system is exactly singular: U[3,3] = 0

Or:

> det(XXdash)
[1] 0

Attribution
Source : Link , Question Author : TsTeaTime , Answer Author : Robert Long

Leave a Comment