t-statistic for linear regression

I know how to calculate t-statistics and p-values for linear regression, but I’m trying to understand a step in the derivation. I understand where Student’s t-distribution comes from, namely I can show that if Z is a random variable drawn from a standard normal distribution Z \sim \mathcal{N}(0,1) and if \chi is drawn from a \chi^2_k distribution, then the new random variable

T = \frac{Z}{\sqrt{\frac{\chi^2}{k}}}

will be drawn from a t-distribution with k degrees of freedom.

The part that confuses me is the application of this general formula to the estimates for the coefficients of a simple linear regression. If I parametrize the linear model as y = \alpha + \beta x + \varepsilon, with \varepsilon a random variable with zero mean characterizing the errors, then the best estimate for \beta is

\hat{\beta} = \frac{\bar{x y} – \bar{x} \bar{y} }{\bar{x^2} – \bar{x}^2 } \, ,

where the bar’s indicate sample means. The standard error of \hat{\beta} is

SE(\hat{\beta})^2 = \frac{\sigma^2}{n \left(\bar{x^2} – \bar{x}^2 \right) } \,.

Here \sigma = \sqrt{\operatorname{Var}(\varepsilon)}. The part I am confused about is why

t \equiv \frac{\hat{\beta}}{SE(\hat{\beta})}

is taken to be drawn from a t-distribution, assuming the null hypothesis. If I could write t in the form of the above variable T, cleanly identifying the Z and the \chi variables, then everything would be clear.

Answer


Z = \frac{\widehat\beta – \beta}{\left( \dfrac \sigma { \sqrt{ n \left( \,\overline{(x^2)} – (\,\overline{x}\,\right)^2}} \right)} \sim \mathrm{N}(0,1).

And

(n-2) \frac{\widehat{\sigma}^2}{\sigma^2} \sim \chi^2_{n-2}.

Notice that \sigma appears in both the numerator and the denominator of Z/\sqrt{\chi^2_k/k} and cancels out.

Independence of these two things is seen by observing that the vector of residuals is independent of the vector of fitted values. To see that, find the covariance between the vector of residuals and the vector of fitted values, and recall that if two random vectors are jointly normally distributed then they are independent if they are uncorrelated.

(The whole story of why these things have the distributions asserted here would take somewhat longer.)

Attribution
Source : Link , Question Author : Surgical Commander , Answer Author : Michael Hardy

Leave a Comment