Assume X and Y have finite second moment. In the Hilbert space of random variables with second finite moment (with inner product of T1,T2 defined by E(T1T2), ||T||2=E(T2)), we may interpret E(Y|X) as the projection of Y onto the space of functions of X.
We also know that Law of Total Variance reads
Is there a way to interpret this law in terms of the geometric picture above? I have been told that the law is the same as Pythagorean Theorem for the right-angled triangle with sides Y,E(Y|X),Y−E(Y|X). I understand why the triangle is right-angled, but not how the Pythagorean Theorem is capturing the Law of Total Variance.
I assume that you are comfortable with regarding the right-angled triangle as meaning that E[Y∣X] and Y−E[Y∣X] are uncorrelated random variables.
For uncorrelated random variables A and B,
and so if we set A=Y−E[Y∣X] and B=E[Y∣X] so that A+B=Y, we get
It remains to show that var(Y−E[Y∣X]) is the same as
E[var(Y∣X)] so that we can re-state (2) as
which is the total variance formula.
It is well-known that the expected value of the random variable E[Y∣X] isE[Y],
that is, E[E[Y∣X]]=E[Y]. So we see that
from which it follows that var(A)=E[A2], that is,
Let C denote the random variable (Y−E[Y∣X])2 so that we can
write that var(Y−E[Y∣X])=E[C].
Now, given that X=x, the conditional distribution of Y has mean E[Y∣X=x]
In other words, E[C∣X=x]=var(Y∣X=x) so that
the random variable E[C∣X] is just var(Y∣X).
which upon substitution into (5) shows that
This makes the right side of (2) exactly what we need and so we have proved
the total variance formula (3).