A while ago a user on R-help mailing list asked about the soundness of using PCA scores in a regression. The user is trying to use some PC scores to explain variation in another PC (see full discussion here). The answer was that no, this is not sound because PCs are orthogonal to each other.
Can someone explain in a bit more detail why this is so?
A principal component is a weighted linear combination of all your factors (X’s).
example: PC1 = 0.1X1 + 0.3X2
There will be one component for each factor (though in general a small number are selected).
The components are created such that they have zero correlation (are orthogonal), by design.
Therefore, component PC1 should not explain any variation in component PC2.
You may want to do regression on your Y variable and the PCA representation of your X’s, as they will not have multi-collinearity. However, this could be hard to interpret.
If you have more X’s than observations, which breaks OLS, you can regress on your components, and simply select a smaller number of the highest variation components.
Principal Component Analysis by Jollife a very in-depth and highly cited book on the subject
This is also good: http://www.statsoft.com/textbook/principal-components-factor-analysis/