Variance-Covariance Matrix
This lesson explains how to use matrix methods to generate a variance-covariance matrix from a matrix of raw data.
Variance
Variance is a measure of the variability or spread in a set of data. Mathematically, it is the average squared deviation from the mean score. We use the following formula to compute population variance.
Var(X) = Σ ( Xi - X )2 / N = Σ xi2 / N
where
N is the number of scores in a set of scores
X is the
mean
of the N scores.
Xi is the ith raw score in the set of scores
xi is the ith deviation score in the set of
scores
Var(X) is the variance of all the scores in the set
Covariance
Covariance is a measure of the extent to which corresponding elements from two sets of ordered data move in the same direction. We use the following formula to compute population covariance.
Cov(X, Y) = Σ ( Xi - X ) ( Yi - Y ) / N = Σ xiyi / N
where
N is the number of scores in each set of data
X is the
mean
of the N scores in the first dataset
Xi is the ithe raw score in the first set of
scores
xi is the ith deviation score in the first set of
scores
Y is the
mean
of the N scores in the second dataset
Yi is the ithe raw score in the second set
of scores
yi is the ith deviation score in the second set of
scores
Cov(X, Y) is the covariance of corresponding scores in the
two sets of data
Variance-Covariance Matrix
Variance and covariance are often displayed together in a variance-covariance matrix, (aka, a covariance matrix). The variances appear along the diagonal and covariances appear in the off-diagonal elements, as shown below.
| V = |
|
where
V is a c x c variance-covariance matrix
N is the number of scores in each of the c datasets
xi is a
deviation score
from the ith dataset
Σ xi2 / N is the variance of elements from the
ith dataset
Σ xi xj / N is the covariance for
elements from the ith and jth datasets
How to Create a Variance-Covariance Matrix
Suppose X is an n x k matrix holding ordered sets of raw data. For example, matrix X might display the scores on k tests for n students, as shown in Problem 1.
Starting with the raw data of matrix X, you can create a variance-covariance matrix to show the variance within each column and the covariance between columns. Here's how.
Transform the raw scores from matrix X into deviation scores for matrix x.
x = X - 11'X ( 1 / n )
where
1 is an n x 1 column vector of ones
1' is the transpose of vector 1
x is an n x k matrix of deviation scores: x11, x12, . . . , xnk
X is an n x k matrix of raw scores: X11, X12, . . . , XnkCompute x'x, the k x k deviation sums of squares and cross products matrix for x.
Then, divide each term in the deviation sums of squares and cross product matrix by n to create the variance-covariance matrix. That is,
V = x'x ( 1 / n )
where
V is a k x k variance-covariance matrix
x' is the transpose of matrix x
x'x is the deviation sums of squares and cross product matrix
n is the number of scores in each column of the original matrix X
In the next section, read Problem 1 for an example showing how to turn raw data into a variance-covariance matrix.
Test Your Understanding
Problem 1
The table below displays scores on math, English, and art tests for 5 students.
| Student | Math | English | Art |
|---|---|---|---|
| 1 | 90 | 60 | 90 |
| 2 | 90 | 90 | 30 |
| 3 | 60 | 60 | 60 |
| 4 | 60 | 60 | 90 |
| 5 | 30 | 30 | 30 |
Note that data from the table can be represented in matrix A, where each column in the matrix shows scores on a test and each row shows scores for a student.
| A = |
|
Given the data represented in matrix A, compute the variance of each test and the covariance between the tests.
Solution
The solution involves a three-step process.
- First, we transform the raw scores in matrix A
to deviation scores in matrix a,
using the transformation formula described at
how to transform raw scores to deviation scores.
a = A - 11'A ( 1 / n )
where
1 is an 5 x 1 column vector of ones
a is an 5 x 3 matrix of deviation scores: a11, a12, . . . , a53
A is an 5 x 3 matrix of raw scores: A11, A12, . . . , A53
n is the number of rows in matrix A
| a | = |
| 90 | 60 | 90 | ||
| 90 | 90 | 30 | ||
| 60 | 60 | 60 | ||
| 60 | 60 | 90 | ||
| 30 | 30 | 30 |
| - |
| 1 | 1 | 1 | 1 | 1 | ||
| 1 | 1 | 1 | 1 | 1 | ||
| 1 | 1 | 1 | 1 | 1 | ||
| 1 | 1 | 1 | 1 | 1 | ||
| 1 | 1 | 1 | 1 | 1 |
| 90 | 60 | 90 | ||
| 90 | 90 | 30 | ||
| 60 | 60 | 60 | ||
| 60 | 60 | 90 | ||
| 30 | 30 | 30 |
| (1/5) |
| a = |
|
- |
|
| a = |
|
Then, to find the deviation score sums of squares matrix, we compute a'a, as shown below.
| a' a = |
|
|
| a' a = |
|
And finally, to create the variance-covariance matrix, we divide each element in the deviation sum of squares matrix by n, as shown below.
V = a' a / n
| V = |
|
| V = |
|
We can interpret the variance and covariance statistics in matrix V to understand how the various test scores vary and covary.
- Shown in red along the diagonal, we see the variance of scores for each test. The art test has the biggest variance (720); and the English test, the smallest (360). So we can say that art test scores are more variable than English test scores.
- The covariance is displayed in black in the off-diagonal elements
of matrix V.
- The covariance between math and English is positive (360), and the covariance between math and art is positive (180). This means the scores tend to covary in a positive way. As scores on math go up, scores on art and English also tend to go up; and vice versa.
- The covariance between English and art, however, is zero. This means there tends to be no predictable relationship between the movement of English and art scores.
If the covariance between any tests had been negative, it would have meant that the test scores on those tests tend to move in opposite directions. That is, students with relatively high scores on the first test would tend to have relatively low scores on the second test.