Variance-Covariance Matrix

This lesson explains how to use matrix methods to generate a variance-covariance matrix from a matrix of raw data.

Variance

Variance is a measure of the variability or spread in a set of data. Mathematically, it is the average squared deviation from the mean score. We use the following formula to compute population variance.

Var(X) = Σ ( X_i - X )² / N = Σ x_i² / N

where

N is the number of scores in a set of scores
X is the mean of the N scores.
X_i is the ith raw score in the set of scores
x_i is the ith deviation score in the set of scores
Var(X) is the variance of all the scores in the set

Covariance

Covariance is a measure of the extent to which corresponding elements from two sets of ordered data move in the same direction. We use the following formula to compute population covariance.

Cov(X, Y) = Σ ( X_i - X ) ( Y_i - Y ) / N = Σ x_iy_i / N

where

N is the number of scores in each set of data
X is the mean of the N scores in the first data set
X_i is the ithe raw score in the first set of scores
x_i is the ith deviation score in the first set of scores
Y is the mean of the N scores in the second data set
Y_i is the ithe raw score in the second set of scores
y_i is the ith deviation score in the second set of scores
Cov(X, Y) is the covariance of corresponding scores in the two sets of data

Variance-Covariance Matrix

Variance and covariance are often displayed together in a variance-covariance matrix, (aka, a covariance matrix). The variances appear along the diagonal and covariances appear in the off-diagonal elements, as shown below.

V =

Σ x₁² / N	Σ x₁ x₂ / N	. . .	Σ x₁ x_c / N
Σ x₂ x₁ / N	Σ x₂² / N	. . .	Σ x₂ x_c / N
. . .	. . .	. . .	. . .
Σ x_c x₁ / N	Σ x_c x₂ / N	. . .	Σ x_c² / N

where

V is a c x c variance-covariance matrix
N is the number of scores in each of the c data sets
x_i is a deviation score from the ith data set
Σ x_i² / N is the variance of elements from the ith data set
Σ x_i x_j / N is the covariance for elements from the ith and jth data sets

How to Create a Variance-Covariance Matrix

Suppose X is an n x k matrix holding ordered sets of raw data. For example, matrix X might display the scores on k tests for n students, as shown in Problem 1.

Starting with the raw data of matrix X, you can create a variance-covariance matrix to show the variance within each column and the covariance between columns. Here's how.

Transform the raw scores from matrix X into deviation scores for matrix x.

x = X - 11'X ( 1 / n )

where

1 is an n x 1 column vector of ones
x is an n x k matrix of deviation scores: x₁₁, x₁₂, . . . , x_n_k
X is an n x k matrix of raw scores: X₁₁, X₁₂, . . . , X_n_k
Compute x'x, the k x k deviation sums of squares and cross products matrix for x.
Then, divide each term in the deviation sums of squares and cross product matrix by n to create the variance-covariance matrix. That is,

V = x'x ( 1 / n )

where

V is a k x k variance-covariance matrix
x'x is the deviation sums of squares and cross product matrix
n is the number of scores in each column of the original matrix X

In the next section, read Problem 1 for an example showing how to turn raw data into a variance-covariance matrix.

Test Your Understanding

Problem 1

The table below displays scores on math, English, and art tests for 5 students.

Student	Math	English	Art
1	90	60	90
2	90	90	30
3	60	60	60
4	60	60	90
5	30	30	30

Note that data from the table can be represented in matrix A, where each column in the matrix shows scores on a test and each row shows scores for a student.

A =

90	60	90
90	90	30
60	60	60
60	60	90
30	30	30

Given the data represented in matrix A, compute the variance of each test and the covariance between the tests.

Solution

The solution involves a three-step process.

First, we transform the raw scores in matrix A to deviation scores in matrix a, using the transformation formula described at how to transform raw scores to deviation scores.
a = A - 11'A ( 1 / n )

where

1 is an 5 x 1 column vector of ones
a is an 5 x 3 matrix of deviation scores: a₁₁, a₁₂, . . . , a₅₃
A is an 5 x 3 matrix of raw scores: A₁₁, A₁₂, . . . , A₅₃
n is the number of rows in matrix A

90	60	90
90	90	30
60	60	60
60	60	90
30	30	30

1	1	1	1	1
1	1	1	1	1
1	1	1	1	1
1	1	1	1	1
1	1	1	1	1

90	60	90
90	90	30
60	60	60
60	60	90
30	30	30

(1/5)

a =

90	60	90
90	90	30
60	60	60
60	60	90
30	30	30

66	60	60
66	60	60
66	60	60
66	60	60
66	60	60

a =

24	0	30
24	30	-30
-6	0	0
-6	0	30
-36	-30	-30

Then, to find the deviation score sums of squares matrix, we compute a'a, as shown below.

a' a =

24	24	-6	-6	-36
0	30	0	0	-30
30	-30	0	30	-30

24	0	30
24	30	-30
-6	0	0
-6	0	30
-36	-30	-30

a' a =

2520	1800	900
1800	1800	0
900	0	3600

And finally, to create the variance-covariance matrix, we divide each element in the deviation sum of squares matrix by n, as shown below.

V = a' a / n

V =

2520/5	1800/5	900/5
1800/5	1800/5	0/5
900/5	0/5	3600/5

V =

504	360	180
360	360	0
180	0	720

We can interpret the variance and covariance statistics in matrix V to understand how the various test scores vary and covary.

Shown in red along the diagonal, we see the variance of scores for each test. The art test has the biggest variance (720); and the English test, the smallest (360). So we can say that art test scores are more variable than English test scores.
The covariance is displayed in black in the off-diagonal elements of matrix V.
- The covariance between math and English is positive (360), and the covariance between math and art is positive (180). This means the scores tend to covary in a positive way. As scores on math go up, scores on art and English also tend to go up; and vice versa.
- The covariance between English and art, however, is zero. This means there tends to be no predictable relationship between the movement of English and art scores.

If the covariance between any tests had been negative, it would have meant that the test scores on those tests tend to move in opposite directions. That is, students with relatively high scores on the first test would tend to have relatively low scores on the second test.