Randomized Block Experiment: Example

This lesson shows how to use analysis of variance to analyze and interpret data from a randomized block experiment. To illustrate the process, we walk step-by-step through a real-world example.

Computations for analysis of variance are usually handled by a software package. For this example, however, we will do the computations "manually", since the gory details have educational value.

Prerequisites: The lesson assumes general familiarity with randomized block designs. If you are unfamiliar with randomized block designs or with terms like blocks, blocking, and blocking variables, review the previous lessons:

• Randomized Block Designs
• Randomized Block Experiments: Data Analysis

Problem Statement

As part of a randomized block experiment, a researcher tests the effect of three teaching methods on student performance. The researcher selects subjects randomly from a student population. The researcher assigns subjects to six blocks of three, such that students within the same block have the same (or similar) IQ. Within each block, each student is randomly assigned to a different teaching method.

At the end of the term, the researcher collects one test score (the dependent variable) from each subject, as shown in the table below:

Table 1. Dependent Variable Scores

IQ	Teaching Method
	A	B	C
91-95	84	85	85
96-100	86	86	88
101-105	86	87	88
106-110	89	88	89
111-115	88	89	89
116-120	91	90	91

In conducting this experiment, the researcher has two research questions:

• Does teaching method have a significant effect on student performance (as measured by test score)?
• How strong is the effect of teaching method on the student performance?

To answer these questions, the researcher uses analysis of variance.

Analytical Logic

To implement analysis of variance with an independent groups, randomized block experiment, a researcher takes the following steps:

• Specify a mathematical model to describe how main effects and the blocking variable influence the dependent variable.
• Write statistical hypotheses to be tested by experimental data.
• Specify a significance level for a hypothesis test.
• Compute the grand mean and marginal means for the independent variable and for the blocking variable.
• Compute sums of squares for each effect in the model.
• Find the degrees of freedom associated with each effect in the model.
• Based on sums of squares and degrees of freedom, compute mean squares for each effect in the model.
• Find the expected value of the mean squares for each effect in the model.
• Compute a test statistic for the independent variable and a test statistic for the blocking variable, based on observed mean squares and their expected values.
• Find the P value for each test statistic.
• Accept or reject null hypotheses, based on P value and significance level.
• Assess the magnitude of effect, based on sums of squares.

Below, we'll explain how to implement each step in the analysis.

Mathematical Model

For every experimental design, there is a mathematical model that accounts for all of the independent and extraneous variables that affect the dependent variable. Here is a mathematical model for an independent groups, randomized block experiment:

X _{i j} = μ + β _i + τ _j + ε _ij

where X _{i j} is the dependent variable score (in this example, the test score) for the subject in block i that receives treatment j, μ is the population mean, β _i is the effect of Block i; τ _j is the effect of Treatment j; and ε _ij is the experimental error (i.e., the effect of all other extraneous variables).

For this model, it is assumed that ε _ij is normally and independently distributed with a mean of zero and a variance of σ_ε². The mean ( μ ) is constant.

Note: Unlike the model for a full factorial experiment, the model for a randomized block experiment does not include an interaction term. That is, the model assumes there is no interaction between block and treatment effects.

Statistical Hypotheses

With a randomized block experiment, it is possible to test both block ( β _i ) and treatment ( τ _j ) effects. Here are the null hypotheses (H₀) and alternative hypotheses (H₁) for each effect.

With a randomized block experiment, the main hypothesis test of interest is the test of the treatment effect(s). For instance, in this example the experimenter is primarily interested in the effect of teaching method on student performance (i.e., test score).

Block effects are of less intrinsic interest, because a blocking variable is thought to be a nuisance variable that is only included in the experiment to control for a potential source of undesired variation. In this example, IQ is a potential nuisance variable.

Significance Level

The significance level (also known as alpha or α) is the probability of rejecting the null hypothesis when it is actually true. The significance level for an experiment is specified by the experimenter, before data collection begins. Experimenters often choose significance levels of 0.05 or 0.01. For this experiment, we'll assume that the experimenter chose 0.05 as the significance level.

A significance level of 0.05 means that there is a 5% chance of rejecting the null hypothesis when it is true. A significance level of 0.01 means that there is a 1% chance of rejecting the null hypothesis when it is true. The lower the significance level, the more persuasive the evidence needs to be before an experimenter can reject the null hypothesis.

Mean Scores

Analysis of variance for a randomized block experiment begins by computing a grand mean and marginal means for independent variables and for blocks. Here are computations for the various means, based on dependent variable scores from Table 1:

• Grand mean. The grand mean (X) is the mean of all observations, computed as follows:
  N = nk = 6 * 3 = 18
  X = ( 1 / N )

  nΣi=1

  kΣj=1

  ( X _{i j} )
  X = ( 1 / 18 )

  6Σi=1

  3Σj=1

  ( X _{i j} )

  X = 87.72

• Marginal means for treatment levels. The mean for treatment level j ( X _. j ) is computed as follows:
  X _. j  = ( 1 / n )

  nΣi=1

  ( X _{i j} )
  X _. 1  = ( 1 / 6 )

  6Σi=1

  ( X _{i 1} ) = 87.33
  X _. 2  = ( 1 / 6 )

  6Σi=1

  ( X _{i 2} ) = 87.50
  X _. 3  = ( 1 / 6 )

  6Σi=1

  ( X _{i 3} ) = 88.33
• Marginal means for blocks. The mean for block i ( X _i . ) is computed as follows:
  X _i .  = ( 1 / k )

  kΣj=1

  ( X _{i j} )
  X _1 .  = ( 1 / 3 )

  3Σj=1

  ( X _{1 j} ) = 84.67
  X _2 .  = ( 1 / 3 )

  3Σj=1

  ( X _{2 j} ) = 86.67
  X _3 .  = ( 1 / 3 )

  3Σj=1

  ( X _{3 j} ) = 87.00
  X _4 .  = ( 1 / 3 )

  3Σj=1

  ( X _{4 j} ) = 88.67
  X _5 .  = ( 1 / 3 )

  3Σj=1

  ( X _{5 j} ) = 88.67
  X _6 .  = ( 1 / 3 )

  3Σj=1

  ( X _{6 j} ) = 90.67

In the equations above, N is the total sample size; n is the number of blocks, and k is the number of treatment levels.

Sums of Squares

A sum of squares is the sum of squared deviations from a mean score. A randomized block design makes use of four sums of squares:

• Sum of squares for treatments. The sum of squares for treatments (SSTR) measures variation of the marginal means of treatment levels ( X _j ) around the grand mean ( X ). It can be computed from the following formula:
  SSTR = n

  kΣj=1

  ( X _j - X )²
  SSTR = 6

  3Σj=1

  ( X _j - X )² = 3.44
• Sum of squares for blocks. The sum of squares for blocks (SSB) measures variation of the marginal means of blocks ( X _i ) around the grand mean ( X ). It can be computed from the following formula:
  SSB = k

  nΣi=1

  ( X _i - X )²
  SSB = 3

  6Σi=1

  ( X _i - X )² = 64.28
• Error sum of squares. The error sum of squares (SSE) measures variation of all scores ( X _i j ) attributable to extraneous variables. It can be computed from the following formula:
  SSE =

  nΣi=1

  kΣj=1

  ( X _{i j}  - X _i  - X _j  + X )²
  SSE =

  6Σi=1

  3Σj=1

  ( X _{i j}  - X _i  - X _j  + X )² = 3.89
• Total sum of squares. The total sum of squares (SST) measures variation of all scores ( X _i j ) around the grand mean ( X ). It can be computed from the following formula:
  SST =

  nΣi=1

  kΣj=1

  ( X _{i j}  - X )²
  SST =

  6Σi=1

  3Σj=1

  ( X _{i j}  - X )² = 71.61

In the formulas above, n is the number of blocks, and k is the number of treatment levels. And the total sum of squares is equal to the sum of the component sums of squares, as shown below:

SST = SSTR + SSB + SSE

SST = 3.44 + 64.28 + 3.89 = 71.61

Degrees of Freedom

The term degrees of freedom (df) refers to the number of independent sample points used to compute a statistic minus the number of parameters estimated from the sample points.

The degrees of freedom used to compute the various sums of squares for an independent groups, randomized block experiment are shown in the table below:

Sum of squares	Degrees of freedom
Treatment	k - 1 = 2
Block	n - 1 = 5
Error	( k - 1 )( n - 1 ) = 10
Total	nk - 1 = 17

Notice that there is an additive relationship between the various sums of squares. The degrees of freedom for total sum of squares (df_TOT) is equal to the degrees of freedom for the treatment sum of squares (df_TR) plus the degrees of freedom for the blocks sum of squares (df_B) plus the degrees of freedom for the error sum of squares (df_E). That is,

df_TOT = df_TR + df_B + df_E

df_TOT = 2 + 5 + 7 = 17

Mean Squares

A mean square is an estimate of population variance. It is computed by dividing a sum of squares (SS) by its corresponding degrees of freedom (df), as shown below:

MS = SS / df

To conduct analysis of variance with a randomized block experiment, we are interested in three mean squares:

• Treatment mean square. The treatment mean square ( MS_T ) measures variation due to treatment levels. It can be computed as follows:

  MS_T = SSTR / df_TR

  MS_T = 3.44 / 2 = 1.72

• Block mean square. The block mean square ( MS_B ) measures variation due to blocks. It can be computed as follows:

  MS_B = SSB / df_B

  MS_B = 64.28 / 5 = 12.86

• Error mean square. The error mean square ( MS_E ) measures variation due to extraneous variables (anything other than the treatment or the blocking variable). The error mean square can be computed as follows:

  MS_E = SSE / df_E

  MS_E = 3.89 / 10 = 0.39

Expected Value

The expected value of a mean square is the average value of the mean square over a large number of experiments.

Statisticians have derived formulas for the expected value of mean squares, assuming the mathematical model described earlier is correct. Those formulas appear below:

Mean square	Expected value
MST	σ2E + nσ2T
MSB	σ2E + kσ2B
MSE	σ2E

In the table above, MS_T is the mean square for treatments; MS_B is the mean square for blocks; and MS_E is the error mean square.

Test Statistics

The main data analysis goal for this experiment is to test the hypotheses that we stated earlier (see Statistical Hypotheses). That will require the use of test statistics. Let's talk about how to compute test statistics for this study and how to interpret the statistics we compute.

How to Compute Test Statistics

Suppose we want to test the significance of an independent variable or a blocking variable in a randomized block experiment. We can use the mean squares to define a test statistic F for each source of variation, as shown in the table below:

Source	Mean square: Expected value	F ratio
Treatment (T)	σ2E + nσ2T	MST MSE
Block (B)	σ2E + kσ2B	MSB MSE
Error	σ2E

Using formulas from the table with data from this randomized block experiment, we can compute an F ratio for treatments ( F_T ) and an F ratio for blocks ( F_B ).

F_T = MS_T / MS_E = 1.72/0.39 = 4.4

F_B = MS_B / MS_E = 12.86/0.39 = 33.0

How to Interpret Test Statistics

Consider the F ratio for the treatment effect in this randomized block experiment. For convenience, we display once again the table that shows expected mean squares and F ratio formulas:

Source	Mean square: Expected value	F ratio
Treatment (T)	σ2E + nσ2T	MST MSE
Block (B)	σ2E + kσ2B	MSB MSE
Error	σ2E

Notice that numerator of the F ratio for the treatment effect should equal the denominator when the variation due to the treatment ( σ² _T ) is zero (i.e., when the treatment does not affect the dependent variable). And the numerator should be bigger than the denominator when the variation due to the treatment is not zero (i.e., when the treatment does affect the dependent variable).

The F ratio for the blocking variable works the same way. When the blocking variable does not affect the dependent variable, the numerator of the F ratio should equal the denominator. Otherwise, the numerator should be bigger than the denominator.

Each F ratio is a convenient measure that we can use to test the null hypothesis about the effect of a source (the treatment or the blocking variable) on the dependent variable. Here's how to conduct the test:

• When the F ratio is close to one, the numerator of the F ratio is approximately equal to the denominator. This indicates that the source did not affect the dependent variable, so we cannot reject the null hypothesis.
• When the F ratio is significantly greater than one, the numerator is bigger than the denominator. This indicates that the source did affect the dependent variable, so we must reject the null hypothesis.

What does it mean for the F ratio to be significantly greater than one? To answer that question, we need to talk about the P-value.

Warning: Recall that this analysis assumes that the interaction between blocking variable and independent variable is zero. If that assumption is incorrect, the F ratio for a fixed-effects variable will be biased. It may indicate that an effect is not significant, when it truly is significant.

P-Value

In an experiment, a P-value is the probability of obtaining a result more extreme than the observed experimental outcome, assuming the null hypothesis is true.

With analysis of variance for a randomized block experiment, the F ratios are the observed experimental outcomes that we are interested in. So, the P-value would be the probability that an F ratio would be more extreme (i.e., bigger) than the actual F ratio computed from experimental data.

How does an experimenter attach a probability to an observed F ratio? Luckily, the F ratio is a random variable that has an F distribution. The degrees of freedom (v₁ and v₂) for the F ratio are the degrees of freedom associated with the mean squares used to compute the F ratio.

For example, consider the F ratio for a treatment effect. That F ratio ( F_T ) is computed from the following formula:

F_T = F(v₁, v₂) = MS_T / MS_E

MS_T (the numerator in the formula) has degrees of freedom equal to df_TR ; so for F_T , v₁ is equal to df_TR . Similarly, MS_E (the denominator in the formula) has degrees of freedom equal to df_E ; so for F_T , v₂ is equal to df_E . Knowing the F ratio and its degrees of freedom, we can use an F table or Stat Trek's free F distribution calculator to find the probability that an F ratio will be bigger than the actual F ratio observed in the experiment.

To illustrate the process, let's find P-values for the treatment variable and for the blocking variable in this randomized block experiment.

Treatment Variable P-Value

From previous computations, we know the following:

• The observed value of the F ratio for the treatment variable is 4.4.
• The F ratio (F_T) was computed from the following formula:

  F_T = F(v₁, v₂) = MS_T / MS_E

• The degrees of freedom (v₁) for the treatment variable mean square (MS_T) is 2.
• The degrees of freedom (v₂) for the error mean square (MS_E) is 10.

Therefore, the P-value we are looking for is the probability that an F with 2 and 10 degrees of freedom is greater than 4.4. We want to know:

P [ F(2, 10) > 4.4 ]

Now, we are ready to use the F Distribution Calculator. We enter the degrees of freedom (v1 = 2) for the treatment mean square, the degrees of freedom (v2 = 10) for the error mean square, and the F value (4.4) into the calculator; and hit the Calculate button.

The calculator reports that the probability that F is greater than 4.4 equals about 0.04. Hence, the correct P-value for the treatment variable is 0.04.

Blocking Variable P-Value

The process to compute the P-value for the blocking variable is exactly the same as the process used for the treatment variable. From previous computations, we know the following:

• The observed value of the F ratio for the blocking variable is 33.
• The F ratio (F_B) was computed from the following formula:

  F_B = F(v₁, v₂) = MS_B / MS_E

• The degrees of freedom (v₁) for the blocking variable mean square (MS_B) is 5.
• The degrees of freedom (v₂) for the error mean square (MS_E) is 10.

Therefore, the P-value we are looking for is the probability that an F with 5 and 10 degrees of freedom is greater than 33. We want to know:

P [ F(5, 10) > 33 ]

Now, we are ready to use the F Distribution Calculator. We enter the degrees of freedom (v1 = 5) for the block mean square, the degrees of freedom (v2 = 10) for the error mean square, and the F value (33) into the calculator; and hit the Calculate button.

The calculator reports that the probability that F is greater than 33 is about 0.00001. Hence, the correct P-value is 0.00001.

Interpretation of Results

Having completed the computations for analysis, we are ready to interpret results. We begin by displaying key findings in an ANOVA summary table. Then, we use those findings to (1) test hypotheses and (2) assess the magnitude of effects.

ANOVA Summary Table

It is traditional to summarize ANOVA results in an analysis of variance table. Here, filled with key results, is the analysis of variance table for the randomized block experiment that we have been working on.

Analysis of Variance Table

Source	SS	df	MS	F	P
Treatment	3.44	2	1.72	4.4	0.04
Block	64.28	5	12.86	33	<0.01
Error	3.89	10	0.39
Total	71.61	17

This ANOVA table provides all the information that we need to (1) test hypotheses and (2) assess the magnitude of treatment effects.

Hypothesis Test

Recall that the experimenter specified a significance level of 0.05 for this study. Once you know the significance level and the P-values, the hypothesis tests are routine. Here's the decision rule for accepting or rejecting a null hypothesis:

• If the P-value is bigger than the significance level, accept the null hypothesis.
• If the P-value is equal to or smaller than the significance level, reject the null hypothesis.

A "big" P-value for a source of variation (an independent variable or a blocking variable) indicates that the source did not have a statistically significant effect on the dependent variable. A "small" P-value indicates that the source did have a statistically significant effect on the dependent variable.

The P-value (shown in the last column of the ANOVA table) is the probability that an F statistic would be more extreme (bigger) than the F ratio shown in the table, assuming the null hypothesis is true. When a P-value for an independent variable or a blocking variable is bigger than the significance level, we accept the null hypothesis for the effect; when it is smaller, we reject the null hypothesis.

Based on the P-values in the table above, we can draw the following conclusions:

• The P-value for treatments (i.e., the independent variable) is 0.04. Since the P-value is smaller than the significance level (0.05), we reject the null hypothesis that the independent variable (training method) has no effect on the dependent variable.
• The P-value for the blocking variable is less than 0.01. Since this P-value is also smaller than the significance level (0.05), we reject the null hypothesis that the blocking variable (IQ) has no effect on the dependent variable.

In addition, two other points are worthy of note:

• The fact that the blocking variable (IQ) is statistically significant is good news in a randomized block experiment. It confirms the suspicion that the blocking variable was a nuisance variable that could have obscured effects of the dependent variable. And it justifies the decision to use a randomized block experiment to control nuisance effects of IQ.
• The independent variable (training method) was also statistically significant with a P-value of 0.04. Had the experimenter used a different design that did not control the nuisance effect of IQ, the experiment might not have produced a significant effect for the independent variable.

Magnitude of Effect

The hypothesis tests tell us whether sources of variation in our experiment had a statistically significant effect on the dependent variable, but the tests do not address the magnitude of the effect. Here are some issues:

• When the sample size is large, you may find that even small effects (indicated by a small F ratio) are statistically significant.
• When the sample size is small, you may find that even big effects are not statistically significant.
• When the blocking variable in a randomized block design is strongly correlated with the dependent variable, you may find that even small treatment effects are statistically significant.

With this in mind, it is customary to supplement analysis of variance with an appropriate measure of effect size. Eta squared (η²) is one such measure. Eta squared is the proportion of variance in the dependent variable that is explained by a source of variation. The eta squared formula for an independent variable or a blocking variable is:

η² = SS_SOURCE / SST

where SS_SOURCE is the sum of squares for a source of variation (i.e., an independent variable or a blocking variable) and SST is the total sum of squares.

Using sum of squares entries from the ANOVA table, we can compute eta squared for the treatment variable ( η²_T ) and for the blocking variable ( η²_B ).

η²_T = SSTR / SST = 3.44 / 71.61 = 0.05

η²_B = SSB / SST = 64.28 / 71.61 = 0.90

The treatment variable (test method) accounted for about 5% of the variance in test performance, and the blocking variable (IQ) accounted for about 90% of the variance in test performance. Based on these findings, an experimenter might conclude:

• IQ accounted for most of the variance in test performance.
• Even though the test method effect was statistically significant, test method accounted for only a small proportion of test variation.

Note: Given the very strong nuisance effect of IQ, it is likely that a different experimental design would not have revealed a statistically significant effect for test method.

An Easier Option

In this lesson, we showed all of the hand calculations for analysis of variance with a randomized block experiment. In the real world, researchers seldom conduct analysis of variance by hand. They use statistical software. In the next lesson, we'll demonstrate how to conduct the same analysis of the same problem with Excel. Hopefully, we'll get the same result.