A dark painting crossed by pastel red, orange, blue and yellow curves rising and levelling off

Week: Factorial Design

Lesson 13: Factorial Design

Testing Multiple Hypotheses in 1 Experiment

Tim Fraser, PhD

Assistant Teaching Professor, Systems Engineering
Cornell University

SYSEN 5300: Systems Engineering & Six Sigma

for Design and Operation of Reliable Systems

Risk Analysis
- Risk assessment and risk characterization
- Failure Modes and Effects Analysis
- Fault trees and event trees under uncertainty
Modeling Reliability
- Component and system reliability
- Physical acceleration models
- Required function, stated conditions, specified time
- One element of the broader risk analysis

Quality Control
- Six sigma and statistical process control
- Detect when performance is deteriorating
- Take corrective action in time
System Improvement
- Optimize system design for reliability
- Design of experiments
- Response surfaces — innovation, problem solving, and discovery

Today's lesson

Housekeeping
Factorial Design & Enumeration
Main Effects
Two-Way Effects
Three-Way Effects
Design Decisions

A corgi in a red sweater teaching equations at a chalkboard to a classroom of corgis

Figured you wouldn't mind me reusing this image again.

Housekeeping (1)

Want to make your poster/report extra glitzy?

I can give out shared project space on my Overleaf account to teams who want to code up their project in Latex.

Optional - not necessary

Want in? Ask in class or on Ed Discussion and I will add your team to a shared Overleaf project.

LaTeX is optional. A clean Word or Google Doc poster scores exactly the same.

Housekeeping (2)

You Choose!

What should we cover in our last week of class?

We've been very efficient this term and have a couple of open days.

Please fill out the topic poll by the deadline on the Canvas item.

On the ballot: monitoring & control systems · scaling up your code · data communication · AI for Six Sigma

Canvas: Topic Poll · Activity page

Factorial Design and Enumeration

Lesson 13

Factorial Design

What is Factorial Design?

Factorial design:

select a fixed number of “levels” of each of a number of factors (variables)
run experiments for all possible combinations.

Factors:

quantitative: temperature, concentration
qualitative: +2 types of catalysts, or presence/absence of a trait

· Unblocked Arrangement · Blocked Arrangement

Comparing 2 entities · T-test · Randomized blocking

Comparing k entities · ANOVA · Factorial Design

What is Factorial Design?

Importance of 2-level factorial designs

few runs per factor;
easy interpretation;
direct further experimentation;
designs can be suitably augmented;

· Unblocked Arrangement · Blocked Arrangement

Comparing 2 entities · T-test · Randomized blocking

Comparing k entities · ANOVA · Factorial Design

Why? The Effects Table

Example A: All Effects with Pooled Standard Error

Table 5.4: calculated effects and standard errors for the 2 to the 3 factorial pilot plant example — main effects of temperature 23.0, concentration -5.0 and catalyst 1.5, two-factor interactions T by C 1.5, T by K 10.0, C by K 0.0, and three-factor interaction T by C by K 0.5, each plus or minus 1.4

Example B: Direct Effects with Pooled Standard Error

Bar chart of differences of means across treatment groups: Torani vs Monin 18.82, skim minus whole milk 1.65, oatmilk minus whole milk 0.21, oatmilk minus skim milk -1.45, machine B minus A -30.38, heart minus foam art 11.74, each with a pooled standard error of 8.39

Factorial Enumeration

Listing out all possible combinations of factor levels

ensures ~equal variance across treatment groups
view full range of variation

Use expand_grid() in tidyr package

Pilot plant table: coded units of factors T, C and K with average yield y from duplicate runs, and the operational levels of factors for runs 1 to 8

Factorial Enumeration

library(dplyr)
library(tidyr)

# Get full factorial grid of combinations
grid = expand_grid(
  # Catalyst K
  k = c("A", "B"),
  # Concentration C
  c = c(20, 40),
  # Temperature t
  t = c(160, 180),
) %>%
  # order columns as shown in the example...
  mutate(run = 1:n()) %>%
  select(run, t, c, k)

import pandas as pd
import itertools

# Get full factorial grid of combinations
# (last factor varies fastest, like expand_grid)
grid = pd.DataFrame(
  itertools.product(
    ["A", "B"],   # Catalyst K
    [20, 40],     # Concentration C
    [160, 180]),  # Temperature t
  columns=["k", "c", "t"])
# order columns as shown in the example...
grid["run"] = range(1, len(grid) + 1)
grid = grid[["run", "t", "c", "k"]]

Example 1

Factorial Enumeration in R

Encode this Factorial Design Cube as a dataframe!

Factorial design cube of observed percent yields: 60, 72, 54 and 68 on the front face and 52, 83, 45 and 80 on the back, with axes for temperature T, concentration C and catalyst K

library(dplyr)
library(tidyr)

# Get full factorial grid of combinations
grid = expand_grid(
  # Catalyst K
  k = c("A", "B"),
  # Concentration C
  c = c(20, 40),
  # Temperature t
  t = c(160, 180),
) %>%
  # order columns as shown in the example...
  mutate(run = 1:n()) %>%
  select(run, t, c, k)

import pandas as pd
import itertools

# Get full factorial grid of combinations
# (last factor varies fastest, like expand_grid)
grid = pd.DataFrame(
  itertools.product(
    ["A", "B"],   # Catalyst K
    [20, 40],     # Concentration C
    [160, 180]),  # Temperature t
  columns=["k", "c", "t"])
# order columns as shown in the example...
grid["run"] = range(1, len(grid) + 1)
grid = grid[["run", "t", "c", "k"]]

Example 1

Factorial Enumeration in R

Get every possible combination of Catalyst K, Concentration C, and Temperature T for your experiment.

Then, encode the outcomes as y!

# We run the experiment once and get these results
data = grid %>% mutate(y = c(60,72,54,68,52,83,45, 80))

# We run the experiment once and get these results
data = grid.assign(
  y=[60, 72, 54, 68, 52, 83, 45, 80])

  run       t     c k         y
<int> <dbl> <dbl> <chr> <dbl>
    1   160    20 A        60
    2   180    20 A        72
    3   160    40 A        54
    4   180    40 A        68
    5   160    20 B        52
    6   180    20 B        83
    7   160    40 B        45
    8   180    40 B        80

Main Effects

Lesson 13

Factorial Design

2³ Factorial Design

Figure 5.3: display of the results from a pilot plant investigation employing a 2 to the 3 factorial design to study the effects of T, C and K on yield — panel a the factorial with runs in standard order, panel b observed percent yields, panel c the twelve treatment comparisons

Estimating Main Effects

Steps: 1. Vary 1 variable, holding others constant 2. Get all within-group differences 3. Get average within-group difference

ID · T (C) · C (%) · K (A/B) · y (%)

1 · 160 · 20 · A · 60

2 · 180 · 20 · A · 72

3 · 160 · 40 · A · 54

4 · 180 · 40 · A · 68

ID · T (C) · C (%) · K (A/B) · y (%)

5 · 160 · 20 · B · 52

6 · 180 · 20 · B · 83

7 · 160 · 40 · B · 45

8 · 180 · 40 · B · 80

Comparison of Concentrations, across all T & K groups (n = 4) | Effect of Concentration Change from 20 to 40%

Group · Concen. C · Temp T · Catalyst K · Effect (Cells) · Effect (Math) · d

a · 40/20 · 160 · A · y₃ - y₁ · 54 - 60 · -6

b · 40/20 · 180 · A · y₄ - y₂ · 68 - 72 · -4

c · 40/20 · 160 · B · y₇ - y₅ · 45 - 52 · -7

d · 40/20 · 180 · B · y₈ - y₆ · 80 - 83 · -3

Main (average) effect (d̄) of change in Concentration ΔC · (y_a+ y_b+ y_c+y_d) / n · (-6 + -4 + -7 + -3) / 4 · -5

Estimating Main Effects Visually

Steps:

Vary 1 variable, holding others constant
Get all within-group differences
Get average within-group difference

Factorial design cube of observed percent yields: 45, 80, 54, 68 on the top face and 52, 83, 60, 72 on the bottom, with axes for temperature T, concentration C and catalyst K

Cube of the four temperature comparisons: +35, +14, +31 and +12 along the T direction

Comparison of Temperatures, across all C & K groups (n = 4) | Effect of Temperature Change from 160 to 180

Group · Concen. C · Temp T · Catalyst K · Effect (Cells) · Effect (Math) · d

a · 20 · 180/160 · A · y₂ - y₁ · 72 - 60 · +12

b · 20 · 180/160 · A · y₆ - y₅ · 83 - 52 · +31

c · 40 · 180/160 · B · y₄ - y₃ · 68 - 54 · +14

d · 40 · 180/160 · B · y₈ - y₇ · 80 - 45 · +35

Main (average) effect (d̄) of change in Temperature ΔT · (y_a+ y_b+ y_c+y_d) / n · (12 + 31 + 14 + 35) / 4 · +23

Example 2

Coding Main / Direct Effects

Estimate the direct effects of Concentration, Temperature, and the Catalyst from this experiment, coding in R.

Factorial design cube of observed percent yields with axes for temperature T, concentration C and catalyst K

Starting with our 'data' data.frame from Example 1…

  run       t     c k         y
<int> <dbl> <dbl> <chr> <dbl>
    1   160    20 A        60
    2   180    20 A        72
    3   160    40 A        54
    4   180    40 A        68
    5   160    20 B        52
    6   180    20 B        83
    7   160    40 B        45
    8   180    40 B        80

Coding Main / Direct Effects

# Calculate the direct (one-way) treatment effects
data %>%
  summarize(
    dbar_c = mean( y[c==40] - y[c==20] ),
    dbar_t = mean( y[t== 180] - y[t==160] ),
    dbar_k = mean( y[k== "B"] - y[k=="A"] )
  )

# Calculate the direct (one-way) treatment effects
# Coded factors: high level (+1) minus low (-1)
d, y = data, data["y"]
pd.DataFrame({
  "dbar_c": [(y[d.c == 40].values -
              y[d.c == 20].values).mean()],
  "dbar_t": [(y[d.t == 180].values -
              y[d.t == 160].values).mean()],
  "dbar_k": [(y[d.k == "B"].values -
              y[d.k == "A"].values).mean()]})

dbar_c dbar_t dbar_k
 <dbl>  <dbl>  <dbl>
    -5     23    1.5

Two-Way Interaction Effects

Lesson 13

Factorial Design

Two-Way Interactions

When two factors have a coupled influence on y beyond their main effects
e.g., temperature and catalyst
measured by difference between the temperature effects at the plus and minus levels of the catalyst factor.

Two-Way Interactions

ŷ₁ = (Y_180,B + Y_160,A) / 2

80, 83 60, 54

ŷ₀ = (Y_160,B + Y_180,A) / 2

52, 45 72, 68

d̄ = ŷ₁ - ŷ₀

Temp T · Catalyst A · Catalyst B

180 72, 68 80, 83

160 60, 54 52, 45

Example 3

Coding Two-Way Interaction Effects

Estimate the two-way interaction effects of Temperature T and Catalyst K together on the outcome, coding in R.

Starting with our 'data' data.frame from Example 1…

  run       t     c k         y
<int> <dbl> <dbl> <chr> <dbl>
    1   160    20 A        60
    2   180    20 A        72
    3   160    40 A        54
    4   180    40 A        68
    5   160    20 B        52
    6   180    20 B        83
    7   160    40 B        45
    8   180    40 B        80

Coding Two-Way Interaction Effects

# Calculate the two-way treatment effects
data %>%
  reframe(
    xbar1 = y[ (t==180 & k=="B") | (t==160& k=="A") ] %>% mean(),
    xbar0 = y[ (t==160 & k=="B") | (t==180& k=="A")] %>% mean(),
    dbar = xbar1 - xbar0
  )

# Calculate the two-way treatment effects
# Model form: lm(y ~ A*B) -> smf.ols("y ~ A*B")
d, y = data, data["y"]
xbar1 = y[((d.t == 180) & (d.k == "B")) |
          ((d.t == 160) & (d.k == "A"))].mean()
xbar0 = y[((d.t == 160) & (d.k == "B")) |
          ((d.t == 180) & (d.k == "A"))].mean()
pd.DataFrame({"xbar1": [xbar1], "xbar0": [xbar0],
              "dbar": [xbar1 - xbar0]})

A tibble: 1 × 3
xbar1 xbar0  dbar
<dbl> <dbl> <dbl>
 69.2  59.2    10

Three-Way Interaction Effects

Lesson 13

Factorial Design

Three-Way Interactions

When 3 factors have a coupled influence
e.g., temperature, catalyst, AND concentration
measured by average difference between the temperature*concentration effects at the plus and minus levels of the catalyst factor.

Interaction TC equals the quantity y8 minus y7 minus the quantity y6 minus y5, all over 2, equals 80 minus 45 minus 83 minus 52, over 2, equals 35 minus 31 over 2 equals 2; when K is at its minus level, catalyst A, interaction TC equals y4 minus y3 minus y2 minus y1 over 2 equals 68 minus 54 minus 72 minus 60 over 2 equals 14 minus 12 over 2 equals 1

Three-factor interaction of T, C, and K. TCK=(2-1)/2=0.5

Factorial design cube with the 45-80 and 52-83 edges circled in blue and the 54-68 and 60-72 edges circled in pink, marking the temperature by concentration comparisons at each level of catalyst K

Example 4

Coding Three-Way Interaction Effects

Estimate the three-way interaction effects of Temperature T, Catalyst K, and Concentration C together on the outcome, coding in R.

Starting with our 'data' data.frame from Example 1…

  run       t     c k         y
<int> <dbl> <dbl> <chr> <dbl>
    1   160    20 A        60
    2   180    20 A        72
    3   160    40 A        54
    4   180    40 A        68
    5   160    20 B        52
    6   180    20 B        83
    7   160    40 B        45
    8   180    40 B        80

# Now get the average difference between these interactions
data %>%
  reframe(
    # Get the TC interaction when K is A
    d1a = y[t==180&k=="A"&c==40] - y[t==160&k=="A"&c==40],
    d0a = y[t==180&k=="A"&c==20] - y[t==160&k=="A"&c==20],
    dbar_a = (d1a - d0a)/2,

    # Get the TC interaction when K is B
    d1b = y[t==180&k=="B"&c==40] - y[t==160&k=="B"&c==40],
    d0b = y[t==180&k=="B"&c==20] - y[t==160&k=="B"&c==20],
    dbar_b = (d1b - d0b)/2,

    # Get three way interaction effect
    dbar = (dbar_b - dbar_a) / 2
  )

# Now get the average difference between these interactions
d, y = data, data["y"]
def at(t, k, c):
  return y[(d.t == t) & (d.k == k) & (d.c == c)].iat[0]

# Get the TC interaction when K is A
d1a = at(180, "A", 40) - at(160, "A", 40)
d0a = at(180, "A", 20) - at(160, "A", 20)
dbar_a = (d1a - d0a) / 2

# Get the TC interaction when K is B
d1b = at(180, "B", 40) - at(160, "B", 40)
d0b = at(180, "B", 20) - at(160, "B", 20)
dbar_b = (d1b - d0b) / 2

# Get three way interaction effect
dbar = (dbar_b - dbar_a) / 2

  d1a   d0a dbar_a   d1b   d0b dbar_b  dbar
<dbl> <dbl>  <dbl> <dbl> <dbl>  <dbl> <dbl>
   14    12      1    35    31      2   0.5

Genuine Replicate Runs

Good experiments rely on not just 1 replicates, but many replicates
Allows us to estimate standard errors

Table of average response values with the results from individual duplicate runs, the difference of duplicates, and the estimated variance at each set of conditions, totalling 64 with a pooled estimate s squared of 8 on 8 degrees of freedom

Each estimated effect T, C, K, TC, … is a difference between two averages of 8 obs., the variance of effect

v = 8 degrees of freedom (2 - 1)*8

n_differences = 8, groups = 2

Var(Effect)=(⅛ + ⅛) s² = 2

SE(Effect)=1.4

Genuine Replicate Runs

Randomization of run order ensures replication is genuine
i.e., variation between runs with same experimental conditions reflects total run-to-run variability
requires all the steps all over again.

Standard Errors of Effects

In general, if each factor combination was replicated, a pooled estimate of the experimental run variance from g factor combinations:

s squared equals v1 s1 squared plus v2 s2 squared plus dot dot dot vg sg squared, all over v1 plus v2 plus dot dot dot vg, where v equals v1 plus v2 plus dot dot dot vg is the number of DOF of the estimate; in the case of g duplicate combinations it is reduced to s squared equals the sum of d squared over 2g with g DOF

What is s² in pilot plant example? What is variance (effect) and SE (effect)?

Activity page: Standard Errors of Effects — s² in the Pilot Plant Example →

Effects Tables

It is important to determine which effects are almost certainly real and which might readily be explained by chance.
A rough rule is that effects greater than 2-3 times their standard error are not easily explained by chance alone.

Flip the factor levels and watch the main effects and the interaction move — then climb the response surface.

Rebuild the pilot-plant effects yourself — in R, without leaving the slide.

The same pilot plant in Python — pandas masks and smf.ols("y ~ t*c*k").

Coding Demo

Lesson 13

Factorial Design

A stack of cartoon corgis piled into a pyramid

https://github.com/timothyfraser/sysen/blob/main/code/13_lesson.R
Use these functions! https://github.com/timothyfraser/sysen/blob/main/functions/functions_factorial.R