Activity: Practice It (Permutation Tests)

1. A permutation test iteratively recomputes the same statistic from your sample while shuffling values, showing what that statistic would look like if it were purely due to chance. (For example, a difference of means near 0 would be most common if X were truly unrelated to Y.)
2. Permuting returns ~1000 statistics, each slightly different by chance, that approximate the shape of the null distribution for that statistic. This is different from bootstrapping, which approximates a sampling distribution.
3. In this activity you will practice permuting using the workshops/lattes.csv dataset.
4. Load in your dataset.
5. Select an outcome variable of interest (Y) and a predictor variable of interest (X).
6. Select a statistic of interest — a difference of means (t-test) or an F-statistic (ANOVA), depending on your predictor — and compute it once. This is your observed statistic (eg. dbar).
7. Permute your dataset to get 1000 versions in which the outcome is not related to the predictor.
8. Grouping by a unique id for each permuted version, calculate the statistic again in each.
9. Calculate the range of your permuted statistics.
10. What percentage of permuted statistics are more extreme than your observed statistic? That percentage is a one-tailed p-value.
11. Explain the result in 1–2 sentences.
12. Copy your commented code, your observed statistic, the range of your permutations, your p-value, and your description into the Canvas text submission box.

# Permutation Test Example #############################################
library(dplyr)
library(readr)
library(ggplot2)

donuts = read_csv("workshops/donuts.csv")

# 1. Observed statistic: difference of mean tastiness between two bakers
obs = donuts %>%
  summarize(dbar = mean(tastiness[baker == "A"]) - mean(tastiness[baker == "B"]))
obs$dbar

# 2. Permute: shuffle the outcome within 1000 copies of the dataset
mydbar = tibble(reps = 1:1000) %>%
  group_by(reps) %>%
  reframe(donuts %>% select(baker, tastiness)) %>%
  group_by(reps) %>%
  mutate(tastiness = sample(tastiness, replace = FALSE)) %>%
  group_by(reps) %>%
  summarize(dbar = mean(tastiness[baker == "A"]) - mean(tastiness[baker == "B"]))

# 3. Compare the observed statistic against the null distribution
ggplot() +
  geom_histogram(data = mydbar, mapping = aes(x = dbar)) +
  geom_vline(xintercept = obs$dbar)

# 4. One-tailed p-value
mean(mydbar$dbar > obs$dbar)

Submit on Canvas (1 pt) — graded on completion; your deadline is on the Canvas assignment.