Introduction
Suppose we run a single hypothesis test and get a p-value of 0.03. At the usual threshold of 0.05, we would call this a statistically significant result. But what if, instead of running one test, we ran twenty? Even if none of the twenty variables we tested had any real effect, just by chance we would expect about one of them to land below 0.05 anyway. The more tests we run, the more opportunities we give randomness to produce a "significant" result that isn't really there.
This is known as the multiple comparisons problem, and the Bonferroni correction is one of the simplest and most widely used tools to deal with it. In this chapter, we build an intuitive understanding of why running many tests inflates our error rate, and how the Bonferroni correction adjusts for it. We use a well-known, public dataset with country-level data to run several independent t-tests, so that the problem—and the fix—become concrete rather than abstract.
The Trouble with Testing Many Things at Once
Let's build some intuition first, before touching any formulas. Imagine we have a coin that we strongly suspect is fair, and we want to test this. We flip it a number of times and run a statistical test; if the p-value is below 0.05, we would conclude the coin looks biased. Because the significance threshold is 0.05, there is a 5% chance that a perfectly fair coin would still be flagged as biased, purely due to random chance.
Now imagine we don't test one coin, but twenty different fair coins, each with its own test. Each individual test still has a 5% chance of falsely flagging a coin as biased. But across twenty tests, the chance that at least one of them is falsely flagged is no longer 5%; it climbs substantially higher. We are no longer asking "what is the probability that this one test is wrong?", but "what is the probability that at least one out of many tests is wrong?", and that is a very different question.
Family-Wise Error Rate
Statisticians call the probability of making at least one false positive across a whole set of tests the family-wise error rate (FWER). If each individual test uses a threshold of 0.05, the family-wise error rate grows with the number of tests performed, and can end up far higher than 5%.
To make this concrete, if we run \(n\) independent tests, each with a false positive rate of \(\alpha\) (usually 0.05), the probability of getting no false positives at all across all of them is \((1 - \alpha) ^ n\). The probability of getting at least one false positive is therefore:
\[ P(\text{at least one false positive}) = 1 - (1 - \alpha) ^ n \]
For a single test (\(n = 1\)), this is simply 0.05, as expected. But for 20 independent tests, this probability rises to about 0.64, meaning we would have roughly a 64% chance of seeing at least one "significant" result purely by chance, even if nothing genuinely differs anywhere in our data.
The Bonferroni Correction
The Bonferroni correction is a simple, direct way to control the family-wise error rate when running several tests at once. The idea is straightforward: rather than judging every test against the usual threshold \(\alpha\) (typically 0.05), we divide \(\alpha\) by the number of tests \(n\), and use this stricter threshold instead:
\(\alpha_{\text{Bonferroni}} = \frac{\alpha}{n}\)
For example, if we were to run 10 independent tests, the adjusted threshold would become:
\(\alpha_{\text{Bonferroni}} = \frac{0.05}{10} = 0.005\)
Only p-values smaller than 0.005 would then be considered statistically significant. This is a much stricter bar than 0.05, and that is exactly the point: since we are giving ourselves 10 separate chances to find a false positive, each individual chance needs to be held to a stricter standard so that the overall chance of a false positive across all 10 tests stays close to 5%.
An intuitive way to think about this division is that the significance level we choose (e.g., 5%) represents our full "budget" for making a false positive. When we run a single test, that test gets to spend the entire budget. When we run several tests at once, the Bonferroni correction simply splits that same budget equally across all of them, so that no single test is allowed to spend more than its fair share.
Two Equivalent Ways to Apply Bonferroni
The Bonferroni correction can be applied in two mathematically equivalent ways: either by dividing the significance threshold \(\alpha\) by the number of tests \(n\) and comparing the original p-values against this stricter threshold, or by multiplying each individual p-value by \(n\) and comparing the adjusted p-values against the original threshold of 0.05. Both approaches lead to exactly the same conclusions. R reports the adjusted p-values directly, capping any that exceed 1 back down to 1.
Bonferroni is Conservative
The Bonferroni correction is known to be a fairly strict, or conservative, method. It guarantees that the overall family-wise error rate stays at or below \(\alpha\), but it does so by making it harder for any individual test to be considered significant, which increases the risk of missing real effects (a false negative)—especially as the number of tests grows very large. When dealing with dozens or hundreds of tests, alternative methods such as the Holm correction or False Discovery Rate (FDR) methods are often preferred, since they retain some control over false positives while being somewhat less strict.
A Practical Example: Comparing Life Expectancy Across Continents
To see this in action, let's use the gapminder dataset, a well-known public dataset that contains life expectancy, population, and GDP per capita for many countries, tracked across several decades (Bryan, 2023). We'll focus on the year 2007, the most recent year included in the dataset, and look at the variable lifeExp, which reflects the average life expectancy of each country.
Let's load the necessary libraries, filter the dataset down to the year 2007 (the most recent year available), and use the head() function to review the first six rows:
# Libraries
library(tidyverse)
library(gapminder)
# Keeping only the most recent year available
gapminder_2007 <- gapminder %>% filter(year == 2007)
# Printing the first few rows
head(gapminder_2007)# A tibble: 6 × 6
country continent year lifeExp pop gdpPercap
1 Afghanistan Asia 2007 43.8 31889923 975.
2 Albania Europe 2007 76.4 3600523 5937.
3 Algeria Africa 2007 72.3 33333216 6223.
4 Angola Africa 2007 42.7 12420476 4797.
5 Argentina Americas 2007 75.3 40301927 12779.
6 Australia Oceania 2007 81.2 20434176 34435. Each country belongs to one of five continents: Africa, Americas, Asia, Europe, and Oceania. A natural question to ask is whether life expectancy differs, on average, between continents. Since we have five continents, there are actually 10 different possible pairs we could compare—Africa vs Europe, Africa vs Asia, Asia vs Oceania, and so on. If we wanted to compare every continent to every other continent, we would need to run 10 separate, independent t-tests.
Let's actually run all 10 t-tests, comparing life expectancy between every pair of continents, and collect the resulting p-values. An easy way to do this is to use the function pairwise.t.test, which automatically runs a t-test for every possible pair of groups in a single call, saving us from writing a separate t.test() call for each pair by hand. We set the argument p.adjust.method to "none", so that the function returns the raw, uncorrected p-values—exactly as if we had run each t-test on its own, without accounting for the fact that several tests are being performed together. We also pipe the result straight into tidy(), from the broom package, which converts the function's default matrix-style output into a clean data frame that's much easier to read and sort:
# Libraries
library(broom)
# Running a t-test for every pair of continents, without any p-value adjustment
results <- pairwise.t.test(
gapminder_2007$lifeExp,
gapminder_2007$continent,
p.adjust.method = "none",
pool.sd = FALSE
) %>%
tidy() %>%
arrange(p.value) %>%
mutate(p.value = round(p.value, 10))
# Printing results
results# A tibble: 10 × 3
group1 group2 p.value
1 Europe Africa 0
2 Americas Africa 0
3 Oceania Africa 0
4 Asia Africa 0
5 Oceania Asia 0.000000382
6 Oceania Americas 0.0000197
7 Europe Asia 0.0000339
8 Europe Americas 0.000375
9 Oceania Europe 0.0129
10 Asia Americas 0.0862 By default, pairwise.t.test() sets pool.sd = TRUE, pooling a single standard deviation across all five continents instead of computing one per pair. We set it to FALSE to keep things simple, essentially running each comparison as an ordinary, independent two-sample t-test.
At the usual threshold of (\(\alpha\) = 0.05), nine out of these ten comparisons come out statistically significant, including the pair Europe vs Oceania, with a p-value of about 0.013. Only the Americas vs Asia comparison fails to reach significance. On the surface, this looks like fairly convincing evidence that life expectancy genuinely differs between almost every pair of continents.
But remember what we established earlier: when we run 10 tests at (\(\alpha\) = 0.05) each, the probability of finding at least one false positive by chance alone is no longer 5%. It is:
\[ 1 - (1 - 0.05)^{10} \approx 0.40 \]
In other words, even in a world where none of these continent comparisons reflected a real underlying difference, we would have roughly a 40% chance of stumbling on at least one "significant" result purely due to random sampling. Given that we found nine significant results here, this is unlikely to be entirely due to chance. Still, the borderline case, Europe vs Oceania at (p = 0.013), is exactly the kind of result multiple testing should make us doubt.
Let's apply the Bonferroni correction to these 10 p-values, using R's built-in p.adjust() function:
# Applying the Bonferroni correction to our p-values
results$p_adjusted <- p.adjust(results$p.value, method = "bonferroni")
results# A tibble: 10 × 4
group1 group2 p.value p_adjusted
1 Europe Africa 0 0
2 Americas Africa 0 0
3 Oceania Africa 0 0
4 Asia Africa 0 0
5 Oceania Asia 0.000000382 0.00000382
6 Oceania Americas 0.0000197 0.000197
7 Europe Asia 0.0000339 0.000339
8 Europe Americas 0.000375 0.00375
9 Oceania Europe 0.0129 0.129
10 Asia Americas 0.0862 0.862 Notice what happens to the ninth row, Europe vs Oceania. Its original p-value of 0.013 was below the usual 0.05 threshold, so it would have been flagged as statistically significant on its own. After the Bonferroni correction, its adjusted p-value rises to about 0.129—well above 0.05—and this comparison is no longer considered statistically significant. The other eight comparisons remain comfortably significant even after the correction, since their original p-values were already extremely small.
This is precisely the value of the correction: it protects us from being misled by the one borderline result that likely surfaced due to running so many comparisons, while leaving the genuinely strong results untouched. If we had only tested Europe against Oceania in isolation, without considering it as one of ten simultaneous comparisons, we would have walked away believing we had solid evidence of a difference, when in fact this result does not hold up once we account for the multiple testing.
Closing
Running many statistical tests at once quietly works against us, inflating the chance of a false positive well beyond the 5% we expect from a single test. A result that looks convincing in isolation may not hold up once we account for how many other comparisons were made alongside it, which is exactly the gap the Bonferroni correction is designed to close.
The broader lesson is that the meaning of a p-value depends on how many other tests were run alongside it. Bonferroni is a simple, conservative way to account for this, though not the only one; methods, such as Holm, trade off some of that caution for more power when many comparisons are involved. Either way, whenever an analysis tests several hypotheses at once, it is worth asking not just what one p-value says, but what it says in the context of everything else being tested alongside it.
References
Bonferroni, C. E. (1936). Teoria statistica delle classi e calcolo delle probabilità [Statistical theory of classes and calculation of probability]. Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze, 8, 3–62.
Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical Association, 56(293), 52–64. https://doi.org/10.1080/01621459.1961.10482090
Bryan, J. (2023). gapminder: Data from Gapminder (R package version 1.0.0). https://CRAN.R-project.org/package=gapminder