April 14, 20266 min read

The Invisible Filter That Decides Whether Healthcare AI Really Works

BiostatisticsClinical TrialsMulti-Agent AIBioinformaticsFDR

Diego Muñoz Casinos

Healthcare AI Engineer

KEGG pathways split by direction of regulation — up- vs. down-regulated proteins. A common format when you want to see at once what goes up and what goes down.

Loading charts...

FWER curve · α = 0.05

FWER as a function of the number of comparisons

Hover over the area to see exact values

Loading chart...

FWER · Family-Wise Error Rate

Probability of at least one false positive

Grows with the number of simultaneous comparisons, under α = 0.05

1 test5 %

20 tests64 %

100 tests99 %

10,000 tests~100 %

500

Expected false positiveswith 10,000 simultaneous tests

P(≥ 1 FP) = 1 − (1 − α)^k · α = 0.05

Loading chart...

Multiplicity control · Three approaches

Three ways to control error under multiple comparisons

FWER

The underlying problem

Problem

P(≥ 1 FP) = 1 − (1 − α)ᵏ

The probability of committing at least one false positive grows rapidly with the number of simultaneous tests k. For large k, per-test control is no longer enough.

Bonferroni

Classical FWER control

Strict

α' = α / m

Divides the threshold α by the number of hypotheses m. Robust under any dependence structure. Ideal for confirmatory analyses with few pre-specified hypotheses, at the cost of reduced power.

Benjamini–Hochberg

FDR control

Balance

P(i) ≤ (i / m) · α

Orders p-values from smallest to largest and accepts more rejections in exchange for controlling the expected false discovery proportion. Modern standard in omics discovery.

α = nominal threshold · m = number of hypotheses · k = number of tests · i = rank of ordered p-value

Loading formulas...