The Invisible Filter That Decides Whether Healthcare AI Really Works

In proteomic and functional enrichment analyses, thousands of hypotheses are tested simultaneously against the same data. Without multiple comparisons correction (Benjamini–Hochberg, Bonferroni, Holm), a non-trivial fraction of findings are statistical noise indistinguishable from real signal. The same principle is a standard piece of Statistical Analysis Plans in confirmatory clinical trials under ICH E9. Author's personal reflection: it could extend to the design of multi-agent AI workflows — an open conjecture, not a thesis.

A familiar scenario in clinical proteomics

Imagine a research team analyzing samples from patients with a disease, trying to identify which biological processes are altered. They use mass spectrometry to measure thousands of proteins, compare levels between patients and controls, and get a list of — say — 300 proteins exceeding the nominal significance threshold.

The next step is functional enrichment analysis: asking a biological pathway database (Gene Ontology, KEGG, Reactome) which functions are over-represented in that list. The typical output is another list: "30 significant pathways" with their associated p-value. How many of those 30 are real findings? The principle that answers this question is cross-cutting: it appears every time many hypotheses are evaluated at once on the same data — biomarker panels, secondary trial endpoints, feature selection in machine learning.

Why 10,000 comparisons generate 500 false findings by pure chance

Each comparison against a biological pathway is a statistically independent test in its formal setup, though in practice tests over GO, KEGG, or Reactome pathways are correlated due to gene overlap between pathways. That correlation does not eliminate the underlying problem: the more comparisons you make, the more likely you are to find something "significant" by pure chance, and the phenomenon has been known in biostatistics for decades.

The math is simple. If we accept a classical threshold of α = 0.05 as the nominal false positive rate conditioned on the null hypothesis being true, the probability that at least one of several tests falls in the rejection region by pure chance grows with the number of tests:

• •With 1 test: 5%
• •With 20 tests: approximately 64%
• •With 100 tests: over 99%
• •With 10,000 tests: virtually 100%, with an expected 500 false positives if all null hypotheses were true

Here it is useful to distinguish two different metrics: the probability of committing at least one false positive (the family-wise error rate or FWER) and the expected proportion of false positives among findings declared significant (the false discovery rate or FDR). They are different questions with different answers, and in omics you almost always care about the second. The consequence is serious: if you stick with raw p-values without any adjustment, part of what you declare as findings is not.

The multiplicity problem has been well documented in the statistical literature for more than half a century — Tukey 1953, Dunn 1961, Miller 1966 among others. FDR control specifically arrived with Benjamini–Hochberg in 1995². This is not a critique of any particular line of research, but an observation about a mathematical phenomenon.

From FWER to FDR: how biostatistics contained the problem

In 1995, Yoav Benjamini and Yosef Hochberg published a paper that changed how this problem is handled in omics². Instead of trying to avoid any false positive (the classical approach, called family-wise error rate control), they proposed controlling the expected proportion of false discoveries among the results declared significant. That metric is called False Discovery Rate or FDR.

The idea is subtle but powerful: if the team fixes an acceptable false discovery threshold — typically 5% or 10%, depending on context — they can afford to be less strict on each individual test without losing global control over error. The result is a larger number of rejections at the cost of a controlled false discovery rate, without implying that each individual rejection is a true finding in a material sense. The BH procedure controls FDR under independence of tests or under a form of positive dependence known as PRDS (positive regression dependency on subsets); when the dependence structure is arbitrary, the Benjamini–Yekutieli variant (2001) provides control at the cost of being more conservative.

A few years later, John Storey and Robert Tibshirani extended the idea with a practical tool: the q-value, defined as the minimum FDR at which a specific test would be declared significant, additionally estimating the proportion of true null hypotheses (π₀) from the data themselves³. In practice, modern enrichment analysis tools (GSEA, g:Profiler, DAVID) report FDR-adjusted values alongside the raw p-value^1,5^.

Bonferroni still exists and has its place: it is mathematically stricter and useful in confirmatory analyses with few pre-specified hypotheses — for example, a clinical trial with five primary endpoints fixed before looking at the data. A sequential variant, Holm-Bonferroni, offers the same FWER control under any dependence structure, without additional assumptions, and with slightly less power loss than classical Bonferroni⁷. In discovery contexts with thousands of tests, FDR has become the standard for good reasons⁴.

The choice between one framework or another depends on context: exploratory vs. confirmatory, pre-specified hypotheses vs. ones generated from the data, consequences of a false positive vs. consequences of a false negative. There is no universal answer. There is a clear gradient: the closer the analysis is to a regulatory or clinical decision, the more justifiable it is to lean toward the strict end of the spectrum.

The same problem in clinical trials

So far the example has been proteomic, but the principle is the same in any domain where the volume of simultaneous hypotheses conditions the interpretation of results. What strikes me most is clinical trials, because correction there stops being a methodological matter and becomes part of what is actually being validated before a regulator.

A typical protocol defines one primary endpoint and several secondary endpoints. Each additional endpoint evaluated on the same patients is another test. If a secondary endpoint turns out "significant" without having been pre-specified hierarchically with an explicit correction, the result cannot be interpreted as if it had been a single endpoint: the cumulative probability of a false positive already includes the search. Confirmatory analyses with multiple endpoints combine hierarchical testing procedures (fixed-sequence, gatekeeping) with classical FWER controls like Bonferroni or Holm-Bonferroni⁷. In interim analyses of group sequential designs, alpha-spending methods explicitly pay the statistical cost of peeking at the data early. Subgroup analyses are different: except when pre-specified hierarchically in the analysis plan, they remain exploratory findings even if adjusted for multiplicity, and must be reported as such.

Three useful questions for anyone evaluating a multi-hypothesis analysis

For anyone evaluating the validity of an analysis that compares many hypotheses simultaneously — an omics analysis, a clinical trial with several endpoints, an ML pipeline with feature selection — there are three questions worth having on the table, and none of them require advanced statistical training.

• 1.How many hypotheses are actually being evaluated, including secondary, exploratory, and those derived from subgroup analysis?
• 2.Does the report show an adjusted p-value (FDR, q-value, Bonferroni, Holm) or only the raw p-value?
• 3.Was the correction criterion fixed before looking at the data, or was it chosen after seeing the results?

The answers are almost always in the methods section of the associated paper and in the statistical analysis plan of the protocol.

The criteria I apply in the omic pipelines I work with

In the multi-omic analysis pipelines I have worked on — both during my bioinformatics master's and in research projects — I keep these criteria as operational habits before signing off on any functional enrichment output:

• •Always report the q-value or FDR-adjusted p-value alongside the raw p-value in any enrichment output.
• •Fix the correction threshold before looking at the results, not after.
• •Explicitly separate exploratory analysis (where FDR is appropriate) from confirmatory analysis with pre-specified hypotheses (where FWER control is the regulatory standard under ICH E9).
• •Document the correction method and justifications for the choice in the deliverable itself, so it is traceable by any external reviewer.
• •Differentiate the set of tests actually run from the set reported: an analysis with "only ten significant pathways at the end" may hide thousands of background comparisons.

None of these practices is novel. FDR in differentially expressed gene selection has been documented since the early years of large-scale gene expression⁵ and recent methodological literature treats them as minimums⁴. What I find interesting as a systems engineer is that, in an automated pipeline, these decisions stop being the responsibility of the individual analyst and become procedural controls applied reproducibly. Inferential validity still depends on the model, the assumptions, and the real power of the analysis, but at least the process becomes auditable.

Reflection: from adjusted p-value to multi-agent workflows

What interests me lately, more as a systems engineer than as a statistician, is applying the same principle to the design of multi-agent AI workflows. A workflow with N agents evaluating hypotheses sequentially or in parallel generates multiple decisions with uncertainty, and the uncertainty of each individual agent and that of the aggregate workflow are different magnitudes. Without a control mechanism equivalent to FWER or FDR for the complete chain, the system can accumulate error without anyone noticing — just like an omics analysis without correction. There is contemporary literature that tackles pieces of the problem from different angles (conformal prediction, hierarchical Bayesian modeling), but I have not seen it articulated as a design discipline for multi-agent pipelines. This is an open conjecture, not a thesis: I offer it as a personal reflection with the hope it serves to open conversation rather than close it. If anyone is working on this kind of metrics over a real workflow, I would be interested to see the numbers.

Closing

"In healthcare, upstream statistical rigor is not a luxury. It is the filter between a system with signal and a system with noise elegantly repackaged."

The formal multiplicity problem has had a statistical solution for over three decades, implemented by default in almost all contemporary tools. What is never automatic is the discipline of applying it consistently along a full pipeline: from the individual test to the report a committee reviews, passing through the model that consumes the selected features. A pipeline that respects that chain is not more complex. It is simply more honest with the data it consumes and with the decisions it feeds downstream.

Peer-reviewed references

• 1.Subramanian A, Tamayo P, Mootha VK, et al. Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles. Proc Natl Acad Sci U S A. 2005;102(43):15545–15550. doi:10.1073/pnas.0506580102. PMID: 16199517.
• 2.Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Ser B. 1995;57(1):289–300.
• 3.Storey JD, Tibshirani R. Statistical significance for genomewide studies. Proc Natl Acad Sci U S A. 2003;100(16):9440–9445. doi:10.1073/pnas.1530509100. PMID: 12883005.
• 4.Goeman JJ, Solari A. Multiple hypothesis testing in genomics. Stat Med. 2014;33(11):1946–1978. doi:10.1002/sim.6082. PMID: 24399688.
• 5.Reimand J, Isserlin R, Voisin V, et al. Pathway enrichment analysis and visualization of omics data using g:Profiler, GSEA, Cytoscape and EnrichmentMap. Nat Protoc. 2019;14(2):482–517. doi:10.1038/s41596-018-0103-9. PMID: 30664679.
• 6.Varoquaux G, Cheplygina V. Machine learning for medical imaging: methodological failures and recommendations for the future. NPJ Digit Med. 2022;5(1):48. doi:10.1038/s41746-022-00592-y. PMID: 35413988.
• 7.Holm S. A simple sequentially rejective multiple test procedure. Scand J Stat. 1979;6(2):65–70.
• 8.Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005;2(8):e124. doi:10.1371/journal.pmed.0020124. PMID: 16060722.
• 9.Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, Robinson ESJ, Munafò MR. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013;14(5):365–376. doi:10.1038/nrn3475. PMID: 23571845.
• 10.Wacholder S, Chanock S, Garcia-Closas M, El Ghormli L, Rothman N. Assessing the probability that a positive report is false: an approach for molecular epidemiology studies. J Natl Cancer Inst. 2004;96(6):434–442. doi:10.1093/jnci/djh075. PMID: 15026468.

This article is educational in nature. It does not substitute for consultation with a biostatistician or clinical specialist for specific applications. All claims are backed by peer-reviewed literature; the references listed are the primary support.