#143 - John Ioannidis, M.D., D.Sc.: Why most biomedical research is flawed, and how to improve it

KEYPOINTS

• 01

  Evidence-based medicine is young.

  John Ioannidis, a Stanford physician-scientist trained in mathematics and medicine, describes a field that until recently ran on expert opinion. He met Thomas Chalmers, a pioneer of randomized trials and meta-analysis, in 1992, about five years before Chalmers died, and credits him with showing how to weigh evidence systematically.

• 02

  Most published research findings are probably false.

  His 2005 paper (close to 10,000 citations, though not his most cited; the PRISMA statement for meta-analyses has more) argued that with small studies, small effects, flexible analyses, bias and many teams chasing the same question, a nominally significant finding at p just under 0.05 has a high chance of being a red herring.

• 03

  Significance thresholds should be stricter.

  Ioannidis supports using p < 0.005 instead of 0.05 for new claims, noting most scientists are trained 'statistics light'. Large databases can produce astronomically small p-values (10^-100), so p-values alone say little about whether an effect matters or is real.

• 04

  Genetics fixed its reproducibility problem; nutrition hasn't.

  Genetics measures exposures with high accuracy (genotyping error under 0.01), nobody has ideological attachments to a polymorphism, and the field adopted genome-wide significance of 10^-8 plus replication. Nutritional epidemiology still relies mostly on post-hoc, unregistered analyses with p = 0.05 thresholds and noisy dietary measurement.

• 05

  Real large effects look like tobacco.

  Smoking studies in the 1950s–60s consistently showed about a 10x, on average about 14x, higher lung-cancer risk across physicians and non-physicians alike. Most nutrition findings are odds ratios like 1.14, a 14% relative increase, not 14x. Ioannidis says that if he ever sees an odds ratio of 20 in his own calculations, he assumes he made a mistake; genetics often deals in real effects of 1.01.

• 06

  Bradford Hill's criteria still guide causal inference.

  Attia and Ioannidis walk through the classic criteria using smoking: strength of association (a roughly 14x risk), consistency (every 1950s–60s series pointed the same way), specificity, temporality and dose-response. Specificity is rarely met for single risk factors, since smoking causes many diseases, but the overall pattern made the tobacco case overwhelming. Nutrition findings usually fail on strength and consistency, and the small effects they report are easily produced by residual confounding, measurement error and selective reporting.

• 07

  Nutrition needs more and larger randomized trials.

  Thousands of nutrition RCTs exist, but most are small and underpowered; there are over 200 large ones, mostly on specific nutrients or supplements. Ioannidis wants more large, long-term trials, and sees Mendelian randomization as a useful quasi-randomized design with its own assumptions.

• 08

  Consortia can bake in belief.

  When investigators pool data, invitations often go to the roughly 100 teams that already believe in an association rather than the 3,000 who might have looked, so the cleaned, combined result confirms the prior. Pre-existing beliefs shape which questions get asked and how data are analyzed.

• 09

  Funding should tolerate failure.

  Ioannidis sees distinct roles for public, private and philanthropic funders. Public and philanthropic money is best placed to back high-risk research asking important questions that industry won't fund, but only if funders accept that most projects will fail. Incentives that reward positive, flashy results push researchers toward questionable practices, from p-hacking to selective reporting.

• 10

  Bias is nearly universal; transparency is the antidote.

  Ioannidis thinks bias may approach 100% if defined broadly, often without researchers being aware. Remedies include registration, data sharing, replication and reporting null results; funders should accept that perhaps five of 1,000 grants find something important and that's what makes the difference.

• 11

  Even top trials can be flawed: PREDIMED.

  The Mediterranean-diet trial PREDIMED, published around 2013, looked like strong evidence until it emerged that randomization had been subverted at some sites; the New England Journal of Medicine retracted and republished it with new analyses. It's now partly randomized and partly observational, with hundreds of spin-off publications making varied claims.

• 12

  COVID science became hyper-politicized.

  Ioannidis' early 2020 seroprevalence study suggested infections were about 50 times more common than documented cases, which drew intense animosity. He emphasizes a very steep risk gradient: many people have minimal risk, while nursing home residents can face an infection fatality rate around 25%.