269 - Good vs. bad science: how to read and understand scientific studies

KEYPOINTS

• 01

  A good study is planned in detail before anyone is enrolled.

  Peter and Bob Kaplan walk through the steps. Start with a hypothesis framed as a null hypothesis: coffee does not change eye color. Design the experiment, ideally randomized and blinded. Use a power analysis to set the number of subjects. Get ethics (IRB) approval, name primary and secondary outcomes, write a statistical plan, pre-register, and secure funding. Treat the hypothesis as 'innocent until proven guilty' and try hard to falsify it.

• 02

  Study types range from anecdote to experiment, and Peter disputes the standard evidence pyramid.

  Case reports and case series generate hypotheses. Peter's own early case report described a melanoma patient whose dangerously high calcium came from a tumor-made parathyroid-hormone-related hormone, not bone metastases. Cohort studies can look backward or forward but are only observational. Non-randomized trials carry selection bias, and randomized controlled trials are the gold standard. Meta-analyses pool studies, weighting each by its precision. Peter rejects the usual pyramid that ranks a meta-analysis above a well-done randomized trial.

• 03

  Drug trials move through four phases.

  Phase 1 tests safety in fewer than about 100 people, often patients with advanced cancer who have run out of options, raising doses step by step in small cohorts. Phase 2 adds efficacy in tens to a few hundred people, often open-label and compared against the disease's natural course. Phase 3 is typically ten times larger: thousands of people, randomized and blinded where possible, measured against placebo or standard care. It's the main basis for approval. Phase 4 follows approval, for example testing semaglutide for obesity after it was approved for diabetes, which can reveal side effects in new populations.

• 04

  Observational studies are dogged by healthy-user bias, recall bias and confounding.

  The WHO's reading of red and processed meat studies found about a 20–30% higher associated risk. But people who avoid meat also tend to smoke less, exercise more and follow medical advice more closely. The same logic explains studies claiming diet soda is more fattening than regular soda: people who are becoming metabolically ill get told to switch. Hot chocolate 'causes' ski accidents only because cold climates drive both. Peter would scrap food frequency questionnaires, which ask people to recall a year of eating, often once at baseline, and which validation studies show correlate poorly with food logs. Recall works better for unforgettable events like pregnancies.

• 05

  Randomized trials have their own traps.

  Performance bias, a relative of the Hawthorne effect, arises when one arm gets more attention, coaching or contact; Peter calls it the most common bias in lifestyle trials. Taking many looks at the data inflates false positives. Ten coin flips almost guarantee at least one heads, so a Bonferroni correction divides the p-value threshold by the number of looks. In PREDIMED, a review of thousands of trials by John Carlisle flagged odd randomization. It turned out entire clinics had been assigned to one diet. The results held on reanalysis, but a cloud remains. Also check blinding (psychedelic trials are nearly impossible to blind), whether the subjects were all men, funding, conflicts of interest and whether the effect matters clinically.

• 06

  Power determines whether a null result means anything.

  The p-value threshold (alpha, usually 0.05) caps false positives. Power is 1 minus the false-negative rate, typically 80–90%. Using a classic table, Peter shows that detecting a rise from 25% to 35% success needs about 358 people per arm at 80% power, or 459 at 90%. A 25% versus 75% gap needs about 18–23, and 25% versus 30% needs roughly 1,300–1,700. Underpowered studies miss real effects. Overpowered ones, common in data-mining studies, make trivial effects 'significant,' like a drug that lowers systolic pressure by 1 mmHg.

• 07

  Always ask about absolute risk, not just relative risk.

  The Women's Health Initiative's widely reported 25% rise in breast cancer on hormone therapy was 4 versus 5 cases per 1,000 women: an absolute increase of 0.1%. Peter calls that a ceiling on the true risk, given what he sees as the study's many flaws. Number needed to treat is 1 divided by the absolute risk reduction. Cutting events from 5 to 4 per 1,000 is a 20% relative reduction but means treating 1,000 people to prevent one event. Clinicians like to see an NNT under 100.

• 08

  Hazard ratios and confidence intervals are easy to read once you practice.

  A hazard ratio minus 1 gives the relative change: 0.82 means 18% lower risk and 2.2 means 120% higher. A 95% confidence interval that crosses 1 isn't significant. Bob cites statistician Andrew Gelman, who prefers 'uncertainty interval.' A hazard ratio of 1.4 with an interval of 1.1–17 deserves little confidence. Trials stop early for three reasons. Torcetrapib was stopped for safety after 82 deaths versus 51 on Lipitor alone. PREDIMED was stopped for benefit. Look AHEAD was stopped for futility, at a hazard ratio of 0.95 with an interval of 0.83–1.09.