Supplying the radiology report cuts how often a swapped scan changes the answer

single source· 1 articles · confidence: medium · first seen 2026-09-13 20:00 UTC

What this means for you

If you evaluate a medical vision-language model against report-derived labels, accuracy alone will not tell you whether it used the image. The audit code, exact prompts and per-number run record are public, so the swap test can be repeated on your own data. This is an evaluation method, not a released model.

A paired audit of a medical vision-language model (software that reads an image and text together) finds that giving the model the written radiology report makes it much less responsive to the image itself. ModaLens replaced each chest X-ray with one from another study, holding the question and report fixed. Under an explicit answer instruction, MedGemma-27B's answer changed on 4.26% of trials with the report and 20.94% without — a paired 16.7-point gap (95% CI 15.6–17.7) across 3,199 MIMIC-CXR cases from 293 patients. The authors say their labels come from reports, which limits conclusions about visual correctness. The direction replicated in two other model lineages.

Key facts

  • ·Under an explicit answer instruction, MedGemma-27B's generated answer changed on 4.26% of image-swap trials with the report present and 20.94% without it. source
  • ·The paired increase was 16.7 points, with a patient-clustered 95% confidence interval of 15.6 to 17.7. source
  • ·The audit used 3,199 paired MIMIC-CXR cases from 293 patients, with 14 questions asked per case (13 finding-specific and one composite). source
  • ·Under the original prompt with a lowercase first-token readout, the figures were 4.70% of trials with the report and 17.07% without. source
  • ·The authors state that labels derived from reports limit conclusions about visual correctness, and that the direction replicated in two further model lineages. source
  • ·Code, the exact prompts and a run record for every number are published at github.com/criticaldata/MODALENS. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire