AI brain MRI labeling: How LLM consensus improves accuracy

How does LLM consensus in healthcare improve data accuracy?

A team at New York University released a preprint showing that a consensus of large language models (LLMs) can reliably assign normal or abnormal labels to brain MRI reports. In their test, 680 reports had been manually adjudicated by experts—372 normal, 90 with only white matter hyperintensities (WMH‑only), and 218 abnormal. The LLM ensemble flagged a report as normal or abnormal only when every model agreed; otherwise it was marked uncertain and sent to a human reviewer. This approach cut the number of ambiguous cases that required manual work and produced a high‑confidence set of labels for research cohorts. The authors note the method could streamline retrospective studies that rely on unstructured imaging notes.

Can automated medical record analysis replace manual review?

The researchers used a retrospective cohort design, pulling 6,718 brain MRI reports from 5,856 patients. They compared two ensembles: a proprietary‑inclusive set (Qwen3, Kimi‑Linear, GPT‑5) and an open‑source set (Qwen3, Kimi‑Linear, DeepSeek‑R1). Each model was run independently, and consensus was defined as unanimous agreement on normal or abnormal. The reference set of 680 reports was created by expert clinicians who adjudicated outcomes from the reports. The study measured how often the ensembles matched the expert labels and how many reports were flagged as uncertain.

What is the role of brain MRI report classification in clinical research?

Because this is a preprint, the findings have not yet undergone peer review, so the methodology and conclusions are preliminary. The study shows correlation between LLM consensus and expert labels, but it does not prove causation or that the method works in every clinical setting. The work also does not address how the approach would perform on other imaging modalities, in different institutions, or with more diverse patient populations. Finally, the ensemble still defers a substantial portion of cases to human review; it does not eliminate manual adjudication entirely.

Implications of LLM consensus for researchers and clinicians

For researchers, the study offers a way to build large, high‑confidence cohorts from unstructured radiology reports without scaling manual chart review. Clinicians might see similar tools applied to clinical decision support, though the current method is not ready for routine patient care. The key advantage is that the ensemble can flag ambiguous reports for further scrutiny, potentially improving data quality in retrospective studies. However, users should remain aware of the preprint status and the need for institutional validation before deployment.

Future directions for AI brain MRI labeling

The authors plan to test the consensus workflow on additional datasets, including other imaging types and institutions with different reporting styles. They also aim to refine the uncertainty threshold and explore hybrid models that combine LLMs with structured data. Peer reviewers will assess the methodology, and if the findings hold up, the approach could be integrated into research pipelines and possibly into clinical audit tools. The next step is to move from a proof‑of‑concept to a production‑ready solution, with rigorous validation and governance.

Key takeaways from LLM‑enhanced brain MRI labeling

- A consensus of LLMs can produce reliable normal/abnormal labels for brain MRI reports.

- The approach uses 680 expert‑labeled reports as a gold standard.

- It reduces the number of cases needing manual review but still defers many.

- Results are from a preprint; further validation is needed.

- The method may accelerate retrospective cohort construction when adopted.