Automating Oncology Registry Abstraction with LLMs

How does LLM clinical data extraction improve registry accuracy?

Open-source, general-purpose large language models are capable of facilitating registry-level oncology data extraction within clinical production settings. A study by Bond University, released as a preprint on October 21, 2026, demonstrates that this technology can successfully structure oncology clinical notes into registry-grade variables. By automating the process, this method mitigates the inaccuracies and high labor requirements inherent in manual registry abstraction, a task that usually necessitates the review of dense, longitudinal, narrative clinical records encompassing staging, biomarkers, treatment lines, diagnoses, and clinical reasoning.

The system extracts clinically meaningful text directly from notes, while manual mapping to standardized systems like ICD-10, ICD-O-3, or RxNorm occurs as a separate step. Researchers concluded that a structure-aware evaluation approach provides more clinically actionable insight than aggregate metrics by separating omissions from fabrications.

How is patient privacy maintained during processing?

The Hide-In-Plain-Sight (HIPS) method preserves patient confidentiality by guaranteeing that no sensitive information exits the hospital's internal infrastructure. Before data enters the cloud for processing, the system performs HTML flattening and de-identification on-premises. During this process, the system replaces sensitive entities with realistic but fictitious equivalents.

Deployment of this pipeline is compatible with GCP, Azure, AWS, or clusters located on-premises.

How does the AI abstraction pipeline architecture function?

The abstraction pipeline relies on the Meta Llama 3.3 70B model, which functions as a multi-step, modular prompting pipeline. The system performs evidence collection, classification, and mapping to registry fields without the use of parameter fine-tuning. The system depends solely on the model's inherent parametric knowledge, eschewing the use of site-specific gazetteers, brand-name dictionaries, or localized lookup tables.

Using this method, the system processed 6,700 clinical notes to generate the Well Characterized Repository (WCR). This repository contains over 50 oncology variables extracted directly from the notes.

Can LLMs handle complex clinical note structuring?

The system performance was assessed using 200 randomly sampled clinical notes with adjudicated ground truth. This evaluation framework classifies model outputs into four categories: Correct, Missing, Spurious, and Incorrect. The system scores scalar and binary fields at the note level, while it evaluates list-valued fields, such as treatment drugs and biomarker test results, at the item level. The pipeline is built to handle the temporal interpretation of longitudinal notes, including the distinction between historical and active disease, first and second primary cancers, and treatment sequencing.

The model achieved an F1 score of 97.2% for First Primary Diagnosis, with only three incorrect extractions out of 158 cases where the field was present. For First Primary Histology, the F1 score was 95.8%, characterized by five erroneous values and absolute specificity, as the model refrained from generating histology outputs in instances where they were not required. Overall Stage achieved an F1 score of 92.6%, with errors consisting of six missed stages and eight spurious outputs.

For list-valued variables, the model reached an F1 score of 91.4% for Biomarker Test Results, with 95.0% precision and 88.0% recall. This means the model was almost always correct when it extracted a biomarker, though it missed approximately one in eight results. The F1 score for Treatment Drugs was 88.1%, with 55 false extractions and 40 omissions identified among 103 cases that included mentions of drugs.

Can this model be transferred to other hospitals?

Initially, the pipeline was implemented at a cancer centre in Singapore to retrieve more than 50 variables from a dataset of 6,700 oncology notes. Although secure multi-party computation represents a branch of cryptography enabling parties to perform functions on inputs while maintaining their privacy, this system currently utilizes local de-identification prior to cloud-based processing. To perform exploratory testing, the identical pipeline and prompt template were applied to 50 oncology notes sourced from a large private hospital in Taiwan.

On the Taiwan test set, the model achieved 100% F1 scores for both Diagnosis and Histology. Performance for other variables included an F1 score of 88% for Overall Stage, 83% for Treatment Drugs, and 86% for Biomarker Test Results. Variations in brand-name usage and prescribing conventions between Taiwan and Singapore are responsible for the reduced performance regarding drug names within the Taiwan dataset.

The authors note that this external set provides only an exploratory transferability signal. They argue that reliable cross-site validation necessitates significantly larger external cohorts that encompass diverse documentation styles and multiple institutions. Future work will prioritize these larger cohorts, including the calculation of per-field confidence intervals.

Why choose prompted models over fine-tuned systems?

The researchers opted for a general-purpose model instead of a smaller, fine-tuned system. While fine-tuned models can match or outperform general-purpose LLMs on specific, static tasks, prompted models offer greater flexibility for a production registry.

In a real-world registry, the list of required variables changes as new research questions and reporting standards emerge. A prompted model allows you to adapt to these changes by simply extending your instructions, whereas a fine-tuned system would require fresh labeled data and retraining every time your schema evolves. For sustained deployment, this ability to pivot is more important than achieving peak accuracy on a fixed set of variables.

How does the system compare to human verification?

To validate the system, researchers used a development set of 200 notes to refine prompts and schemas, ensuring they covered a broad range of cancer types, stages, and treatments. They then evaluated the model on a separate, randomly sampled test set of 200 notes, ensuring no overlap with the development data. Ground-truth labels were generated via two independent clinical reviews followed by a senior adjudication process, during which validators assessed model outputs against the original notes.

The reference standard achieved inter-annotator agreement ranging from 92.8% F1 for list-valued variables to 96.8% F1 for scalar diagnostic fields. The researchers computed 95% confidence intervals using non-parametric bootstrap resampling with 5,000 replicates, which preserved the correlation structure among fields within each note. They also conducted exploratory transferability testing on 50 notes from Taiwan.

The model performs at a level comparable to human verification, with independent reviewers achieving an F1 score of 93% to 97% on the same verification tasks. The system acts as an augmentation of your existing manual abstraction workflow rather than an alternative architecture, and it meets the standard for core fields. While this study focuses on this workflow, a direct comparison against a separately trained classifier remains a goal for future work.

What are the regulatory considerations for deployment?

When deploying these models, you must address privacy concerns related to global standards like HIPAA, GDPR, and Singapore’s PDPA. Curie Oncology in Singapore provided the de-identified clinical notes used in this study, with the research team first accessing the data on 19 September 2025.

Curie Oncology determined that this specific study did not constitute Human Biomedical Research under the Singapore Human Biomedical Research Act 2015. Consequently, the project did not require an Institutional Review Board review.