AI video analysis vs human raters in EMS training reliability
How does AI video analysis compare to human EMT raters?
The preprint shows the AI video‑analysis platform matched or beat the human evaluators on the cervical‑collar skill but fell short on bag‑valve‑mask ventilation. Des Moines Area Community College ran the test with five certified EMS raters and three EMT tasks. The AI’s correlation with the human panel was high for the simple collar task (r = 0.857) and low or negative for the more complex ventilation skill.
What are the limitations of automated EMT skill assessment?
Researchers used a fully crossed design: five EMS evaluators and the AI each scored the same video recordings. The videos captured 15 collar applications, 14 bag‑valve‑mask ventilations, and 15 medical assessments. Scorers used dichotomous checklists that flagged critical failures. Agreement was measured with Fleiss kappa, Krippendorff’s alpha, Gwet’s AC1, and intraclass correlation coefficients (ICC 2,1 and ICC 2,k).
Why does EMS rater reliability matter in medical simulation technology?
The AI’s alignment tracked rubric observability, not task complexity: r = 0.857 for collar (exceeding every human), –0.173 for ventilation, and 0.664 for the medical assessment.
What the study does not prove
The paper is a preprint and has not undergone peer review. It examined only video‑recorded simulations, not live field performance. Correlation does not equal causation, and the AI’s poor score on ventilation suggests it cannot yet replace human judgment for all tasks.
What are the potential implications for EMT programs?
If the AI can reliably score straightforward tasks like collar placement, programs could reduce the time each student spends under direct human observation. That would free instructors to focus on more nuanced skills where the AI struggled. However, the mixed performance means human oversight remains essential, especially for complex psychomotor actions.
What are the next steps for AI video analysis technology in EMS training?
The authors recommend larger trials with diverse learner cohorts and real‑world emergency scenarios. Refining the algorithm to better capture ventilation dynamics is a priority. Integration into EMT curricula will likely proceed only after peer‑reviewed validation and regulatory clearance.
- Agreement between an artificial intelligence video-analysis platform and human evaluators scoring emergency medical technician psychomotor skills: a fully crossed reliability study — Des Moines Area Community College, Oct 9, 2026
- Agreement between an artificial intelligence video-analysis platform and human evaluators scoring emergency medical technician psychomotor skills: a fully cross — medRxiv, Oct 9, 2026
Add NewsPulse Time as a preferred source on Google
Frequently asked questions
Yes. Recent research shows AI video analysis achieves reliability scores comparable to trained human raters when evaluating standardized EMT skill performances.
Current AI systems may struggle with nuanced clinical judgment, atypical patient scenarios, and video quality issues, requiring human oversight for edge cases.
Consistent rater reliability ensures that trainees receive accurate feedback, supports valid competency assessments, and maintains accreditation standards for EMS programs.
AI is expected to augment, not fully replace, human raters. It can handle routine assessments efficiently, allowing educators to focus on complex decision‑making and mentorship.
