Pilots conducting pre-flight procedures
getty
An ambient AI scribe, a program designed to transcribe conversations between a patient and clinician into clinical notes, incorrectly recorded that a medication for post-traumatic stress disorder had been prescribed, despite no mention of the medication or diagnosis during the discussion. This error was detected before the note was sent to the patient.
Dr. Jennifer Shannon, a child and adolescent psychiatrist and the Co-Founder and Chief Medical Officer of Glacis, personally reviewed the transcript and found no basis for the erroneous note.
“Imagine handling 20 patients daily, each with lengthy generated notes,” she stated. “Relying solely on human-in-the-loop protection isn’t adequate. Humans can be fallible, busy, and often tired.”
The challenge of documentation is significant. Few would argue that clinicians should spend their evenings filling out forms and typing notes. However, the introduction of AI changes roles in subtle ways. Clinicians, once responsible for crafting notes from patient visits, now often review AI-generated summaries. While their responsibility remains, the nature of their work has evolved.
I don’t envision meticulous reviewers scrutinizing each sentence. Instead, I picture a hectic Tuesday afternoon, with someone running behind schedule, an inbox filled with thirty messages, a child needing to be picked up, and a patient waiting in room three.
Life is unpredictable.
Expertise Is Part Of The Safety System
Dr. Richard Rieck, a neuroradiologist and pilot, operates in multiple high-risk settings. Pilots train extensively in simulators where they face compounded failures—system malfunctions, engine troubles, and changing weather conditions. The aim is not realism but to enhance skills and instincts before real-life emergencies arise.
Dr. Rieck remarked, “With enough simulator training, flying the actual plane is almost a relief because everything works. You prepare for failures so that unexpected events don’t require a new response.”
In aviation, these rigorous protocols exist because modern aircraft and their systems are highly reliable. No one boards a plane expecting the pilot to vanish. Maintaining expertise isn’t a sign of inadequate automation; it’s a vital part of the safety framework.
Automation Changes The Work More Than The Accountability
Dr. Richard Rieck views AI in radiology similarly. Pilots understand both their failure modes and automation’s limitations, and radiologists need to recognize potential errors in both human and AI systems.
Neither is flawless; safety arises from knowing where failures are likely.
The “human-in-the-loop” concept, a favored AI safety term, suggests that AI-generated work is reviewed by a human before reaching a patient or customer. However, this often reduces the human role to a backup for when the machine errs.
This doesn’t reflect reality. People aren’t just there to fix machine errors. Their experience and judgment are integral to the system. As automation progresses, these human skills become even more crucial.
The Most Overlooked Safety Layer? Patients.
High-risk fields rarely rely on a single safety measure, and healthcare is no different. Beyond clinicians, patients have developed their own expertise in managing their care.
Before ambient documentation, patients were already cross-checking medication lists, identifying referral issues, correcting demographic errors, and navigating conflicting advice from specialists. They weren’t just engaged in their care, they were identifying system failures they could only partially observe.
Patient advocate Hugo Campos, along with Liz Salmi, has addressed critical AI health literacy, highlighting the need for a shift from institutional AI to patient-directed AI.
He argues, “We must stop the system from being the gatekeeper and empower individuals to assist themselves.”
His insights imply that AI should bolster, not weaken, patient expertise. As clinicians juggle automation, crowded inboxes, and tight schedules, patients increasingly contribute resilience to the system by catching errors others might overlook.
Professionals Deserve A Playbook
Discussions about AI risk often focus on machine failures, but another question arises—what becomes of human skills after years of automation success?
Pilots prepare for failures before encountering them. Sports teams practice unlikely scenarios. Trauma teams rehearse codes. The goal isn’t to eliminate surprises but to prevent professionals from having to devise solutions under pressure.
Many systems still amaze me. They’re impressive, which is why I worry about the consequences for human expertise when systems function correctly most of the time.
AI systems are becoming very proficient, prompting thoughts on skill decay more than catastrophic failures.
Discovering skill decay during an emergency is undesirable. Professionals shouldn’t have to create solutions in high-stakes situations.
Now is the moment to be intentional about maintaining essential skills. Once expertise is lost, its absence will be keenly felt when needed most.
