AI systems have to be pressure-tested in real-world clinical settings to ensure correct reasoning, accuracy and reliability.
getty
While it’s commonly accepted that AI systems can occasionally make factual errors during regular use, a new and concerning problem is capturing the attention of clinical and technology leaders: AI can appear confident and competent even when it’s entirely mistaken and lacks true expertise in a particular field. This phenomenon, known as “cognitive spoofing,” describes AI’s tendency to convincingly demonstrate knowledge without the necessary clinical context or judgment for decision-making, particularly in healthcare settings. AI’s ability to generate polished responses that seem evidence-based can lead clinical professionals and doctors to place undue trust in these systems, resulting in workflow disruptions.
A study conducted by Microsoft revealed that recent advanced models achieved exceptionally high scores on medical benchmarks and exams. However, stress tests indicated that these systems relied on clever answering tactics rather than genuine knowledge or robust reasoning. “Leading systems often guess correctly even when key inputs like images are removed, flip answers under trivial prompt changes, and fabricate convincing yet flawed reasoning. These aren’t glitches; they expose how today’s benchmarks reward test-taking tricks over medical understanding.” Despite their high scores on objective tests, these systems may not be ready for real-world clinical environments. The study highlights that clinical benchmarks for AI often focus on correctness rather than the reasoning process behind the correct answer. In real-time clinical settings, this approach could lead to major problems: “Medical readiness is a multidimensional construct. In real-world settings, models must tolerate missing or noisy data, justify their decisions in a manner clinicians can understand, and reason across time, modality, and context. Performance must be not only accurate but also reliable, interpretable, and safe under uncertainty.”
This raises an important question: how can users and physicians detect when they are being misled by AI, or when the AI has provided an incorrect answer that seems convincing?
A crucial solution is the concept of explainability, which involves making AI systems clarify how they reach their conclusions. This understanding is vital as it allows users to see the reasoning the model employed to arrive at its final answer. As explained by IBM, “It is crucial for an organization to have a full understanding of the AI decision-making processes with model monitoring and accountability of AI and not to trust them blindly…Explainable AI also helps promote end user trust, model auditability and productive use of AI. It also mitigates compliance, legal, security and reputational risks of production AI.”
So, what steps should users take, particularly in clinical environments?
Continuously challenge the systems. When an output is provided, inquire about the AI’s reasoning process, ask for its sources, and request the system to clearly demonstrate its thought process. Verify all sources by clicking on the links to ensure the outputs correspond with the source material. While this requires extra time, it is undoubtedly worthwhile to minimize errors, especially in clinical settings.
