Surgeons followed 60.6% of wrong AI advice: make oversight measurable

A peer-reviewed study had surgeons follow 60.6% of deliberately wrong AI suggestions, so a staff member at the end of the chain is not a reliable correction layer. Treat automation bias as a measurable system risk: make AI advice contestable, log overrides and test oversight with deliberately wrong advice.

In November 2026, Surgical Endoscopy published the study The surgeon in the loop. In a controlled experiment with nineteen surgeons, 60.6% of the deliberately incorrect AI suggestions were followed; correct suggestions were followed in 76.9% of cases. The authors conclude that human oversight should be empirically measured and designed rather than assumed.

That is the core point for anyone building AI advice into a decision with consequences. The presence of an expert who signs off does not mean that errors are corrected. In our assessment this figure is not a universal error margin for professionals, but it is hard evidence that the human step in the chain does not work by itself.

What exactly did the surgery study find and why is that relevant beyond the operating theatre?

The researchers presented experienced surgeons, in a controlled experiment, with AI suggestions, some of which were deliberately wrong. More than three in five wrong suggestions were followed anyway, while the study also reported changes in participants' confidence. In our analysis this points to a broader mechanism that also operates outside the operating theatre: people tend to rely on an automated recommendation. The study itself, however, demonstrates this only for this controlled surgical context.

This phenomenon is called automation bias. In our analysis the same risk may be relevant for other professionals assessing AI recommendations under pressure, such as lawyers, doctors and analysts; the supplied study does not, however, demonstrate this beyond the surgical context studied. For legal or medical decision-making this means you must check AI output yourself at source level.

What is the difference between formal and meaningful human oversight of AI?

Formal oversight is a signature, a review button or an assigned staff member. Meaningful oversight requires that the reviewer has time, expertise, source information, authority and a genuine opportunity to reject the advice. In our assessment the human step without those conditions is mostly decoration.

A systematic review of 28 studies, published via the ACM, describes how limited interpretability, automation bias and organisational incentives that discourage criticism can leave human-in-the-loop controls procedural rather than substantive. In our analysis, this means that defaults or mandatory workflow steps can narrow the reviewer’s practical room to deviate even when formal responsibility remains in place. Compare this with the finding that a director remains liable despite an AI summary: responsibility does not shift along with the automation.

Which concrete measures make oversight of AI advice genuinely verifiable?

The European Data Protection Supervisor (EDPS) published in May 2026, primarily for EU institutions, bodies, offices and agencies, a checklist for human oversight of automated decision-making. It is a self-assessment instrument, not a formal compliance certification, but in our assessment it also offers other organisations a practical reference framework because it translates oversight into concrete organisational checkpoints. Our reading of those building blocks, translated into practical steps, is:

• Give reviewers enough information to understand and question the system's output, including relevant reasons or explanations, rather than treating the output as automatically determinative.
• Require independent four-eyes checking for high-risk decisions, as the checklist describes, and record that requirement where your risk assessment justifies it.
• Record deviations, overrides and escalations, and measure indicators such as error detection, override frequency and response time to assess whether oversight works in practice.
• Give designated people formal authority to override or suspend the process, including stop-the-line authority for serious concerns.
• Periodically test reviewers with scenario-based anomalies, including deliberately wrong advice, to assess whether they detect and challenge errors.
• Keep the workload realistic so that speed or management pressure does not discourage criticism.

The open question the sources do not answer is how to introduce these controls without creating new bias or workload yourself. In our assessment the answer lies in restraint: apply the heaviest controls only to decisions with real consequences and keep the measurement light enough not to become a tick-box ritual. An override log that no one reads is, in our analysis, just another formal layer.

What does this news mean for directors, lawyers and CISOs?

Our analysis: because experienced professionals in this experiment followed well over half of the wrong AI advice, the assumption that a human final check reliably catches errors is not tenable; therefore we advise the responsible director to establish now at which decisions AI advice has become a default choice and to reverse that presentation to explicitly contestable advice. Because the ACM review describes that governance configures the actual decision authority, in our assessment much of the controllable risk sits in how the workflow is designed; therefore we advise the CISO or process owner to set up override, escalation and response time as measurable KPIs and to audit the decision logs periodically. Because Article 14 of the EU AI Act requires human overseers to be enabled to remain aware of the tendency to rely automatically or excessively on AI output, the provision treats overreliance as a relevant human-oversight risk. The analysis in the European Journal of Risk Regulation argues that compliance should not be reduced to a warning or training alone; therefore we advise the lawyer to record in contracts with suppliers which context-sensitive measures the provider and the customer each take. Because the EDPS checklist describes scenario-based testing and simulated anomalies, we advise every organisation that supports consequential decisions to schedule such tests and record the outcomes before a regulator or court asks for them. See also the topic hub on AI governance and human oversight and how the EDPB tests anonymity on isolation, linking and inference in AI-driven data processing.

In short, in our analysis this news yields a concrete agenda: map where AI advice has become a default, make that advice contestable, measure overrides, distribute responsibility contractually and test oversight with deliberately wrong advice. Whoever fails to do so is, in our assessment, left with a signature that corrects little.

Sources: The article relies on the study in Surgical Endoscopy, an ACM review, an analysis in the European Journal of Risk Regulation and the EDPS checklist for human oversight.