AI Over-Reliance: Designing Calibrated Trust (2026)

AI over-reliance is no longer an aviation-psychology term buried in a 1997 journal. It is sitting in every office where someone pastes a model’s answer into a deliverable without reading it twice, and in every meeting where a chart nobody questioned turns out to be wrong.

The twin failure is growing just as fast, and it gets a fraction of the attention. For every person who believes the machine every time, there is another who quietly stopped believing it after one bad answer, and both of them are costing their organizations real money.

This post is the trust-calibration chapter of my hub on gamification and AI, where I argued that AI keeps removing the struggle from human work while leaving the motivation question standing. Here the question is sharper than anywhere else in the series: once the machine produces the answer, what keeps a human motivated to check it?

The research points at one Core Drive doing most of the work, and it is the one most AI interfaces accidentally switch off. By the end of this post you will know why presenting every answer with the same confident tone suppresses the exact instinct that keeps people checking, and what to build instead.

Speed Run Notes

• Trust researchers have defined the goal for two decades: calibrated trust, where your reliance on a machine tracks what it can actually do. Over-trusting and dismissing it are the same dial turned to opposite ends.
• Human judgment of AI help is measurably wrong by default: in a 2025 METR trial, experienced developers using AI tools took 19% longer while believing they had been 20% faster.
• People read an AI’s tone instead of its accuracy. Research in Nature Machine Intelligence found longer explanations raised user confidence without raising correctness.
• A CHI 2025 survey of 319 knowledge workers found higher confidence in AI predicted less critical thinking, while higher self-confidence predicted more.
• Forcing people to slow down reduces over-reliance, and users hate it. The Harvard team that proved the effect also found their best designs got the worst ratings.
• Core Drive 7 curiosity is the sustainable lever: interfaces that show honest uncertainty give the brain an open loop worth closing, so checking happens by desire instead of decree.

Author Credibility: Yu-kai Chou

Yu-kai Chou created the Octalysis Framework after studying gamification since 2003 — years before the term entered mainstream vocabulary. As a Human-Systems Architect & Behavioral Designer, his framework has been applied by LEGO, Microsoft, Porsche, Coca-Cola, Salesforce, and MrBeast, impacting over 1.5 Billion Users.

Chou has taught the Octalysis methodology at Harvard, Stanford, Yale, Tesla, Google, BCG, and IDEO.

His work has been cited by Harvard, Stanford, MIT, Forbes, Wall Street Journal, Wired, US Department of Energy, NIST, NSF, NCBI, US Department of Education, ClinicalTrials.gov, and Google Scholar — with 3,700+ more academic publications. Explore his books here.

Two Ways to Get the Machine Wrong

The research community saw this coming long before chatbots did. In 1997, Raja Parasuraman and Victor Riley published a paper in Human Factors with a title that still maps the whole territory: “Humans and Automation: Use, Misuse, Disuse, Abuse.”

Misuse, in their taxonomy, is relying on automation when you should not. Disuse is rejecting it when it would have helped.

Both failures were documented in cockpits and control rooms decades before anyone typed a prompt. What has changed is the population exposed to them: a problem that used to belong to pilots and radiologists now belongs to anyone with a browser tab open.

In 2004, John Lee and Katrina See gave the field its north star in a Human Factors review called “Trust in Automation: Designing for Appropriate Reliance.” Their argument: the goal of design is calibrated trust, where how much you rely on a system tracks what that system can actually do.

Read that definition again, because it quietly reframes the whole AI adoption conversation. The goal was never to maximize how much people trust the system. A person who trusts a 90%-accurate system 100% of the time is exactly as miscalibrated as a person who trusts it 0% of the time, and the second person at least avoids shipping the errors.

Over-reliance and under-reliance are the same dial turned to opposite ends. Any team measuring “AI adoption” without measuring calibration is watching the dial spin and calling it progress.

You have already seen both ends of the dial this month, whether you noticed or not. The colleague who forwards a model-generated summary of a contract without opening the contract is at one end, and the senior engineer who refuses to let an assistant touch the codebase because of something it got wrong in 2024 is at the other.

Each of them would describe their position as prudence. That is what makes trust miscalibration such a slippery design target: it never feels like a failure from the inside.

The Perception Gap Is the Default State

If humans came with accurate internal meters, calibration would solve itself. The cleanest evidence we have says the meters are broken in a specific direction.

In mid-2025, the research group METR ran a randomized controlled trial with 16 experienced open-source developers working on 246 real issues in their own repositories. When allowed to use AI tools, the developers took 19% longer to complete their tasks.

The number that matters for this post is the other one. Before starting, the developers predicted AI would speed them up by 24%, and after finishing, with the slowdown already in their bodies, they still believed it had sped them up by 20%.

METR’s early-2026 follow-up suggests the tools themselves have improved since then, while calling its newer data a weak signal. The durable finding is the gap itself: the feeling of being helped and the fact of being helped are measured by different instruments, and only one of them is attached to reality.

I covered the full story of that trial in my post on AI coding assistant productivity, because it deserves its own field manual. For trust calibration, the lesson compresses to one line: self-perception is the instrument people actually use, and it reads high.

Why does it read high? A 2025 study in Nature Machine Intelligence by Mark Steyvers and colleagues, “What Large Language Models Know and What People Think They Know,” measured what the team calls a calibration gap: the distance between how confident a model actually is in an answer and how confident its user believes it is.

With default responses, people overestimated the model’s accuracy. When the researchers made explanations longer, user confidence rose further while accuracy did not move at all.

Sit with that result for a moment, because it explains most of this post. Length and fluency, the two things generative models produce most cheaply, are precisely the signals humans read as competence.

The same study found the fix: when explanations were adjusted to reflect the model’s internal uncertainty, the gap narrowed and users’ sense of when to trust the model improved. Honesty about uncertainty is not a vibe choice; it is the one intervention shown to repair the meter.

The Octalysis Read: Uniform Confidence Switches Off Checking

Now run this through the Octalysis Framework, because the calibration literature describes what happens and the 8 Core Drives explain why it happens.

Checking a machine’s output is a behavior, and behaviors need fuel. Three Core Drives are candidates for fueling it, and modern AI interfaces are quietly draining all three.

Core Drive 7 (CD7): Unpredictability & Curiosity is the drive to find out what happens next, and it only fires when an outcome feels uncertain. A question you think is settled generates no curiosity at all.

This is where interface tone becomes a motivational event. When a system delivers its 60%-confidence answers and its 99%-confidence answers in the same fluent, authoritative voice, it tells the user’s brain that nothing here is uncertain, and a brain with no uncertainty signal has nothing left to wonder about.

The checking instinct does not get argued away. It gets starved, one confident-sounding answer at a time.

Core Drive 8 (CD8): Loss & Avoidance is how most organizations currently motivate verification: review checklists, compliance sign-offs, “you own what you ship.” Fear of loss does produce checking, and as with all Black Hat motivation it produces resentful, minimal, exhausting checking that decays into rubber-stamping the moment nobody is watching.

The same Core Drive 8 machinery also powers the opposite failure. One public, embarrassing AI error can flip a whole team from over-trust into refusal, because loss avoidance does not calibrate; it lurches.

Core Drive 3 (CD3): Empowerment of Creativity & Feedback is the drive that makes work feel like your move in a game: you make a judgment, the world responds, you adjust. Verification as currently designed is CD3-starved work, a long row of approve buttons where your judgment rarely changes anything visible.

Worse, the feedback loop in most AI tools closes at the wrong moment. The little hit of progress fires when you accept the suggestion, while the real outcome, whether the thing was correct, arrives hours or weeks later, unattached to the moment of judgment.

Put the three together and the diagnosis writes itself. We built interfaces whose tone suppresses CD7, whose accountability systems run on CD8, and whose feedback loops bypass CD3, and then we wrote think pieces wondering why humans stopped checking the machine.

What Believing Costs: The Evidence

The cost of uncalibrated trust is now measurable in the general workforce, and the newest evidence points at confidence as the control variable.

At CHI 2025, the leading academic conference on human-computer interaction, researchers from Carnegie Mellon and Microsoft Research published a survey of 319 knowledge workers covering 936 real examples of AI use at work. The finding that made headlines: the more confident a worker was in the AI, the less critical thinking they reported applying to its output.

The finding that should have made headlines sits one line lower. Workers with higher confidence in their own skills reported more critical evaluation, even while using the same tools.

Trust in the machine and trust in yourself pull in opposite directions, and interfaces currently inflate the first while doing nothing for the second. That is a design choice, even when nobody remembers making it.

Medicine learned this earlier and more expensively. A 2012 systematic review in the Journal of the American Medical Informatics Association by Kate Goddard, Abdul Roudsari, and Jeremy Wyatt synthesized 74 studies of automation bias, the tendency to over-rely on automated advice, much of it from clinical decision support.

Their list of what drives the bias reads like a description of the modern AI workplace: trust in the system, workload, task complexity, and time pressure. Their list of what reduces it is the striking part, because it was published a decade before ChatGPT: attach updated confidence levels to the system’s output, train users, keep them accountable for the final decision, and present information rather than finished recommendations.

That answer sheet has been public since 2012. The industry mostly built the opposite, because confident-sounding assistants demo better than humble ones.

Under-Reliance Is the Quieter Failure

Everything above could be read as “trust less,” and that reading walks straight into the second ditch.

In 2015, Berkeley Dietvorst, Joseph Simmons, and Cade Massey published “Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err.” In their forecasting experiments, people who watched a statistical model make mistakes became far less willing to use it afterward, even though the model still outperformed the human forecasters.

We forgive our own errors as bad luck and prosecute the machine’s errors as character flaws. An employee who makes one bad call gets coaching; a model that makes one bad call gets turned off.

The cost of this failure is quieter because nothing visibly breaks. The team that stopped using the diagnostic aid just runs slower and misses more, with no incident report attached, and in a world where gamification and AI increasingly decide who gets the benefit of machine help, the under-reliers silently fall behind the calibrated.

This is why I keep insisting the thesis cuts both ways: over-reliance and under-reliance are both calibration failures. A trust dial has no safe direction to be wrong in.

Here is the whole argument as one table, with the motivational engine named for each state.

The middle column is the part most teams skip. Every trust state has a motivational engine underneath it, and you do not move people between states by telling them to be more careful; you move them by changing which Core Drive the interface feeds.

Curiosity Is the Calibration Engine

So how do you actually get a human to keep checking a machine that is right most of the time? The honest answer from the research is that force works and people hate it, which is exactly the kind of answer motivation design exists for.

In 2021, Harvard’s Zana Buçinca and Krzysztof Gajos, with Phoebe Malaya, tested what they called cognitive forcing functions: interventions that make you engage before or while accepting an AI recommendation, like committing to your own answer first or having the AI’s suggestion arrive on a delay. Across 199 participants, these reduced over-reliance where ordinary explainable-AI approaches had failed to.

Then comes the line every product manager should tattoo somewhere visible: the designs that reduced over-reliance the most received the worst ratings from the people using them. Pure friction calibrates the trust and loses the user.

This is the gap Octalysis was built to close. The forcing-function researchers discovered that System 2 thinking needs a trigger; game designers have known for decades that the best trigger is not an obligation but an open loop.

Core Drive 7 sits on the Black Hat side of Octalysis, the same machinery that keeps a player opening loot boxes, and the craft is in where you point it. Pointed at model outputs, that pull will keep an analyst spot-checking all day, provided the interface preserves the one ingredient curiosity cannot live without: visible uncertainty.

A confidence score that actually varies gives the user something to wonder about. A model that occasionally says “I am unsure about the middle section” creates a genuine open loop, and two models that disagree in front of you practically beg to be adjudicated.

Prediction is the strongest version of the pattern. Ask the user to commit a guess before the reveal, the way good trivia and good classrooms do, and the AI’s answer transforms from a verdict to be accepted into feedback on a bet you placed, which moves the interaction from passive CD8 compliance into a loop where CD7 anticipation pulls you in and White Hat CD3 judgment does the work.

Designed this way, checking stops being a tax on the work and becomes a round of the game. The trade-off the Harvard study measured softens, because the friction now pays the user in curiosity resolved and judgment exercised.

The Calibrated-Trust Playbook: Five Moves

If I were designing an AI product, an oversight workflow, or a team’s operating rules tomorrow, these are the five moves I would make, in order.

1. Show honest uncertainty, even when it demos badly. The Steyvers team showed that explanations tailored to the model’s real confidence narrow the calibration gap, and the clinical literature listed confidence display as a mitigator back in 2012. A system that sounds equally sure of everything is lying about something, and your users’ Core Drive 7 can only engage with the doubt it is allowed to see.
2. Ask for the human’s prediction before the reveal. Commit-then-compare is the cognitive forcing function that feels least like homework, because it converts verification into a bet. A radiologist who logs her read before seeing the model’s, or an analyst who writes his estimate before the dashboard loads, is running a self-calibrating loop that also happens to be fun.
3. Make catching the machine’s errors count as an accomplishment. Whoever finds the model’s error did the most valuable work of the day, so treat it that way: track catches, celebrate them, and let a record of good catches build visible mastery over the tool. I wrote in my post on human-in-the-loop jobs that oversight work collapses when it has no progression; a catch ledger is the smallest unit of progression that works.
4. Vary the ritual so checking never becomes a rubber stamp. Fixed checklists decay because sameness kills Core Drive 7. Rotate which outputs get deep review, run occasional seeded-error drills where a known mistake is planted and finding it is the win, and let the audit schedule itself be a little unpredictable. Casinos learned variability keeps attention alive; your review process can use the same physics for honest ends.
5. Design the recovery from the first big failure in advance. Algorithm aversion means one visible error can flip a team from misuse to disuse overnight. Publish the tool’s error rate next to the human baseline, hold blameless post-mortems that treat a model failure as data rather than betrayal, and re-onboard trust deliberately. Calibration after an incident is a designed experience, and leaving it to emotion guarantees the lurch.

None of the five requires new model capabilities. Every one of them is a motivation-design decision sitting in the interface and the operating rules, available to any team willing to stop optimizing for how trustworthy the machine sounds.

What This Means for Leaders, Designers, and Individuals

For leaders, the metric to add is calibration, and the metric to retire is raw adoption. A team that accepts 100% of AI suggestions is not your success story; it is your largest unreviewed risk, and the CHI 2025 data says its members are thinking less with every confident answer they swallow. Track how often people check, how often checking catches something, and whether trust in the tool moves when the tool’s accuracy moves. I made the case in my post on agentic AI governance that oversight is becoming a real job; calibration is that job’s core skill.

For designers, the uncomfortable truth is that your interface’s tone is a motivational instrument whether you tuned it or not. Uniform fluent certainty suppresses the user’s curiosity, inflates their confidence in the system, and manufactures the over-reliance your legal team will later call user error. The spec is twenty years old, from Lee and See through Goddard to Steyvers: expose uncertainty, present information over verdicts, and let the user’s judgment stay in the loop as a move they make rather than a box they tick.

For individuals, the practice is smaller and harder: notice when you have stopped reading the output. The moment an AI’s answers start going straight into your work untouched is the moment your own meter has drifted, and the METR developers are proof that the drift feels like speed. Keep one private ritual of prediction, on the tasks where being wrong costs you most, and treat every error you catch as evidence that your judgment is still load-bearing, because in my post on AI and employee motivation I argued the people who thrive with these tools are the ones who stayed deliberate about what they kept doing themselves.

FAQs

What is AI over-reliance?

AI over-reliance is accepting an AI system’s output without appropriate verification, so that your reliance exceeds what the system’s actual accuracy justifies. It is the modern form of what human-factors research calls automation misuse, and it shows up as uncaught errors shipped downstream. Its mirror image, under-reliance, means rejecting machine help that would have improved your results, and both are failures of trust calibration.

What is automation bias, and how does it relate to AI?

Automation bias is the documented tendency to over-weight automated advice, studied for decades in aviation and clinical decision support. A 2012 systematic review in JAMIA covering 74 studies found it is driven by trust in the system, workload, task complexity, and time pressure. Generative AI inherits the bias and amplifies it, because fluent, confident language is exactly the signal humans misread as accuracy.

What is calibrated trust in AI?

Calibrated trust means your level of reliance on a system tracks that system’s real capabilities: you lean on it where it is strong and check it where it is weak. The concept comes from Lee and See’s 2004 work on appropriate reliance in automation. In Octalysis terms, sustainable calibration runs on Core Drive 7 (Unpredictability & Curiosity), because a user who can see honest uncertainty stays curious enough to keep verifying.

How do you prevent over-reliance on AI?

The evidence supports five design moves: display the model’s real uncertainty instead of uniform confidence, have users commit their own prediction before seeing the AI’s answer, reward error catches as accomplishments, vary the review ritual so checking stays engaging, and plan trust recovery for the first visible failure. Harvard research on cognitive forcing functions shows friction alone reduces over-reliance but gets rejected by users, which is why the motivational layer decides whether calibration sticks.

Is distrusting AI safer than over-trusting it?

Distrust carries a measured cost of its own, and the research has priced it. Studies of algorithm aversion show people abandon models after seeing one error, even when the model still outperforms human judgment, and teams that quietly stop using capable tools fall behind without any visible incident. Safety lives in calibration: matching your reliance to the system’s demonstrated accuracy on each kind of task, and updating as the system changes.

Your Next Step

Trust calibration is the clearest case yet for the argument running through this whole series: the scarce resource in the AI era is well-designed human motivation, and the interfaces winning demos today are quietly spending it down.

If you want the full motivational architecture behind this post, the 8 Core Drives, White Hat versus Black Hat design, and the frameworks for building systems where humans stay sharp by choice, explore my books. The teams that thrive with AI will be the ones whose humans never stopped wanting to check.

Related Reading

• AI ROI Pressure: When AI Theater Replaces the Work
• AI Tool Sprawl: The Motivation Cost of Tool Fatigue
• AI Patient Adherence: The Motivation Problem Reminders Miss
• Self-Determination Theory: The Complete Guide

WOULD YOU LIKE YU-KAI CHOU TO WORK WITH YOUR ORGANIZATION?