AI Adoption Telemetry: What Usage Dashboards Miss (2026)

Your AI dashboard is almost certainly lying to you, and it is not lying about something small. AI adoption telemetry exists because the two numbers most companies watch, active users and output volume, stopped meaning what executives think they mean the moment generative tools arrived.

Here is the measured version of that problem. A 2025 randomized trial found experienced developers finishing tasks 19% slower with AI while believing they were 20% faster, and behavioral data from over 163,000 employees shows that while 80% now touch AI tools, the majority of those users spend less than 1% of their working hours inside them.

Adoption looks universal on the dashboard, yet depth, the thing adoption was supposed to buy, stays rare, and the instruments most organizations use cannot tell the two apart.

This post closes the arc of my hub on gamification and AI, where I argued that AI keeps hollowing out motivation faster than organizations rebuild it. Every arena in that hub ended with the same unanswered question: how would you even know? This is the field manual for knowing.

Speed Run Notes

• Output volume no longer indexes human effort. A METR trial found experienced developers 19% slower with AI while believing they were 20% faster, so any dashboard counting output is now measuring the wrong layer.
• Self-report cannot rescue the dashboard: a Nature Human Behaviour meta-analysis of 106 comparisons found self-reported digital behavior matched the logs in only about 5% of studies.
• Surveys say 75% of knowledge workers use AI. Telemetry from 163,638 employees says 80% touch the tools while 57% of users spend under 1% of their work time in them, so depth is the number that still needs measuring.
• AI adoption telemetry reads motivation from production signals: session depth, iteration loops, task success, and retention curves, interpreted through an explicit model of behavior change.
• In Octalysis terms, telemetry is a Core Drive instrument. Depth signals track real accomplishment and live feedback loops, while usage mandates and prompt-count leaderboards drift into fear-driven compliance.
• Goodhart’s law is the standing hazard: the moment a telemetry signal becomes a target, people optimize the signal. Pair every adoption number with an outcome number and keep measurement at the population level.

Author Credibility: Yu-kai Chou

Yu-kai Chou created the Octalysis Framework after studying gamification since 2003 — years before the term entered mainstream vocabulary. As a Human-Systems Architect & Behavioral Designer, his framework has been applied by LEGO, Microsoft, Porsche, Coca-Cola, Salesforce, and MrBeast, impacting over 1.5 Billion Users.

Chou has taught the Octalysis methodology at Harvard, Stanford, Yale, Tesla, Google, BCG, and IDEO.

His work has been cited by Harvard, Stanford, MIT, Forbes, Wall Street Journal, Wired, US Department of Energy, NIST, NSF, NCBI, US Department of Education, ClinicalTrials.gov, and Google Scholar — with 3,700+ more academic publications. Explore his books here.

The Day Output Stopped Indexing Effort

For a hundred years of management practice, output volume was a usable proxy for human effort. If a team shipped more, wrote more, closed more, you could assume the people in it were more engaged, more skilled, or at least more present.

Generative AI severed that link quietly and completely. A report that took four hours of thinking last year can now be produced in nine minutes by someone who did almost no thinking at all, and the two artifacts look identical in every system of record your company owns.

The cleanest evidence of the severed link comes from software, because software is the one arena where researchers ran the controlled experiment. In a study published in July 2025, the research group METR randomized 246 real tasks across 16 experienced open-source developers, allowing AI tools on some tasks and banning them on others.

Developers using AI took 19% longer to finish. Before the study they predicted AI would make them 24% faster, and after finishing the tasks they still believed it had made them about 20% faster. METR itself notes the result reflects early-2025 tooling, and the tools keep improving, but the perception gap is the finding that matters here, and no tooling upgrade closes a gap between feeling and measurement.

That gap is the whole measurement crisis in one number. The subjective experience of working with AI diverged from the measured outcome by roughly 40 percentage points, in the population best equipped to judge, on their own projects.

I walked through the motivational mechanics of that illusion in my post on AI coding assistant productivity. The short version is that the reward signal fires when the tool produces something, while the cost arrives later as review, rework, and debugging that never feels like part of the AI’s bill.

GitHub’s own research team documented where that misfiring signal lives. In a study of 2,047 Copilot users whose survey answers were matched to their usage telemetry, the single best predictor of whether a developer felt more productive was the rate at which they accepted suggestions.

An acceptance is a click that happens before the code is tested, reviewed, or shipped, which means the feeling of productivity is anchored to the moment of least information about whether the work was any good.

Once you see this pattern in software, you see it everywhere the hub described. Students submit AI work that no longer proves learning, which is why assessment needs redesigning. Employees produce AI output that no longer proves understanding, which is part of why the most AI-fluent employees quit first while their dashboards glow green.

When output stops indexing effort, every engagement dashboard built on output starts mismeasuring motivation. The question for the rest of this post is what an honest instrument would read in its place.

Why Surveys Cannot Rescue the Dashboard

The reflex answer is to ask people. Send the quarterly survey, add three AI questions, and report the results as adoption.

The measurement literature is brutal about this reflex. In 2021, Doug Parry and colleagues published a meta-analysis in Nature Human Behaviour comparing self-reported digital media use against device logs across 47 studies and 106 comparisons covering roughly 50,000 people.

Self-reports matched the logs in about 5% of studies. People are not lying on these surveys; they simply cannot retrieve accurate accounts of their own digital behavior, and measures of heavy or problematic use tracked the logs even more poorly.

Now apply that to the AI numbers circulating in every boardroom. Microsoft and LinkedIn’s 2024 Work Trend Index, a survey of 31,000 people across 31 countries, reported that 75% of knowledge workers were already using generative AI and that 78% of those users were bringing their own tools to work.

Those are self-reports, gathered by a method whose published accuracy checks came back clean in about one study in twenty. The same report found 59% of leaders worried they could not quantify the productivity gains, which is the honest sentence in the document: the instrument panel and the engine were never connected.

Behavioral data drawn from the machines themselves tells a sharper story. ActivTrak’s 2026 State of the Workplace report, drawn from monitoring telemetry across 1,111 companies and 163,638 employees, found 80% of employees now use AI tools, up from 53% two years earlier, with 92% average monthly retention.

The same research program then supplies the depth number. In its companion analysis, the largest segment of AI users, 57%, spends less than 1% of total work time in AI tools.

Both readings are true at the same time, and holding them together is the point. Nearly everyone has touched the tools and keeps touching them monthly, and most of those same people barely use them. A survey compresses that whole distribution into a yes, and a usage dashboard compresses it into an active-user count, and both report a transformation that has not happened.

Engagement surveys have the same blind spot from the other direction. Gallup’s State of the Global Workplace 2025 measured global engagement falling from 23% to 21% in the first full year of mass AI adoption, at an estimated cost of $438 billion in lost productivity.

So tool usage climbed while human engagement fell, and most organizations watched both lines without owning a single instrument that could explain the divergence. The hub called this the motivation vacuum. Here it shows up as a measurement vacuum, and the two are the same hole.

There is one more reason self-report fails specifically for AI work. A 2025 study from Microsoft Research and Carnegie Mellon, presented at CHI, the leading human-computer interaction conference, covering 319 knowledge workers and 936 real AI tasks found that the more people trusted AI, the less critical thinking they reported applying, while confidence in their own skills predicted the opposite.

The authors are careful to note their data is self-reported and non-causal, and they recommend tool telemetry as the stronger instrument for future work. When the researchers studying self-report recommend telemetry, the direction of travel is clear.

What AI Adoption Telemetry Actually Means

AI adoption telemetry is the continuous measurement of behavior change from production system signals, read through an explicit model of how adoption is supposed to progress. Instead of asking people whether they use AI, it watches what the systems already log: sessions, turns, task completions, repeat visits, and who the usage is concentrated in.

The clearest formalization I have seen comes from an August 2026 paper by Damon A. Young of PolyWise Partners, titled Adoption Telemetry: Measuring Enterprise AI Adoption from Production Signals. It argues that an adoption measurement is only trustworthy when it has four properties.

The measurement must be computable, so stages derive mechanically from logged behavior rather than from anyone’s impression. It must be model-explicit, with thresholds stated openly instead of hiding inside a vendor’s score. Reading has to stay population-level, diagnosing cohorts rather than policing individuals. And every detected stall must map to a specific class of intervention, so the instrument produces decisions rather than decoration.

The paper’s model walks users through five stages: Notice, Attempt, Navigate, Transform, and Embed. Early stages are cheap, and the dashboards most companies run today only ever measure those. Value lives in the later stages: Transform requires sustained multi-week activity, a majority of deep multi-step sessions, and a task success rate above half, while Embed requires activity in at least 90% of weeks since first use.

The gap between those layers is exactly the gap the ActivTrak data exposed: an organization can sit at 80% Notice-and-Attempt while its Transform population rounds to nothing. Young’s framework calls the signature a shallow plateau, detected when the divergence between breadth and depth crosses a stated threshold.

Two more failure signatures are worth naming because every leader I talk to recognizes them on sight. Champion dependency is when usage concentrates in a handful of enthusiasts, measured with a concentration index, so the dashboard’s healthy average is really three people carrying a department. Usage regression is when late-period activity collapses relative to the launch window, meaning the pilot’s numbers were novelty, and the novelty is gone.

Honesty requires one caveat, and it reinforces the post’s argument rather than weakening it. Young’s reference implementation was validated on synthetic populations, with thresholds the author states are proposed defaults awaiting real-world calibration, and the broader evidence base it cites is grim about how rarely anyone measures outcomes at all: an MIT preliminary report found 95% of generative AI pilots showing no measurable profit impact, and S&P Global found the share of companies abandoning most of their AI initiatives jumping from 17% to 42% in a single year.

In other words, the field is young, the instruments are early, and the organizations that learn to read production signals first will be operating on information their competitors do not have.

The Core Drive Read: What Each Signal Is Evidence Of

A telemetry stream tells you what happened. The Octalysis Framework, which I created to map the eight Core Drives behind why humans do anything, tells you what the stream is evidence of, and that interpretive layer is what most adoption programs are missing.

Start with Core Drive 2 (CD2): Development & Accomplishment, the drive to make progress, build skill, and overcome challenges. A healthy CD2 loop leaves a particular telemetry fingerprint: task success rates that climb over weeks, harder task types attempted over time, and usage that survives long after the launch announcement.

The METR result is what a broken CD2 loop looks like in the data. The developers felt accomplishment, the acceptance click delivered it on schedule, and the measured outcome moved the other way. When the feeling of progress and the fact of progress separate, CD2 keeps paying out on the feeling, which is precisely why self-report fails here.

Then Core Drive 3 (CD3): Empowerment of Creativity & Feedback, the drive activated when people experiment, get meaningful feedback, and adjust. CD3 is the depth drive, and it is the one that separates a Transform-stage user from someone stuck in the middle stages.

Its telemetry fingerprint is iteration rather than volume. A user running twelve-turn sessions, refining prompts, feeding results back in, and routing outputs into their actual workflow is in a live feedback loop with the tool. A user pasting one question and copying one answer is running an errand. Both count identically as daily active users, and only the session structure tells them apart.

The darker drives show up in the telemetry too. When companies tie AI usage to performance evaluation, the resulting activity spike is Core Drive 8 (CD8): Loss & Avoidance, fear wearing a productivity costume, and I documented where that leads in my post on AI ROI pressure and adoption theater, where executives admit that proving AI usage has started to displace the work the usage was supposed to improve.

A CD8-driven adoption curve and a CD2-driven one can look identical for a quarter. The difference appears in the signals mandates cannot fake: session depth stays shallow, iteration never develops, task success stays flat, and usage regresses the moment attention moves elsewhere. This is the White Hat versus Black Hat distinction rendered in log files, and it is why I keep insisting that measurement design and motivational design are the same discipline.

One warning before the instrument list, because it decides whether any of this works. Telemetry read at the population level is diagnosis, but the same stream read at the individual level becomes surveillance, and surveillance is a CD8 engine that corrodes the Core Drive 4 (CD4): Ownership & Possession feeling that makes anyone invest in a tool voluntarily. The moment people believe the logs exist to police them, they will perform for the logs, and your instrument will measure the performance.

The Instrument Stack: Seven Signals Worth Logging

Here is the field manual, in the order I would install it. Seven signals, each readable from systems you already run, each mapped to the motivational question it actually answers.

1. Depth share. Report what fraction of AI users clear a real depth bar, such as multi-step sessions in multiple weeks, alongside the headline adoption percentage. One organization reporting 80% adoption and 4% depth knows more than a competitor reporting 80% alone. This is your shallow-plateau detector, and it answers the CD2 question of whether anyone is actually building skill.

2. Iteration loops per session. Count turns, refinements, and re-submissions rather than prompts. Rising iteration is CD3 in motion: the person is experimenting and reading feedback. A corpus of single-turn sessions means your workforce is using a thinking tool as a vending machine.

3. Task success, sampled honestly. Acceptance clicks and generation counts flatter the rollout without proving anything shipped. Sample completed AI-assisted tasks and score whether the output shipped without rework, the way the Copilot researchers matched perception to telemetry and the way Young’s Transform stage requires success above a stated floor. This is the signal that would have caught the METR illusion a year early.

4. Retention curves by cohort. Novelty decays on a schedule you can plot. Compare each monthly cohort’s late-window usage to its launch window, and treat regression below the floor as a finding about the design, because blaming the people reads the signal backwards. ActivTrak’s 92% monthly retention shows people keep coming back to AI broadly; your job is knowing whether yours do.

5. Concentration index. Measure how unevenly usage distributes across the population. If a handful of champions carry the whole average, you have a champion-dependency signature, and the intervention is spreading Core Drive 5 (CD5): Social Influence & Relatedness scaffolding, the cohorts and peer teaching I detailed in the post on AI upskilling programs people actually finish, rather than buying more licenses.

6. Verification effort. The CHI study’s deepest finding was that work shifts from doing to checking. Log how much review, correction, and validation surrounds AI output, because that number is your exposure to the over-reliance failure mode and the honest denominator for any productivity claim. When verification rounds to zero, the likeliest explanation is that nobody is looking, and that is how skill decay compounds silently.

7. A paired human number. The hub’s instruction to leaders was to track felt ownership of outcomes next to every adoption metric, and telemetry sharpens that instruction instead of retiring it. Production signals tell you what behavior changed, and one well-built question tells you whether the humans still feel like authors of the work. Divergence between those two lines is the single most valuable alarm this whole stack can produce, because it is the human-in-the-loop job design problem announcing itself before the resignation letters do.

Notice what is absent from the list: prompt counts, generation volume, time-saved estimates multiplied across headcount, and every number produced by asking people to remember their own usage. Those are the instruments the last section buried.

A 90-Day Instrumentation Plan

Knowing the seven signals is not the same as running them, so here is the sequence I would give any team starting from a standard usage dashboard. It takes one quarter, and nothing in it requires buying a new platform before day sixty.

Days 1 to 15: baseline without announcing anything. Pull ninety days of history from the logs you already have, and compute depth share, iteration distribution, and the concentration index before anyone changes behavior to look good. The unannounced baseline matters because it is the last uncontaminated reading you will ever get. Publishing the plan first would hand everyone a quarter to perform for the instruments.

Days 16 to 30: state your model out loud. Write down what counts as shallow, developing, and embedded use for your organization, with explicit thresholds, the way Young’s paper insists on. The specific numbers matter less than their visibility, because a stated threshold can be argued with and corrected, while a vendor’s proprietary adoption score can only be believed.

Days 31 to 60: classify cohorts and find the stall. Sort teams into stages and look for the signatures: a shallow plateau where breadth is high and depth never develops, champion dependency where three people carry a department’s average, or usage regression where the launch spike is quietly bleeding out. Each signature points at a different Core Drive deficit, which is the entire reason to run the classification.

Days 61 to 90: map one intervention per stall, and measure it as an experiment. A shallow plateau is usually a Core Drive 2 (CD2) problem, so the response is visible skill ladders and harder sanctioned use cases rather than another mandate. Champion dependency is a CD5 problem, answered with cohorts and peer teaching. Regression after novelty is a CD3 problem, meaning the tool never entered a real feedback loop with anyone’s actual work. Pick the one stall that costs the most, intervene, and read the same signals again at day ninety.

Two quarters of this cadence puts an organization in a position few of its competitors occupy: it knows which stage its people are in, why the stalled ones stalled, and whether its interventions move the lines. The 59% of leaders who told Microsoft they cannot quantify AI’s productivity gains do not need a smarter survey. What they are missing is this loop, and it costs exactly one quarter to install.

Goodhart’s Trap: When the Telemetry Becomes the Target

Everything above comes with a standing hazard that is older than AI. Charles Goodhart observed in 1975 that any statistical regularity tends to collapse once pressure is placed on it for control purposes. The principle now circulates as Goodhart’s law, usually in the sharper form that a measure which becomes a target stops being a good measure.

Adoption telemetry is not exempt from that law. Publish a leaderboard of multi-step sessions and you will get multi-step sessions, performed by people padding turns the way students pad word counts. The signals in the instrument stack are only honest while they are diagnostic, which is why three design rules travel with them.

First, pair every behavioral signal with an outcome it is supposed to predict, so a gamed signal exposes itself as divergence. Depth share pairs with task success, retention pairs with cycle-time on real work, and adoption pairs with the felt-ownership number.

Second, keep the unit of analysis at the cohort. The software measurement community learned this lesson a generation ago: the SPACE framework (Satisfaction, Performance, Activity, Communication, and Efficiency) from Nicole Forsgren and her colleagues at GitHub and Microsoft Research opens by demolishing the myth that one productivity metric can tell you everything, and it flatly warns that activity metrics should never be used alone to reward or punish individuals.

Third, never attach an individual consequence to a signal that was installed as a diagnostic. The moment usage telemetry feeds a performance review, you have converted a CD3 feedback instrument into a CD8 compliance instrument, and every reading after that moment is contaminated by exactly the performative adoption the ROI-theater post dissected.

Follow those three rules and telemetry stays what it should be: the first instrument in twenty years of this work that can tell a motivated workforce from a compliant one without asking either to fill out a form.

FAQs

What is AI adoption telemetry?

AI adoption telemetry is the continuous measurement of how people’s behavior with AI tools changes over time, read directly from production signals such as sessions, iteration depth, task success, and retention, and interpreted through an explicit stage model of adoption. It replaces self-reported usage surveys, which a Nature Human Behaviour meta-analysis found matched logged behavior in only about 5% of studies.

Why are AI usage dashboards misleading?

Standard dashboards count breadth signals like active users and generation volume, which stopped indexing human effort once generative tools arrived. Telemetry across 163,638 employees shows 80% of workers touching AI tools while 57% of users spend under 1% of their work time in them, so a breadth dashboard reports transformation where mostly light experimentation exists.

What should companies measure instead of AI usage counts?

Seven signals carry most of the value: depth share, iteration loops per session, honestly sampled task success, cohort retention curves, usage concentration, verification effort, and a paired felt-ownership question. Each maps to a specific Core Drive, so a stall points at a motivational diagnosis rather than a generic adoption problem.

Can AI adoption telemetry become surveillance?

Yes, and that is its central design risk. Read at the population level, telemetry diagnoses system health; read at the individual level, it becomes monitoring that triggers Core Drive 8 (CD8): Loss & Avoidance, pushes people to perform for the logs, and corrupts the very signals being collected. Keep analysis at cohort granularity and never wire a diagnostic signal into individual performance consequences.

Does measuring motivation through telemetry replace engagement surveys?

Not entirely. Production signals show what behavior changed, while one well-designed question about felt ownership of outcomes reveals whether people still experience the work as theirs. The alarm worth building is the divergence between those two lines, because rising usage paired with falling ownership is the motivation vacuum showing up in the instruments.

The Last Dashboard Question

This post closes the twenty-arena arc that began with the hub’s observation that AI creates motivation vacuums faster than organizations rebuild what was lost. Every arena, from coding to classrooms to civic life, ended at the same cliff: the systems we use to know whether people are motivated were built for a world where output proved effort.

That world is gone, and what replaces it is a choice between two instruments: asking people to describe their own behavior, which the measurement literature caught matching the logs in about one study in twenty, or reading the behavior itself from the systems already logging it, through a model honest enough to state its own thresholds.

The organizations that get this right will notice something unexpected. The act of instrumenting depth, iteration, and ownership forces you to design for depth, iteration, and ownership, because for the first time the dashboard stops rewarding theater.

If you want the full motivational architecture underneath these instruments, the eight Core Drives, the White Hat and Black Hat distinction, and the design process for systems people return to voluntarily, explore my books, including Actionable Gamification, which walks through every drive this measurement stack was built to detect. The dashboard question that matters was never how many people used the AI. It was always whether the humans still wanted to do the work, and now you know how to read the answer.

Related Reading

• AI Coding Assistant Productivity: Why Developers Feel Fast and Finish Slow
• AI ROI Pressure: When Proving You Use AI Replaces Doing the Work
• AI Deskilling: What Happens to Mastery When AI Does the Reps
• AI Over-Reliance: Designing for People Who Believe the Machine
• The AI Upskilling Program People Actually Finish

WOULD YOU LIKE YU-KAI CHOU TO WORK WITH YOUR ORGANIZATION?