Most AI dashboards answer the easiest question: How much did we use?

Seats, sessions, tokens, total spend. Useful for billing. Not enough to run an AI program, which needs to know where the money went, whether the usage produced work, and where sessions burn time. That takes a different scorecard:

1. AI spend by outcome

2. Cost per meaningful engineering outcome

3. Productive AI usage versus exploration and overhead

4. AI capability adoption

5. AI session efficiency

They sit alongside DORA and quality metrics rather than replacing them.

Why traditional engineering metrics are not enough

DORA, cycle time, throughput, and defect rates remain the backbone of engineering measurement. Keep them. They just cannot answer the AI questions:

AI also moves those metrics in opposite directions at once. The 2025 DORA report associated higher AI adoption with increased delivery throughput and increased instability. Watching only those two shows the effect without the cause.

Perception is no substitute either. In a randomized controlled trial by METR, experienced developers in mature repositories took 19% longer with AI, having predicted a 24% speedup and still believing afterward they were 20% faster.

A survey would have recorded a win here.

Metric 1: AI spend by outcome

Definition: The share of AI spend allocated to features, fixes and rework, tests and reviews, and exploration and overhead.

AI spend by outcome (%) =
AI cost attributed to an outcome category / total AI cost × 100

AI spend by outcome (%) =
AI cost attributed to an outcome category / total AI cost × 100

Total spend tells you how large the bill is. This tells you what it represents. Two teams, same $30,000 a month, illustrative allocations:

The invoice total is identical. The management decision is not.

Entelligence research across more than 1 million pull requests and 2,444 organizations estimates that about $0.18 of each dollar spent on AI coding tools becomes shipped product. That is vendor-produced research, not a universal benchmark, but it shows why allocation matters.

Watch for unbounded exploration that never converts, or rework rising with no change in delivery. Zero exploration is not the target. Segment it, because an org-wide average hides one team spending 60% on exploration while another spends 5%. Decides where to raise, hold, or cut a budget. Agent Insights shows the allocation, including untracked spend.

Metric 2: Cost per meaningful engineering outcome

Definition: AI spend divided by a countable unit of delivered work. Start with a denominator you can measure consistently.

Cost per merged pull request =
AI spend attributed to development work / AI-assisted merged pull requests

Cost per merged pull request =
AI spend attributed to development work / AI-assisted merged pull requests

Those hypothetical figures do not prove Team 1 is better. The teams may work on different systems, and as Microsoft Research notes, a merged PR is not equivalent to the value it delivers. It still beats reporting 38 billion tokens.

Watch for spend rising faster than shipped work. Compare within similar work and pair it with change failure rate and PR size, so nobody improves the number by splitting work. Decides model and vendor selection: if frontier models double the cost of routine outcomes with no quality gain, the placement policy should change, which is the case for an LLM router. Upgrade the denominator toward shipped features as attribution matures.

Metric 3: Productive AI usage versus exploration and overhead

Definition: The share of usage that reaches accepted or shipped work, against usage lost to retries, repeated edits, abandoned sessions, and oversized context.

Productive usage rate =
AI usage attributed to accepted or shipped work / total AI usage

Productive usage rate =
AI usage attributed to accepted or shipped work / total AI usage

Perfect attribution is hard. Start with one minus the share of tokens consumed by identifiable overhead. High consumption can look like engagement while being thrash:

Watch for outlier sessions eating a large share of tokens, rising retries, and expensive models looping on tasks they never finish. Segment by session type, since interactive pairing and background agents have different shapes. Do not call all exploration waste: bounded prototyping that informs delivery is productive. Decides where to fix context hygiene, guardrails, or task specs before adding budget.

Metric 4: AI capability adoption

Definition: The percentage of a team actively using specific agent capabilities, not merely holding a license.

Capability adoption rate (%) =
engineers who used capability X during the period / engineers with access × 100

Capability adoption rate (%) =
engineers who used capability X during the period / engineers with access × 100

Track each capability separately. Each signals something different:

Seat activation is the weakest possible signal. One accepted autocomplete makes a developer look active in a vendor report.

Watch for paid seats with little recurring use, an expert minority with no wider spread, or heavy autonomous-agent use without task tracking and guardrails. Decides training, license rightsizing, and which champions' practices to standardize. One caveat matters most: adoption stage is a coaching signal, not a performance score. Do not rank engineers by it.

Metric 5: AI session efficiency

Definition: How effectively a session converts context and tool use into progress, complemented by retry rate.

Retry rate (%) = retried agent actions / total agent actions × 100

Retry rate (%) = retried agent actions / total agent actions × 100

Efficiency itself is usually a composite. Define yours explicitly and keep it stable, because a metric that changes meaning each quarter cannot show a trend. Two teams with identical spend and adoption can still work in very different environments:

Delivery metrics surface that gap weeks later, when the workflow is no longer fresh.

Watch for tokens rising while efficiency falls, retry hotspots in one repository, or sudden deterioration after a model change. Retries usually expose system problems: ambiguous tasks, missing permissions, unreliable tests, or tool failures. Decides what to fix around the agent, and whether a model or routing change helped, the way our 89-task Terminal-Bench comparison measured one. Never a developer-quality score: it describes the interaction among engineer, task, context, model, and tooling.

Put the five metrics into one scorecard

The last column is the critical one. If a metric does not change a decision, it is not yet part of the operating system.

Review at three speeds

For most AI metrics there is no credible universal benchmark. Set an internal baseline over 60 to 90 days and manage the trend.

Metrics that look useful but mislead on their own

Do not discard them. The right-hand column is the point: a usable ingredient, a misleading headline.

How to build an AI insights dashboard

Perfect attribution on day one is not required.

1. Connect the AI sources. Usage from the coding agents and model providers your teams actually use.

2. Map identities, teams, repositories, and projects. Bad mapping makes every comparison downstream less trustworthy.

3. Set project budgets. Budgets turn spend into a managed variable with run rate and variance.

4. Validate outcome attribution. Check a sample of known projects before reporting categories upward.

5. Configure the five-metric scorecard. Give each metric an owner, cadence, and decision.

6. Record a baseline. Observe 60 to 90 days before treating a movement as a pattern.

7. Set responsible-use guardrails. Team and individual data should improve systems and coaching, never become a standalone performance score.

8. 
   

Order matters more than speed. Steps 1 to 3 are plumbing, and stopping there produces connected data nobody owns.

Entelligence Agent Insights brings spend, usage, adoption, session behavior, and outcomes together across Claude Code, Codex, and Cursor. GitHub Copilot support is listed as coming soon.

Measure what the AI investment produces

You need enough visibility to set a baseline, expose waste, and connect AI activity to outcomes.

• Spend by outcome so budgets reflect what AI produces.

• Cost per outcome so the investment survives a finance review.

• Productive usage so activity stops impersonating progress.

• Capability adoption so licenses become repeatable practice.

• Session efficiency so rework is fixed at the system level.

The five are not a scoreboard. They are a control system.

Explore Entelligence Agent Insights or read the documentation.