When Should AI Adoption Be Measured by Business Metrics? — Revisiting AHEAD and LEVER 2/2

TL;DR

• While learning within one team, track changes in the harness and review burden.
• When expanding to other workflows, track total delivery cost and business outcomes.
• AHEAD and LEVER are lists of questions, not composite scores.

Introduction

In Part 1, I described AI adoption as connecting the work environment, reducing failures within one team, and extending a proven approach to other workflows.

But asking about cost savings both while a team was learning and after expansion blurred the distinction between those stages. It also failed to show clearly how costs saved in code generation could move into review, CI, and operations.

Writing the post on review cognitive load clarified what I wanted to check first: whether the burden of understanding and approving results is actually falling.¹

While working on the software delivery cost post, I came to think that after expansion we should measure the full cost of getting software to customers.²

In this post, I want to revise the questions we ask at each stage accordingly.

1. Evaluate from the beginning, but ask different questions

Before adoption, record the time, quality, cost, and human intervention involved in the current workflow.

In the earlier post, I called the stage of connecting the work environment Streamlining, the stage of improving the harness within one team Shape, and the stage of extending a proven approach Scale. A harness means the context, tools, permissions, and validation system an agent uses.

I named the question list for Shape AHEAD and the list for Scale LEVER. Both are criteria I propose, not validated standards or composite scores. CTS-SW measures the total cost per unit of software delivered to customers.²

Applying Scale’s return-on-investment criteria directly to Streamlining and Shape makes it easy to mistake upfront investment for failure.

Kiro’s guide for managers reports that high-performing teams initially slowed down, distinguishing them from teams that expected immediate gains without changing their workflows.³ This observation supports my distinction between asking about learning early and outcomes after expansion. Kiro did not propose or validate AHEAD and LEVER.

The opposite is also dangerous. If a production workload is permanently labeled a learning experiment, nobody has to explain its cost or results. Stage names should change the next decision, not merely justify investment.

2. AHEAD — measure harness learning and review burden

During Shape, the important question is not how many artifacts an agent produced, but whether recurring failures and decisions are becoming less common.

I changed AHEAD’s E from Efficiency, meaning cost efficiency, to Evidence Quality, meaning the quality of validation evidence. The updated AHEAD consists of the following five questions.

Autonomy Boundary does not mean maximizing the automation rate.

Low-risk normal paths with clear contracts should let the agent complete implementation and verification. Contract changes, conflicts with existing decisions, and security or data risks must still reach humans reliably.

Low human involvement alone does not prove good autonomy. If necessary escalations disappear as well, Dependability has declined.

Harness Learning asks whether recurring review decisions have become contracts, tests, rules, or tools that the next agent can use. If people keep finding the same problem, it is difficult to say that the harness has learned.

Kiro’s continuous-improvement principle recommends updating rules, tools, and context whenever an agent takes a wrong turn or unnecessarily calls on a human, so the same problem does not recur.⁴ Translating that principle into an evaluation question leads me to examine whether the causes of recurring intervention diminish in later runs, rather than measuring usage.

The new Evidence Quality item directly addresses review cognitive load.

If an agent reports only that the tests passed, the human still has to read the code from the beginning. To narrow review, the result must show what satisfied each contract, which evidence supports it, which risks remain unverified, and which new decisions were made during implementation.

Look at whether the scope people need to understand has narrowed to contract changes and exceptions, rather than at report length. Keep code-level review where it is needed, such as security, payments, and data migrations, without demanding the same depth of review for every change.

The managers’ guide describes a team whose review focus moved from code style and naming to interface definitions and architectural decisions after it introduced earlier automated checks and tests.³ This supports examining what humans now judge during review. Using contract-level evidence quality to assess that change is my proposal here.

Adoption looks at operational ownership rather than tool logins.

The domain team must trust the output, classify failures, and change evaluation criteria directly. If the workflow stops when the harness engineer leaves, it has not yet become an organizational capability.

The five AHEAD dimensions should not be collapsed into one score.

Chasing automatic completion alone can hide risks. Adding manual review to every change in the name of reliability, however, can reduce the incentive to improve evidence and make the workflow harder for domain teams to use. I think we should examine how the method chosen to improve one dimension affects the others. This follows the same reasoning behind SPACE’s warning against reducing productivity to one activity metric.⁵

3. The boundary between Shape and Scale

Passing a representative evaluation set several times is not enough to declare Scale.

The following conditions need to appear together:

• Recurring failures become contracts, tests, rules, and tools.
• Normal paths complete without repetitive human approval.
• Humans receive contract changes and important exceptions accurately.
• The domain team manages the harness and operational metrics directly.
• Monitoring, rollback, and responsibility boundaries are ready.

If review queues keep growing or humans must reconstruct every implementation, Shape is not complete.

Declaring Scale after improving only code generation increases change volume while moving the bottleneck into review, CI, and operations. DORA’s description of AI as an amplifier of existing organizational strengths and weaknesses reflects the same dynamic.⁶

Moving to Scale is not a declaration that the harness is perfect.

It means the defined scope can operate with repeatable quality and risk controls, and that new contracts or exceptions can send the workload back into Shape.

4. LEVER — measure end-to-end delivery cost and value capture

During Scale, examine what changes when the same harness is reused for the next requirement.

The updated LEVER is:

I changed Extraction Efficiency to End-to-end Efficiency because cost moves.

Even when code generation time and token use fall, efficiency has not improved if review queues, CI retries, and incident response grow. The total cost of models, tools, humans, and operations must be connected to software that reaches customers.

CTS-SW can provide a starting point for this dimension.²

CTS-SW itself is not business value. A cheaper deployment can still deliver a feature that customers do not want.

That is why Value Realization remains separate.

The definition of value depends on the workload. At the beginning, decide whether success means faster customer request handling, more capacity with the same team, higher revenue, or lower operational risk.

Extension & Reuse examines additional preparation rather than feature count.

If every new workload requires prompts, tools, evaluations, and permissions to be rebuilt from scratch, the earlier investment has not been reused. If existing contracts and evaluation cases need only a small extension, the effect of Scale is starting to appear.

Kiro also recommends introducing organization-wide rules during expansion and sharing early adopters’ experience and context during sustained operation.³ This aligns with Extension & Reuse: examine what teams actually reuse alongside how many teams use the tools.

Reliability checks whether cost reduction has been transferred elsewhere.

If CTS-SW falls while change failures and on-call work rise, it is difficult to call the result an improvement. Trends within the same team need to be read alongside quality measures.

The managers’ guide explicitly recommends measuring correctness as well as speed during expansion.³ This aligns with LEVER’s position that lower delivery cost accompanied by worse quality does not count as an improvement.

5. When code generation accelerates but review queues grow

Suppose a team adds several agents. Pull requests now appear quickly, but the review queue becomes longer.

Looking only at code volume makes the rollout look successful.

AHEAD exposes a different picture:

• Autonomy Boundary: even normal changes wait for human approval.
• Harness Learning: the same review comments never become tests or rules.
• Evidence Quality: humans read the entire diff because contract-level evidence is missing.
• Adoption: the domain team uses the output but cannot improve the harness.
• Dependability: nobody has defined which risks appear if review is reduced.

Adding more agents in this state only produces unread changes faster.

First, move recurring review decisions into contracts and tests. Require the agent to present requirement-level evidence and remaining risks. Only after humans can review exceptions instead of reconstructing the full implementation will generation speed become delivery speed.

flowchart LR
    G["More code generated"] --> Q["Longer review queue"]
    Q --> A["Use AHEAD to find the cause"]
    A --> H["Improve contracts, tests, and evidence"]
    H --> D["Humans judge exceptions"]
    D --> L["Use LEVER to check cost and value"]

LEVER comes next.

Check whether shorter review queues reduced the time to reach customers, whether CTS-SW fell without harming quality, and whether the change produced an actual business outcome.

6. Use metrics as the order of a conversation, not as targets

Turning every AHEAD and LEVER dimension into a KPI recreates numerical optimization.

A target for automatic completion can encourage teams to avoid necessary human review. A target for lower CTS-SW can encourage them to exclude maintenance and incident costs or split software units into meaningless fragments.

I would instead establish a baseline for one workload in one team and hold the conversation in this order:

1. Where is the largest current bottleneck or risk?
2. Which AHEAD dimension should this change affect?
3. How did review burden and quality actually change?
4. After Scale, what changed in CTS-SW and LEVER?
5. Did cost move into another stage or onto another person?

Every dimension does not need to improve every week.

A new security requirement may temporarily increase human review and lead time. If that decision becomes a contract and guardrail that reduces repeated review in later work, it still represents learning during Shape.

Metrics are closer to a record of where to improve next than a score that declares success.

Conclusion

For now, I think AHEAD and LEVER are best used as checklists that keep stage-appropriate questions from being skipped, not as organizational scoring formulas.

Applying them to real workloads will probably change some names and signals. I still expect the order to remain: first examine learning and review burden during Shape, then ask about end-to-end cost and value during Scale.

• #ai
• #agent
• #agentic-development
• #harness-engineering
• #organization
• #developer-experience
• #cts-sw