Why Did Some Teams Get Up to 10x Faster with the Same AI Tools? — Five Habits of Frontier Development

TL;DR

• Teams using the same tools achieved very different results.
• Frontier teams increased agent autonomy through five habits.
• Review and decision-making became the next bottlenecks after coding.

Introduction

While working on the CTS-SW post about software delivery costs across development, review, and operations, I looked for evidence about how much AI coding tools actually change delivery speed.

The numbers varied widely. Some teams reported modest improvements, while others claimed gains of several times or even more than 10x.

I then found a talk that explained this difference through teams inside the same organization.

AWS Senior Principal Engineer Clare Liguori presented the characteristics of Frontier Development that Amazon observed across its internal teams.¹

The talk ranges from a system estimated to require 30 people for 18 months but built by six people in 76 days, to a comparison of 50 teams maintaining existing codebases.

The speaker organized the working practices of teams with larger gains into five habits. I was particularly interested in the differences she observed among teams using the same tools.

1. AI coding is entering a fourth phase

Clare divides the evolution of AI coding into four phases.

The first is inline code completion, which proposes the next line or function.

The second is chat, where developers ask questions about code. The third is Vibe Coding, where a high-level request is implemented through a conversation with an agent.

Clare says that, anecdotally, these three phases made her feel roughly 10–20% more productive.

She calls the fourth phase, which is only beginning, Frontier Development.

In the earlier phases, a person continuously controls each line, function, or exchange. In Frontier Development, an entire task—including how it will be validated—is delegated to an agent.

flowchart LR
    C1["1. Inline Completion<br/>Next line · function"] --> C2["2. Chat<br/>Questions about code"]
    C2 --> C3["3. Vibe Coding<br/>Implementation through conversation"]
    C3 --> C4["4. Frontier Development<br/>Delegate the whole task"]
    C1 -. "Continuous human control" .-> C3
    C4 -. "Agent executes and validates" .-> OUT["Validated result"]

Clare defines frontier developers through three behaviors rather than a particular product or model.

1. They write only about 1–2% of the code they produce.
2. They let agents work for hours without human intervention.
3. They minimize idle time by having several agents process a backlog in parallel.

These numbers describe early adopters inside Amazon whose productivity improved substantially, rather than standards every developer should follow.

2. Six people built it in 76 days—but the result was not automatically reproducible

The first case is the Bedrock Mantle team.

The Bedrock organization needed a new system for handling AI model inference requests and responses—an inference data plane—and initially estimated that the work would require 30 people for about 18 months.

It was a large project: build a new system, then migrate customers and models from the existing one.

Instead, six engineers built it with Kiro in 76 days.

Amazon had not seen a result like this internally. Measured by commits, the saved changes to a code repository, this initial pathfinder experiment showed an improvement of roughly 20x.

The problem was that these were not six ordinary engineers.

The team included two Distinguished Engineers and some of the company’s strongest experts in distributed systems, large language models (LLMs), and the architecture itself. It was also a new system built without the constraints of an existing codebase.

The case proved that the result was possible. It did not prove that another team could reproduce it during ordinary work.

The next experiment took place in Prime Video.

Six different engineers used Kiro without restriction during a ten-day sprint. Based on their progress, the estimated project duration fell from 90 weeks to 24.

This suggested that engineers outside the Bedrock Mantle team could also work faster. But 24 weeks was a revised estimate based on progress during the experiment, not the actual time to complete the project. The experiment also came with unusual conditions.

The team had almost no on-call responsibilities or meetings, and the interruptions common in an engineer’s normal day were deliberately limited. A senior engineer had also spent three weeks preparing small, well-scoped tasks and detailed requirements.

This sprint—a short, focused development period—was deliberately arranged to give agents favorable conditions, unlike the team’s normal working environment.

Amazon Stores therefore ran a pilot, a trial adoption under more realistic conditions.

It observed 50 teams with normal seniority distributions, working on existing systems and codebases, over an extended period.

The metric also changed from commit volume to how quickly changes reached production.

The teams split into two groups.

• Half improved deployment velocity by less than 3x.
• The other half reached a median of 4.5x, with some exceeding 10x.

Ninety percent of the teams used Kiro along with nearly the same internal tools.

Tool choice alone could not explain the performance differences between teams.

The teams with larger gains did more than place AI tools on top of the existing development process. They intentionally changed how they worked.

Across the three cases, the question shifts from what expert engineers can build from scratch to whether teams maintaining existing systems can also work faster.

flowchart LR
    B["Bedrock Mantle<br/>6 engineers · 76 days"] --> P["Prime Video<br/>6 engineers · 10-day sprint"]
    P --> S["Amazon Stores<br/>50 teams on existing systems"]
    B --> BC["Top experts · new system"]
    P --> PC["Meetings and on-call limited<br/>3 weeks of task preparation"]
    S --> L["Half: less than 3x"]
    S --> H["Half: median 4.5x<br/>some above 10x"]

These figures come from Amazon’s internal observations as presented in the talk. The raw data and team-level methodology have not been published as a complete research study.

I therefore would not treat 4.5x as an expectation that can be transferred to another organization. I would focus on the differences in working practices observed among teams using the same tools.

3. Five habits of frontier teams

Amazon interviewed the Bedrock Mantle team, the Prime Video sprint, and teams from the 50-team pilot, then identified five shared habits.

Clare uses habit rather than practice to emphasize a way of working repeated every day, beyond one exceptional experiment.

1) Invest in agent context

People carry a great deal of knowledge that never appears in documentation.

They pass it to colleagues through Slack conversations, onboarding, mentoring, code reviews, and planning meetings. Information agents will need again can be kept in files.

Frontier teams kept this information in Skills and steering files, which provide task instructions and project rules. Whenever an agent made a mistake or worked in a way the team would not have chosen, they asked:

What was missing from the Skill or steering file that the agent needed?

They did not correct the result once and move on. They preserved the lesson as context available to the next execution.

Context should not only grow.

A do not rule added to work around an older model’s behavior may no longer be necessary for a newer one. Keeping it can leave the agent with more instructions to read without helping the task.

The habit is therefore two-sided: add rules when new failures reveal missing context, and remove rules when stronger models no longer need them.

2) Slow down to speed up

Almost every interviewed team reported that productivity initially fell as it adopted frontier practices.

Giving an agent a coding tool does not immediately make it productive in an existing codebase.

The teams first made their environments easier for agents to work in.

• They improved error messages so failures explained what went wrong.
• They built new tools and MCP servers, which connect tools to agents through a common protocol, for tasks the agent could not perform.
• They restructured codebases that were difficult for agents to navigate.
• They added linters, which check code for errors or rule violations, and tests.
• When necessary, they moved to languages whose type systems and compilers returned more useful feedback.

The talk mentions teams moving from Python or JavaScript to TypeScript, and others choosing Rust because its compiler returns specific errors.

The useful lesson is why teams changed languages. They made substantial engineering investments to reduce how much agents had to guess and to show them what to fix after a failure.

3) Feed agents instead of babysitting them

One of Clare’s strongest phrases is feeding agents, not babysitting agents.

In Vibe Coding, a person may spend the entire day exchanging short messages with an agent.

The person waits 30 seconds or a minute for code, reviews it, and sends the next instruction. That pattern makes it difficult to run several agents in parallel.

Frontier teams provided the following information before the agent began:

• What needs to be done
• Which constraints must hold
• How the agent should validate its own work
• Which quality bar it must meet before returning

The agent runs and compiles the code, checks test results and coverage—how much of the code the tests exercised—and returns only after meeting the quality bar.

Repeated instructions move into steering files so they do not need to be typed again for the next task.

This allows the agent to correct its own failures while the person does other work or starts another agent.

4) Make intent explicit before writing code

In a typical Vibe Coding session, a developer gives a high-level prompt, receives a large amount of code, and then corrects the intent while reviewing the output.

The conversation becomes: "That is not what I meant," "You misunderstood the requirements," or "I did not want it structured this way."

Clare argues that discussing intent through code is inefficient when the intent itself is still wrong.

For complex or ambiguous features, Amazon teams first documented the behavior and result expected in a particular situation. Expressing expected behavior through concrete examples is the approach used in Behavior-Driven Development (BDD).

The agent generated a draft specification, and the person and agent adjusted requirements and technical design in a document that was easier to change than code. Code generation began only after the intent was aligned.

5) Bring testing earlier into development

For an agent to work for hours without human intervention, it needs fast feedback.

To help agents detect and correct mistakes quickly, frontier teams added linters, unit tests, integration tests, performance tests, and security tests.

These are established engineering practices, but an added test can now help an agent each time it retries a task. That makes investment in codebases machines can read and repair more valuable.

Several teams invested in local substitutes, or mocks, that return a defined response for a given input during tests instead of calling the real external service.

Connecting to cloud services and real environments during every iteration makes feedback slower and less predictable. When the same local input produces the same response, an agent can run more correction loops in less time.

I would connect the five habits in the following way.

flowchart LR
    subgraph BABY["Babysitting an agent"]
        P["Short prompt"] --> W["Wait for the result"]
        W --> C["Human corrects errors and intent"]
        C --> P
    end

    subgraph FEED["Feeding an agent"]
        I["Explicit intent and context"] --> A["Agent works"]
        T["Tools and local tests"] --> A
        A --> V{"Self-validation"}
        V -->|Fail| A
        V -->|Pass| R["Return validated result"]
    end

4. Frontier Development is hard on people too

Clare does not claim that adopting the five habits solves every problem.

This is still an early-adopter phase, and teams are learning a different way to work.

One risk is burnout: engineers may stay up late trying to create the perfect prompt that will run overnight and leave completed code ready in the morning.

Running several agents in parallel also means constantly switching between terminal tabs. The mental effort, or cognitive load, removed from implementation moves into tracking agent state and reviewing output.

Reviewing AI-generated code may feel harder than writing it directly.

The burden can be especially high for early-career engineers who have not yet spent much time reviewing other people’s code.

The organization has to change as well.

Productivity may decline first while a team repairs its codebase and tools and builds new habits.

If leaders ask, "We gave you strong AI tools, so why are you not faster?" while demanding the same feature output, the team cannot make that foundational investment.

Clare says teams may need to spend roughly two months changing the codebase and how they work.

Rolling the approach out across the entire organization too quickly creates another risk.

Amazon did not turn the result of one pathfinder team into an immediate company-wide standard. It ran a constrained sprint and learned from a 50-team pilot. At the time of the talk, it was working on how to extend the approach to the next 2,000 teams.

5. Decision-making becomes the next bottleneck after code

The talk ends with a new bottleneck encountered by frontier teams.

The talk uses an example in which building the code for a new product takes nine to twelve months.

Two months to decide whether to build the product and another two months to approve the launch were less visible within the overall schedule.

When coding falls to one or two months, the two-month stages before and after it become the longest parts of the process.

Clare says frontier teams often spend more time making decisions than writing code.

Placing the same workflow before and after Frontier Development makes the bottleneck shift easier to see.

flowchart LR
    subgraph BEFORE["Traditional development"]
        BD["Product decision<br/>about 2 months"] --> BC["Coding<br/>9–12 months"]
        BC --> BL["Launch approval<br/>about 2 months"]
    end

    subgraph AFTER["Frontier Development"]
        AD["Product decision<br/>about 2 months"] --> AC["Coding<br/>1–2 months"]
        AC --> AL["Launch approval<br/>about 2 months"]
    end

    BC -. "Shorter coding phase" .-> AC
    AD -. "New bottleneck" .-> AL

As code generation accelerates, the following steps become visible bottlenecks:

• Deciding which product to build
• Reviewing product and technical design
• Checking security and operational conditions
• Approving the launch

If even easily reversible decisions remain under lengthy review, organizational decision-making absorbs the speed gained from code.

Once the five habits had accelerated implementation, the teams also needed to shorten product decisions and release approvals.

6. Start with the current state before applying it to an organization

After watching the talk, I was left with a question: what should an organization inspect first before adopting this way of working?

The rest of this section is my proposal, based on earlier posts rather than the talk itself.

Establish a baseline for the current workflow

Before treating AI-tool adoption rates or generated code volume as outcomes, choose one workflow in one team and lay out the path from requirement to production.

• How long does it take to decide the requirement?
• Where do humans intervene during implementation and review?
• How much time is spent waiting for continuous integration (CI), which automatically builds and tests code changes, and for deployment?
• How often do rework, rollbacks, and incident response occur?
• Does the agent have the data, tools, and permissions required to finish the work?

Without a pre-adoption baseline, a team may not notice that time removed from code generation moved into review or operations.

Connect one team, then turn failures into the harness

The background information, tools, permissions, execution environment, and checks described above together form the harness. Once the current workflow is understood, the next step is to equip the agent with the harness it needs to finish the work.

Rather than expanding immediately to several agents and teams, I think it is better to first verify that one agent can complete one small task without human intervention.

When the same mistake repeats, preserve it in project rules. When the agent cannot perform an action, connect a command-line tool or MCP server; when it needs instructions, write a Skill. When it cannot detect a failure, add a linter or test.

The practical sequence is described in the post on building the EncBird harness layer by layer.²

At the organizational level, this can be divided into three stages.³

• First, connect workflow boundaries, data, tools, and permissions.
• Learn from a pilot in one team and turn failures and review feedback into the harness.
• Reuse the validated approach for other teams and requirements.

flowchart LR
    B["Current state<br/>Workflow boundary · baseline"] --> S1["Connect the workflow<br/>Context · tools · permissions"]
    S1 --> S2["Learn in one team<br/>Feed failures into the harness"]
    S2 --> S3["Expand a validated approach<br/>Reuse across teams and workflows"]

    B --> BM["Time from request to deployment<br/>Review · CI waiting<br/>Incidents · human intervention"]
    S2 --> M1["Fewer recurring failures<br/>Autonomy · evidence · dependability"]
    S3 --> M2["Total delivery cost<br/>Customer value · reuse · reliability"]

Measure different effects at different stages

Demanding immediate cost savings and business outcomes from an early pilot makes the talk’s slow down to speed up period look like failure.

While one team is learning, examine whether the same failures recur, whether humans still have to reconstruct all the code, and whether the agent can independently finish routine work that has no exceptions.

The questions change when the validated approach expands.

Examine whether requirements reach customers faster, whether total cost across development, review, and operations falls without reducing quality, and whether earlier investments are reused in other workflows.

The questions for these two stages are discussed in more detail in an earlier post.⁴ CTS-SW can serve as a starting point for measuring total delivery cost per unit of customer-delivered software.⁵

The 50-team case reports changes in working practices among teams whose deployment speed increased substantially. It does not establish how much each habit contributed, but it can help select practices to examine during a pilot.

After applying it to an organization, the next question is whether faster deployment improved total delivery cost and customer value.

Learn the way of working before execution becomes cheaper

The workflow described in the talk consumes tokens continuously.

Tokens are required not only for code changes and tests but also for feeding lessons into context, tools, and tests and for removing obsolete rules from the harness.

This remains an ongoing operating cost of maintaining and improving the workflow.

Recent studies consistently show a rapid decline in the cost of reaching the same performance level. Estimates vary widely, and long-reasoning frontier tasks can still become more expensive in total, but the downward price trend appears across several sources.⁶⁷⁸

METR’s evaluations also show growth in the length of software tasks agents can complete with a 50% success rate. Length here means the time a person would need to finish the task; it does not mean an agent can work reliably for that long on a company’s actual tasks.⁹

These studies do not directly measure business value per token. Even so, as the same performance becomes cheaper and models finish longer tasks, I think the business value available from a given token budget is increasing.

The recent trend should not be extrapolated mechanically. Still, a scenario in which the same work costs tens of times less in two or three years is worth considering when designing a workflow.

If an organization waits until execution costs are low enough before beginning the transition, it may struggle to catch teams that have already spent years accumulating context, tools, tests, and organizational habits. The cost of doing the same work can fall without the organization changing how it works at the same pace.

When designing workflows, I therefore think we should consider several scenarios for token prices and usage over the next three years, the work that budget can support, and how we will improve the harness during that time.

Conclusion

What stayed with me most from the talk was that the teams whose performance improved accepted slower feature delivery in the short term. They first spent time improving the context, tools, and tests their agents would use.

If I were applying this in an organization, I would start with a small workflow in one team. I would secure time and a token budget for improving the harness, then expand after checking whether recurring failures and repeated human intervention had decreased.

• #ai
• #agent
• #frontier-development
• #harness-engineering
• #organization
• #developer-experience
• #kiro