Do AI Coding Agents Improve Productivity?

The paper Adoption and Impact of Command-Line AI Coding Agents was published on 1 July 2026. It studies tens of thousands of engineers during Microsoft's early rollout of Claude Code and GitHub Copilot CLI.

The researchers asked three practical questions: who tries the tools, who keeps using them and whether adoption changes measurable output enough to justify substantial token spend.

This is observational organisational research rather than a laboratory benchmark. Its strength is scale and real workplace behaviour. Its limitation is that adoption is not randomly assigned in the same way as a controlled trial.

First use spread primarily through social networks. Engineers were more likely to try a coding agent when visible peers used it. Adoption was not simply a function of demographics or formal training.

Retention was more closely associated with coding activity than with demographics. People with work that created repeated opportunities to use the tools were more likely to continue.

Most prominently, adopters merged roughly 24% more pull requests than the researchers estimated they would have without adoption. The lift persisted across the four-month observation window, arguing against a short novelty spike.

These findings suggest coding-agent rollouts are partly social systems. Product access matters, but examples, peer confidence and suitable tasks influence whether access becomes routine use.

A merged pull request is observable and consistent across a large organisation. It is a reasonable proxy for completed software work, but it is not the same as business value.

PRs vary enormously. A documentation correction and an architectural change each count as one. More merged PRs can reflect useful decomposition, but they can also reflect smaller batches or work that shifts review effort to colleagues.

The result is best read as evidence that command-line agents increased one concrete form of engineering output among adopters in this setting. It is not a universal productivity coefficient and should not be applied directly to headcount planning.

Adopters may differ from non-adopters in motivation, workload or comfort with new tools. The study uses counterfactual estimation, but no observational method removes every unmeasured difference.

The paper does not reduce code quality to PR count. Teams still need defect rates, review time, rollback frequency, security findings and customer outcomes.

There can also be spillovers. An agent may help one engineer create more changes whilst increasing review work for others. Organisation-level productivity must include the whole delivery system.

Token spend matters as well. A lift can be real but uneconomic if poorly scoped tasks trigger long, repeated runs. Cost per accepted task is more informative than cost per prompt.

Visible peer use matters. Seed a rollout with respected engineers working on representative tasks, then let them share concrete examples, failures and permission practices.

Train around workflows, not clever prompts. Teams need patterns for issue selection, context management, test execution, code review and safe terminal access.

Keep the human responsible for intent, constraints and verification. The agent can explore and implement, but the engineer should approve architecture, inspect the diff and own the outcome.

Do not reward raw PR volume. That can encourage fragmentation and low-value changes. Reward reliable delivery, maintainability and reduced time to validated outcomes.

Coding agents are well suited to bounded maintenance, test creation, repetitive migration, documentation updates and changes with clear acceptance criteria.

They are less dependable when requirements are ambiguous, architecture is contested or the repository lacks tests. Faster code generation can accelerate the wrong plan.

Teams should maintain an explicit task taxonomy. Compare agent-assisted and unassisted outcomes within similar classes instead of averaging migrations, bug fixes and new product work into one number.

Start with output: accepted tasks, merged PRs and cycle time. Add quality: review rounds, escaped defects, rollbacks and security findings. Add system cost: reviewer time, CI minutes and model spend.

Then measure experience. Developers should report whether the tool reduces tedious work, creates cognitive overhead or makes unfamiliar code easier to understand. Retention without satisfaction may simply reflect organisational pressure.

Finally, examine distribution. An average gain can hide groups that benefit greatly and groups that lose time. Tooling, repository maturity and task mix should guide where the rollout expands.

The Microsoft study is stronger evidence than a vendor demo because it observes sustained use at organisational scale. It suggests command-line agents can produce a meaningful lift in merged software output.

It does not prove every merged PR is valuable or that every team will achieve the same result. The 24% figure belongs to a particular rollout, population, tool set and proxy.

Run a measured rollout, instrument the full delivery process and treat adoption as a social and workflow change. Compare model and coding-agent results in the AI Tools Review benchmarks hub.