Do AI Coding Agents Improve Productivity?

The paper Adoption and Impact of Command-Line AI Coding Agents was published on 1 July 2026. It studies tens of thousands of engineers during Microsoft's early rollout of Claude Code and GitHub Copilot CLI.

The researchers asked three practical questions: who tries the tools, who keeps using them and whether adoption changes measurable output enough to justify substantial token spend.

This is observational organisational research rather than a laboratory benchmark. Its strength is scale and real workplace behaviour. Its limitation is that adoption is not randomly assigned in the same way as a controlled trial.

First use spread primarily through social networks. Engineers were more likely to try a coding agent when visible peers used it. Adoption was not simply a function of demographics or formal training.

Retention was more closely associated with coding activity than with demographics. People with work that created repeated opportunities to use the tools were more likely to continue.

Most prominently, adopters merged roughly 24% more pull requests than the researchers estimated they would have without adoption. The lift persisted across the four-month observation window, arguing against a short novelty spike.

These findings suggest coding-agent rollouts are partly social systems. Product access matters, but examples, peer confidence and suitable tasks influence whether access becomes routine use.

The headline figure is more precise than "roughly 24%". The paper's CausalImpact analysis estimates a +24.0% lift in merged PRs per engineer per day across the post-period, with a 95% credible interval running from +14.5% to +33.7%. That interval matters: the true effect in this population is very unlikely to be zero, but it could plausibly be anywhere from modest to substantial.

More interesting is the dose-response pattern. Engineers who used an agent about one day per week saw an estimated lift of around +15.0%, whilst those using one on five or more days per week saw a lift of about +50.1%. The effect scales with intensity of use rather than arriving as a flat bonus for having access.

Two readings of that gradient are possible, and both are probably partly true. Heavier use may genuinely compound, as engineers build workflows, prompt patterns and trust that make each session more productive. But heavier users may also be the people whose task mix suits agents best, so the gradient partly reflects selection rather than pure treatment effect.

The practical implication for leaders is the same either way: a licence that gets opened once a fortnight delivers little. If you believe the tools are worth deploying, the evidence favours investing in the habits and task-routing that turn occasional users into daily ones.

The paper also breaks the effect down by tool. In weeks with any use, Copilot CLI adopters showed an estimated PR lift of +24.9% (interval +23.0% to +26.8%), against +11.4% (interval +9.4% to +13.6%) for Claude Code adopters, roughly a 2.2 times difference in this setting.

Resist the temptation to read that as a general verdict on tool quality. The study ran inside Microsoft, which builds Copilot CLI, so internal documentation, support channels and integration with in-house systems plausibly favoured the home tool. Nor were engineers randomly assigned a tool: the populations choosing each agent may differ in role, codebase and task mix.

The comparison is also frozen in time. The observation window closed in late April 2026, and the underlying models have since turned over: Anthropic's line-up now centres on Opus 4.8, Sonnet 5 and Fable 5 (restored on 1 July after its June suspension), whilst OpenAI shipped GPT-5.6 on 9 July. A tool gap measured in the first quarter tells you little about the tools as they exist today.

The durable lesson is not which agent won, but that tool-level differences within one organisation can be as large as the headline effect itself. Teams evaluating agents should run their own within-team comparisons rather than importing another company's ranking, and factor in running costs using guides such as our Claude Code pricing breakdown.

The adoption analysis quantifies just how social the spread was. Engineers whose skip-level network (colleagues two reporting hops away) had at least 25% of members already using an agent showed +216% higher odds of trying one themselves. Where at least a quarter of an engineer's code-review peers were users, odds of trying rose by around +54%.

Management example mattered too: engineers whose direct manager used a coding agent were about +82% more likely to try one. Formal channels, demos, mandates and training courses were consistently weaker predictors than simply seeing trusted colleagues use the tools.

The paper defines retention operationally as activity on at least five of the fourteen days after first use, which is a deliberately demanding bar. By that standard, sticking with the tools correlated with having coding-heavy work, not with seniority or demographics: people whose weeks contained many agent-suitable tasks kept using them.

One caveat the authors themselves flag: observational network data cannot fully separate peer influence from homophily. Engineers who sit near early adopters may resemble them in ways that independently predict adoption. Even so, the sheer size of the network effects makes the practical advice hard to ignore.

A merged pull request is observable and consistent across a large organisation. It is a reasonable proxy for completed software work, but it is not the same as business value.

PRs vary enormously. A documentation correction and an architectural change each count as one. More merged PRs can reflect useful decomposition, but they can also reflect smaller batches or work that shifts review effort to colleagues.

The result is best read as evidence that command-line agents increased one concrete form of engineering output among adopters in this setting. It is not a universal productivity coefficient and should not be applied directly to headcount planning.

Adopters may differ from non-adopters in motivation, workload or comfort with new tools. The study uses counterfactual estimation, but no observational method removes every unmeasured difference.

The paper does not reduce code quality to PR count. Teams still need defect rates, review time, rollback frequency, security findings and customer outcomes.

There can also be spillovers. An agent may help one engineer create more changes whilst increasing review work for others. Organisation-level productivity must include the whole delivery system.

Token spend matters as well. A lift can be real but uneconomic if poorly scoped tasks trigger long, repeated runs. Cost per accepted task is more informative than cost per prompt. The paper itself notes that at organisational scale, token spend can run into millions of dollars annually, which is why the size and durability of the lift matters commercially, not just academically.

For readers who want to weigh the evidence properly, the design is worth understanding. The observation window ran from 5 January to 29 April 2026, with a thirteen-week pre-period from 1 October 2025 to 4 January 2026 used to establish each engineer's baseline output before the tools arrived.

The headline +24.0% estimate comes from a Bayesian structural time-series analysis (Google's CausalImpact approach) applied to a cohort of early adopters who first used an agent between 5 and 11 January 2026. A synthetic control built from comparable non-adopters models what the cohort's PR output would have looked like without adoption; the lift is the gap between observed and modelled output.

A second analysis uses fixed-effects Poisson regression with engineer and week fixed effects. This within-person design compares each engineer's weeks of agent use against their own non-use weeks, absorbing stable individual differences such as skill or role, and it is the analysis behind the per-tool and dose-response estimates.

The authors are candid about the limits: one company, one four-month period, no randomised assignment, an outcome metric that ignores PR value, and no visibility into effects beyond the window. Those are real constraints, but the combination of two independent identification strategies pointing the same direction is considerably stronger than most vendor-published evidence in this space.

Visible peer use matters. Seed a rollout with respected engineers working on representative tasks, then let them share concrete examples, failures and permission practices.

Train around workflows, not clever prompts. Teams need patterns for issue selection, context management, test execution, code review and safe terminal access.

Keep the human responsible for intent, constraints and verification. The agent can explore and implement, but the engineer should approve architecture, inspect the diff and own the outcome.

Do not reward raw PR volume. That can encourage fragmentation and low-value changes. Reward reliable delivery, maintainability and reduced time to validated outcomes.

Coding agents are well suited to bounded maintenance, test creation, repetitive migration, documentation updates and changes with clear acceptance criteria.

They are less dependable when requirements are ambiguous, architecture is contested or the repository lacks tests. Faster code generation can accelerate the wrong plan.

Teams should maintain an explicit task taxonomy. Compare agent-assisted and unassisted outcomes within similar classes instead of averaging migrations, bug fixes and new product work into one number.

Start with output: accepted tasks, merged PRs and cycle time. Add quality: review rounds, escaped defects, rollbacks and security findings. Add system cost: reviewer time, CI minutes and model spend.

Then measure experience. Developers should report whether the tool reduces tedious work, creates cognitive overhead or makes unfamiliar code easier to understand. Retention without satisfaction may simply reflect organisational pressure.

Finally, examine distribution. An average gain can hide groups that benefit greatly and groups that lose time. Tooling, repository maturity and task mix should guide where the rollout expands.

The Microsoft result lands in a literature that is genuinely mixed. The most cited counterpoint is METR's randomised controlled trial, published in July 2025, which found that experienced open-source developers completed real tasks about 19% slower when allowed to use AI tools, even though the same developers estimated afterwards that AI had made them roughly 20% faster.

The two studies are less contradictory than they first appear, because they measure different things in different settings. METR studied 16 highly experienced maintainers working on 246 issues in mature repositories they knew intimately, primarily using Cursor Pro with Claude 3.5 and 3.7 Sonnet between February and June 2025. The Microsoft study observed tens of thousands of engineers using a newer generation of terminal agents across ordinary corporate work in early 2026.

A coherent reading is that AI assistance helps least where the human is already near the performance ceiling on deeply familiar code, and helps most across the broad middle of everyday engineering: unfamiliar corners of a codebase, boilerplate, tests, migrations and well-specified changes. Tool generation matters too; agentic CLI tools that run tests and iterate autonomously are a different proposition from the editor-assist workflows METR measured.

The perception gap METR documented is the enduring warning for both camps. Developers systematically misjudge whether AI is helping them, in either direction, so self-reported productivity is close to worthless as a rollout metric. Instrumented outcomes of the kind the Microsoft paper uses, however imperfect, are the only defensible basis for deployment decisions.

The Microsoft study is stronger evidence than a vendor demo because it observes sustained use at organisational scale. It suggests command-line agents can produce a meaningful lift in merged software output.

It does not prove every merged PR is valuable or that every team will achieve the same result. The 24% figure belongs to a particular rollout, population, tool set and proxy.

Run a measured rollout, instrument the full delivery process and treat adoption as a social and workflow change. Compare model and coding-agent results in the AI Tools Review benchmarks hub.