AI-generated code: what the data says about quality

A coding agent is an executor. That is not a criticism, it is the design. It does what it is told, at a speed no human matches, and it does not know whether what it was told was a good idea.

Which means the ceiling on the output is set somewhere else: by whoever writes the instruction and by whoever decides the result is acceptable. The interesting question is not whether AI writes code. It obviously does. The question is what moved when it started, and there is now enough data to answer that without guessing.

What the research actually says, including the part that got retracted

Three bodies of evidence get cited in this argument. Two say something real. One is quoted constantly and should not be.

The study everyone cites is out of date. In July 2025 METR ran a randomised trial: sixteen experienced open-source developers, 246 real issues in repositories they already knew. Tasks took 19% longer when AI tools were allowed, while the same developers believed they had been about 24% faster. It was a good study and it travelled a long way.

In February 2026 METR said that finding is outdated. Their follow-up work suggests speedups are now likely, but that changes in developer behaviour made the new results unreliable enough that they describe the evidence as very weak. The authors also never claimed the original result generalised to most developers or to beginners.

So if you see "AI makes developers 19% slower" quoted as current fact, in a blog post or a sales deck, the people quoting it have not checked. We are not going to use it either.

Delivery stability is the finding that holds. DORA's research consistently associates higher AI adoption with higher throughput and lower delivery stability at the same time. More changes ship, and a larger share of them need fixing. The mechanism usually described is batch size: AI produces larger diffs, larger diffs are harder to reason about, and review quality falls as the diff grows. Recent coverage of DORA's work calls this the instability tax and describes a J-curve, where the throughput payoff arrives after a dip. These are associations in observational data, not proof of causation.

Maintainability is the finding with the clearest trend. GitClear's 2025 report looked at 211 million changed lines authored between 2020 and 2024 and found an eightfold increase in duplicated code blocks during 2024, while refactored or moved code fell from around 25% of changed lines in 2021 to under 10% in 2024. Its June 2026 follow-up, across 623 million changes from 2023 to 2026, put duplication at 73.0 blocks per million changed lines against 40.3 in 2023, about 81% higher. Again: correlated with AI adoption over the same period, not isolated as the only cause.

Put together, the honest summary is not "AI code is bad". It is that volume went up, and the two things that depend on judgement went down. Duplication rises because an agent asked to solve a problem solves it, rather than noticing that something ninety per cent like it already exists four directories away. Stability falls because a bigger change gets a smaller proportion of real review.

The finding that survives everything

Strip out the numbers that moved and one result from the METR work is more useful than any of them: the developers were wrong about their own productivity, by a wide margin, in the direction that flattered the tool.

That is the thing to design around. If experienced engineers working in repositories they know cannot accurately judge whether the tool helped, then nobody is going to self-assess their way out of this. You need measurement, and you need someone whose job is to say no.

Where the work moved

The work did not disappear. It moved from producing code to deciding about code, and it concentrated in four places.

Specifying. An agent given "build a checkout" builds a checkout. An agent given the constraint that the order record is the single source of truth, that every total must be recomputed server side, and that the flow has to survive a double-submitted payment, builds a different one. The difference is not prompt phrasing. It is knowing which three constraints matter out of the thirty you could state, and that comes from having seen the failure modes.

Reviewing at the right size. The defence against the batch-size problem is boring: small changes, reviewed properly. An agent will happily produce a two-thousand-line diff. Accepting it is a decision, and it is usually the wrong one.

Noticing what is missing. Generated code is reliably plausible. What it omits is the thing nobody asked about: the authorisation check on the second endpoint, the index that the new query needs, the migration's behaviour on the rows that already exist, the error path that leaves the record half written. These are absences, and absences do not announce themselves in a diff.

Refusing duplication. This is the direct counter to the maintainability trend, and it is the one agents cannot do for themselves, because it requires knowing what already exists in a codebase they are seeing in fragments.

What this means if you are buying development

Three practical consequences.

Velocity claims are now nearly meaningless. Everyone can produce code quickly. Ask instead what the change failure rate is, how big a typical pull request is, and who reviews it. A supplier who cannot answer those is selling throughput into the instability tax.

Ask who is accountable for the parts the model cannot see. Architecture, data modelling, authorisation, the migration path. Those are the decisions that are expensive to reverse, and they are exactly the ones an executor is worst at, because they depend on context outside the file it was given.

Expect to inherit AI-written code, and price it. A growing share of takeover work is codebases where a model wrote a large part and nobody retained the reasoning. It is a recognisable shape: it works, it has no tests, the same logic appears in four places with small differences, and the original author cannot explain any of it. We wrote about the wider version of this problem in scaling an AI MVP that is still delivered by hand.

How we use them, concretely

We use coding agents daily. Pretending otherwise would be silly, and refusing to use them would be worse. What we do not do is let the output reach a client without passing the same gates as anything else.

In practice that means: changes small enough to review, a human who can explain every line in a pull request before it merges, tests that exist because they were asked for rather than because the agent felt generous, and a standing bias against accepting a second implementation of something the codebase already does.

The honest version of the value is this. An agent makes a good engineer considerably faster at the parts that were always typing. It does not make a non-engineer into one, because the skill was never the typing. It was knowing which of the plausible answers is correct here, and that judgement is the thing you are actually buying.

What to do

Measure the two numbers that matter. Change failure rate and time to restore. If throughput is up and those have moved the wrong way, you are paying the instability tax and you can see it.

Cap diff size and mean it. This is the single highest-leverage policy available, and it costs nothing.

Run a duplication check in CI. The trend in the data is duplication. It is measurable, so measure it rather than hoping.

Decide who says no. Every piece of generated code needs a named person who can reject it and explain why. If that person does not exist, the ceiling on your output is whatever the model produced on the first attempt.

---

This is the work we do. Smart-UI is a backend and integration team, and a growing part of what we are asked to do is take over code somebody else started, including the parts a model wrote. We will tell you plainly which parts of it are worth keeping.

If you have a codebase nobody can fully explain, the fixed-price technical audit covers the state of the code, the tests, the architecture and the risks, and comes back as ranked findings with reproduction steps. If you need the work done rather than assessed, tell us what you have and we will tell you where we would start.

Sources: METR's July 2025 trial, METR's February 2026 update, The Register on the original study, DORA's research on AI tensions, GitClear's Maintainability Gap report.