The Real Test of AI Code Is the Next Prompt

AI coding agents are incredible at the first move.

Give one a clean repository, a clear task, and a few minutes, and it can produce something that looks almost magical. A feature appears. The tests pass. The interface works. The demo is convincing.

The problem begins after the demo.

Real software does not stay clean. Requirements change. Features collide. Old assumptions remain hidden in the code. A decision that looked harmless in the first prompt becomes a constraint five prompts later. The product keeps moving, while the code preserves every version of what it used to be.

The real test of AI code is the next prompt.

EvoCode-Bench was built around this problem. Instead of giving an agent one specification and judging one final answer, it keeps the same workspace alive through five to fifteen rounds. Each round adds, corrects, or reverses requirements. The tests are cumulative, so the agent has to satisfy the new request while preserving everything that still matters from earlier rounds.

Its earlier output becomes its future environment.

The difference between isolated performance and persistent performance is huge. Across most agents in the benchmark, single-round scores from a clean reference-completed workspace were 22 to 40 percentage points higher than scores earned while maintaining the agent's own evolving codebase. The agent that ranked highest in the clean single-round condition did not rank highest in persistent execution.

Performance also fell as the interaction continued. The average multi-round pass rate dropped from 46.7 percent in the first round to 21.3 percent by round five and 7.7 percent by round ten. The later rounds included fewer tasks, so the far end of the curve deserves caution, but the broader pattern is difficult to dismiss. The isolated rounds remained similarly solvable. The accumulated workspace did not.

This is a different failure from simply being unable to code.

The agent can often implement the newest feature. It changes one part of the system successfully while quietly damaging another. It follows the latest instruction but loses track of what still needs to remain true. When a requirement is reversed, the agent may add the new behavior without fully removing the old assumptions embedded elsewhere.

The current task passes. The project becomes less trustworthy.

That distinction matters because most coding benchmarks reward visible completion. The patch is evaluated against the issue it was asked to solve. The benchmark rarely asks whether the patch left the repository easier or harder to change next week.

Software teams live inside the second question.

A codebase is a sequence of decisions that future decisions must inherit. Every schema, helper, abstraction, dependency, test, naming convention, and exception shapes the possible moves that come later. A piece of code can be correct today and still be a bad inheritance.

Several newer benchmarks are beginning to measure this inheritance directly.

SWE-EVO builds tasks from the release histories of mature open-source projects. Its changes span an average of 21 files and are checked against test suites averaging 874 tests. The benchmark asks an agent to interpret release-level requirements and coordinate changes across a larger system. The best results remain far below the scores agents achieve on isolated issue resolution.

EvoClaw reconstructs continuous development sequences from real commit histories. Models that scored above 80 percent when milestones were treated separately reached at most 38 percent when they had to carry their own work through the full sequence. The task did not suddenly require a new programming language. The difficulty came from temporal dependency, technical debt, and error propagation.

SlopCodeBench goes further by measuring what happens to the shape of the code. Agents repeatedly extend their own solutions as specifications evolve. Across the evaluated trajectories, structural erosion increased in 80 percent of cases and redundant or duplicated code increased in 89.8 percent. Agent-written code was 2.2 times more verbose than the human-maintained open-source repositories used for comparison.

The code could continue passing checkpoints while becoming harder to extend.

This may be the most important result. A green test suite can confirm that the known behaviors still work. It cannot fully tell us whether the system is becoming easier to reason about, easier to modify, or safer to inherit.

The benchmark can mark the feature as complete while the repository absorbs a future cost.

Software engineering has had a name for this problem for decades. In 1980, Meir Lehman described software as something that must continually adapt to remain useful. One of the laws later associated with this work says that complexity tends to increase as a system changes unless deliberate effort is made to maintain or reduce it.

AI has not escaped this law. It has accelerated the rate at which the law can operate.

A human team might accumulate architectural debt across months or years. An agent can generate a similar sequence of locally reasonable compromises during one long afternoon of prompting. The codebase becomes old without having existed for very long.

Each prompt leaves a layer behind.

The first request creates a structure. The second request extends it. The third discovers that the original assumption was incomplete. The fourth adds compatibility. The fifth introduces a special case. The sixth asks the agent to reverse the behavior from the second prompt without breaking the fourth.

By then, the agent is no longer generating inside an empty space. It is negotiating with its previous self.

This is why the phrase "AI coding productivity" can be misleading. Productivity is usually measured close to the person and close to the present moment. How quickly did the developer finish the task? How many lines were produced? How many tickets were closed? How much faster did the prototype appear?

The system pays on a longer timeline.

Google's DORA research has repeatedly found that AI behaves as an amplifier of the surrounding development system. Earlier research associated higher AI adoption with improved individual flow and reported productivity while delivery stability and throughput suffered. The 2025 report found a more positive relationship with throughput, while still warning that AI magnifies weak foundations and can trade speed for instability.

METR's randomized study of experienced open-source developers produced another uncomfortable result. Developers using early-2025 AI tools took 19 percent longer on tasks inside repositories they already knew well, even though they believed the tools had made them faster. The study involved only sixteen developers and reflects tools from a rapidly changing period. METR's later work also warned that newer measurements were difficult to interpret because developers increasingly refused to work without AI and used several agents at once.

The exact productivity number will keep moving.

The deeper issue will remain. Generating code faster can increase the amount of code that needs to be understood, reviewed, integrated, tested, and maintained. The bottleneck moves away from typing and toward judgment.

AI increases the supply of code faster than it increases the supply of responsibility.

Responsibility is the ability to understand why something exists, what depends on it, which requirements are still active, and what a new change could quietly destroy. It includes knowing when the latest request should replace an old behavior and when both must continue to coexist. It includes noticing that a local improvement may create a system-level regression.

Humans struggle with this too. We built tests, specifications, design systems, architecture diagrams, code review, decision records, release processes, and institutional memory because no individual can hold the entire evolving system in their head.

Agents need external memory for the same reason.

EvoCode-Bench found that tasks using persistent specification documents such as AGENTS.md scored much higher than conversational or contractual tasks. The authors are careful about the conclusion. Only four of the twenty-six tasks used this format, and task structure may explain part of the difference. The finding is promising rather than causal proof.

Still, the direction makes sense. A living specification gives the agent somewhere to find the current truth of the product. It can record which requirements replaced earlier ones, what must remain compatible, which tradeoffs were intentional, and which behaviors are protected.

Documentation stops being background material. It becomes part of the runtime.

The model may be temporary. The project memory must persist.

Tests can preserve behavior. Decision logs can preserve reasoning. Architecture maps can preserve relationships. Design systems can preserve interaction rules. A requirements file can preserve the current contract. Together, these artifacts form an external brain that survives individual prompts, sessions, models, and agents.

Memory alone will not solve the problem.

An agent can remember every requirement and still choose an implementation that makes the next change unnecessarily difficult. It can preserve behavior through duplicated branches, growing conditionals, oversized functions, and patches that avoid confronting a weak architecture. It can satisfy every known test while reducing the number of safe moves available later.

The standard therefore has to become stronger than correctness.

A mature engineering contribution leaves the system more understandable. It reduces ambiguity. It preserves useful constraints while removing obsolete ones. It makes future changes easier to locate, reason about, and verify.

This suggests a different way to evaluate coding agents. Give the agent a sequence of changing requirements. Let it inherit its own work. Then give the resulting repository to another agent, or to a human, and measure what happens next.

Can the next feature be added without rewriting the system? Can someone explain why the architecture looks the way it does? Can obsolete behavior be removed cleanly? Do the tests reveal the important contracts? Does the documentation still match reality? How many exceptions have accumulated? How much confidence does the next developer have before touching anything?

The output of one prompt should become the input of the evaluation that follows.

This also extends beyond coding. AI can create the first screen, first strategy, first campaign, first article, first research summary, and first organizational plan. The first version is becoming cheap.

The harder problem is keeping the hundredth version coherent with the ninety-nine that came before it.

Every project accumulates history. New decisions must coexist with old decisions. Some ideas need protection. Others need complete removal. The system has to change without losing the qualities that made it useful.

AI agents will face this problem at machine speed.

An agent can generate more changes than a human team can review. If each request creates a small invisible regression, the product may look increasingly productive while becoming increasingly fragile underneath. The greater danger is a long sequence of plausible successes that slowly reduces the system's ability to survive the next prompt, rather than a dramatic collapse after one terrible request.

The agent begins as the creator of the codebase. A few rounds later, it becomes confused by the world it created.

Its own decisions become legacy systems.

The next breakthrough in AI coding will require better models, but model intelligence cannot be the only answer. The development environment has to preserve memory, expose consequences, measure erosion, and make long-term coherence visible. The agent must be evaluated on the condition in which it leaves the repository.

The valuable coding agent will understand what already exists, what changed, which assumptions expired, what must remain true, and what its latest decision could quietly destroy.

The real test of AI code is the next prompt.

The real breakthrough will be an agent that leaves the system easier to change than it found it.

Connected reading

• The Coding Assistant Is Becoming the Development System
• The Prototype Has Become Too Cheap to Trust
• The Most Dangerous Interface Is the One That Says Everything Worked
• The End of the Software Request
• The Real Bottleneck Is Human Attention

AI authorship note

This essay began with Kris Haamer reading EvoCode-Bench and asking what its findings meant beyond one coding benchmark. Kris identified the Raw Drafts angle that AI is strong at beginnings and weak at inheriting responsibility for an evolving system, then asked for wider research and a published article. AI researched long-horizon coding benchmarks, software-evolution theory, developer-productivity studies, and agent-memory approaches; developed the argument; drafted and revised the essay; and handled the Raw Drafts publishing workflow. Kris directed the framing, paragraph structure, title, evidence threshold, and final publication decision.