Self-Improving Coding Agents: How Software Factories Learn From Their Own Mistakes

Why software factories need memory

A software factory is what engineering becomes when agents write most of the code. Humans set intent and review. Agents plan, explore, edit and open PRs, hundreds of sessions a day. Throughput grows with the number of agents, but learning doesn't.

A human engineer who gets a review comment remembers it and doesn't repeat the mistake. An agent session starts cold every time. So the factory ships the same defect over and over, and the people reviewing its output end up as its memory, which is the most expensive kind there is.

A self-improving factory closes that gap with a feedback loop, the way manufacturing did with statistical process control and machine learning did by training on its own errors. The loop has four stages:

1. Pinpoint. Take a failure signal (a review comment, a failed test, an incident) and trace it to the exact step in the agent session where it entered.
2. Aggregate. Cluster those root causes across many sessions until isolated mistakes show up as recurring patterns.
3. Inject. Turn each pattern into a learning and hand it to future sessions at the right moment, through the right mechanism.
4. Prove. Measure against a control whether the learning prevents the error, and retire it if it doesn't.

Each stage is easy to state and hard to get right. Sections 1 to 4 cover the principles: why agents repeat mistakes, what makes a good error signal, how to read an agent trace, and why most failures turn out to be context failures. Sections 5 to 9 show a working implementation, the Entelligence Learning Loop, with numbers from an eight-week rollout. Sections 10 and 11 look at where it goes next.

Models get better every quarter, but none of them will ship knowing your codebase's contracts, conventions and incidents. The only place to learn those is your own factory's mistakes, and the lessons have to be stored where the next session can use them.

1. The problem: agents repeat themselves

Anyone who reviews agent-written PRs at volume has seen this. Cluster a few weeks of substantive review comments and the same failures come back week after week, across engineers, repos and agents.

An agent passes user_id where the attribution contract expects cohort_id. It hand-rolls pagination offsets in a repo that already has a paginate() helper. It calls db.session.execute_raw, a SQLAlchemy-style method that doesn't exist in the in-house ORM wrapper. A reviewer catches it, the engineer fixes it, the PR merges, and three days later a different session makes the identical mistake.

The cause is structural. Every agent session starts cold, and the agent has no equivalent of the memory a human uses to avoid repeating a mistake. Static rules files (CLAUDE.md, AGENTS.md, .cursorrules) are the industry's current answer, but they are hand-written, rarely updated and never measured, and they grow until agents stop reading them carefully.

In the rollout described later in this paper, 94% of substantive review comments across six repos traced back to a specific agent edit. Those were agent mistakes, not human ones slipping through, which means they can be learned.

2. Code review is the error sensor

A learning loop needs a reliable signal of failure, and for coding agents that signal already exists in code review. A review comment is a labeled example written by an expert, pinned to a file and line, on a PR that an agent session produced.

Ordinary code review doesn't guarantee three properties the loop needs:

• Classified. Substantive issues (a broken contract, a wrong API, a violated convention, security, logic) are separated from nits. Nits stay out of the loop by default, because training agents on style preferences produces noise.
• Linked to the edit that caused it. A comment on views.py L140 has to join to the exact edit event in the agent trace (say, edit #612) that wrote that line. Without the link you know what's wrong but not where the agent went wrong.
• Inclusive of human reviewers. Comments from engineers flow into the same pipeline as automated ones, so the loop learns the team's real standards.

Precision matters more than recall. A loop fed false positives teaches agents to avoid problems that don't exist and spends context doing it. The factory's memory can only be as good as its failure signal.

3. Trace forensics: down to the step and turn

A review comment tells you what went wrong. The agent trace tells you how and why. The first step is to normalize sessions from every harness in use (Claude Code, Codex, Cursor and others) into a single event model.

Every event in a session is typed and indexed:

The example session in this paper is prod-db agent PR review investigation plan, run on Claude Code for 4h 45m. It had 921 events, 8 subagents and 11 errors, and cost $13.11. It produced PR #4182, which drew three substantive review comments and one nit.

For each comment we compute a causal path, the minimal chain of events leading from what the agent knew to the flagged edit. For the attribution bug the path is four events long:

1. #388, Sub 4 (Explore): reads model_attribution.py, which defines the contract.
2. #402, Sub 4: returns a handoff summary to the main agent that omits cohort_id.
3. #598, Main: reads views.py.
4. #612, Main: edits views.py L140, passing user_id. Flagged in review.

The subagent had the right answer and the main agent never received it. The diff can't show that. Only the trace can.

We also compute ghost nodes, which are files the fix required but the agent never opened. Here, attribution_serializer.py defines the contract and was never read. Ghosts are often the most actionable output of the analysis because they name exactly what context was missing.

4. Root cause: most failures are context failures

The model usually isn't the problem. When we classify causal paths, the large majority of substantive errors trace to the agent not having, not finding or not keeping the right context at the moment it edited. Only a minority are genuine reasoning failures.

We sort every causal path into one of six root causes:

Three of these (missing context, ignored convention, hallucinated API) are fixed by supplying a fact. Lost handoffs are fixed by changing how the harness moves context between agents. Plan drift and tool failures need good timing: the right reminder at the right step.

This split drives the design of the memory layer. Adding more to CLAUDE.md only helps the first category, and not by much. Each root cause needs a different delivery mechanism, so Section 7 treats where, when and how context is added as an experimental variable.

From principles to practice: the Entelligence Learning Loop

The four stages are a design pattern any team can build toward. At Entelligence we built the full loop as a product, and the rest of this paper covers how it works and what it measured.

We started with the error sensor. Entelligence's code reviewer ranked #1 of 8 tools on real production bugs (47.2% F1) in our Code Review Benchmark 2026, which gave us a high-precision failure signal. On top of that we added trace ingestion, root-cause analysis, a shared memory we call the World Model, and the Entelligence CLI, which builds each agent session with the right learnings installed. As far as we know, no one else closes this loop end to end.

The Learning Loop has one view per stage: Pinpoint (one session), Aggregate (all sessions), Inject (the audit) and Prove (over time). Sections 5 to 9 walk through them. The numbers come from an eight-week rollout, Aug 4 to Sep 29, across 1,284 Claude Code sessions in six repos. Sessions built with learnings drew 53% fewer substantive review comments than a holdout, at $2.41 of injected context in total.

5. Where in the session each error was made, and why

For every flagged line, the engineer sees the exact moment in the session where the error entered, the path that led there, and what the agent should have read. The session replay puts each review comment on the timeline at the edit that wrote it, then draws the causal path back through subagents and reads.

The diagram below pulls out that one causal path.

The subagent did its job and the handoff didn't, which changes the fix. This isn't a rule for CLAUDE.md. It's a learning about how Explore subagents must report back (L-9), plus a path-specific fact about the attribution contract (L-14).

The replay also shows the session's full context tree: every file, MCP call, subagent and insight source the agent touched, sized by how often, with ghost nodes dashed. Across the whole PR #4182 session, the tree covered 328 exploration events, 113 tool calls, 11 MCP calls and 11 errors.

6. From one session to recurring patterns

One causal path is an anecdote. The same path in five sessions is a pattern worth teaching. We cluster root-caused comments across every session in the org by root cause, trigger path and the contract or convention involved, and we propose a learning when a cluster crosses an evidence threshold.

Each learning carries its evidence trail: every PR comment it came from, with a link and a replay of the session that produced it. A human reviews that trail at the approval gate, and it keeps each learning auditable later.

Across 1,284 sessions and 412 substantive comments over eight weeks, the loop produced six learnings:

L-9 spans three repos because it has nothing to do with any one codebase. It's about the harness itself. Some of the most useful learnings fix how agents work rather than what they know.

7. Closing the loop: injection through the Entelligence CLI

A learning is worthless if it doesn't reach the next session when it matters. The Entelligence CLI builds each agent session. Before the harness starts, it works out which learnings apply to this repo, this task and these paths, and installs them through one of four mechanisms.

Every learning has a trigger, which is a path glob plus a diff condition (for example backend/**/views.py · diff touches attribution). Triggers keep context lean. Across 1,027 sessions built with learnings, the CLI made 2,356 insertions at an average of just +0.9K tokens per session.

What we experiment on

We treat injection as an experiment with four variables and run it continuously:

• What to add: the rule itself, the evidence behind it, a pointer to the file that defines it, or all three. Pointers to ghost files often beat prose rules.
• Where to add it: rules file, skill, MCP or hook. The same learning can work as a hook and fail as a rules line.
• When it fires: at session build, at task-type match, at first read of a trigger path, or just before an edit. Later is usually better, because the agent acts on what's in its recent context.
• How much: a per-session token budget, ranking when many learnings match, and suppression of learnings that have stopped paying off.

The audit log separates delivery from use. A learning can be delivered and still ignored. In PR #4214, L-14 was retrieved, then ignored, and the error recurred. In PR #4182, the L-11 hook never fired because the agent wrote execute_raw through an Edit path that bypassed it. Both are logged as failures of the mechanism, and both feed the next round of experiments.

Governance

Learnings move through an explicit lifecycle: Proposed (clustered from review comments), Approved (a human gate, and nothing enters a session without it), Active (added by the CLI) and Retired (removed when it stops paying off). Engineers stay in control of what their agents are told.

8. The World Model: where the memory lives

The World Model is a graph of what the loop has learned about how agents behave in your org. Sessions, PRs, review comments and people are nodes, and the causal links between them are edges. The view below draws the top 238 patterns, people and themes from a graph of 162,130 nodes.

In this view the graph holds:

The view has three lenses: Recurring Issues (what keeps breaking), Coding Agent (how agents behave) and Merged (both, linked). 283 links connect the loaded nodes, and hovering an orb traces them. Ask Ellie answers questions against the same graph in natural language.

Every learning resolves back through a pattern to specific issues, to the edits that caused them, and to the sessions and people involved. Nothing an agent is told is unsourced.

Agents read the graph live through the world_model.search MCP tool, which the CLI registers at build. That's how rich, path-specific learnings like L-14 get delivered. Instead of bloating every session's rules file, the agent retrieves the relevant slice of memory only when it touches backend/**/views.py.

9. Proving it: the same errors, measured over time

Sessions built with learnings drew 53% fewer substantive review comments than the holdout (0.62 vs 1.31 per session; 95% CI −41% to −63%). We make every improvement claim against a control, never against last month.

Since Aug 4, 20% of sessions have been a standing holdout. For those sessions, the CLI still matches triggers and logs which learnings would have been added, but adds nothing. That gives a like-for-like control: same repos, same engineers, same agents, same weeks. The holdout itself improved about 10% over the period, which is the team-effect baseline (people get better, models get better), so we report the gap above that line and not the raw drop.

The two lines keep separating as more learnings go live. The holdout barely moves, so the improvement comes from the memory and not from the model or the calendar.

Because every comment is root-caused, we can also see which failures the memory fixes.

Context injection works best on failures caused by missing facts and lost handoffs. It does much less for plan drift, which is a long-horizon execution problem. That tells us where to push next: mid-session reminders timed to plan checkpoints, rather than more context at build.

Each learning also gets its own scorecard, so a weak one can't hide behind a strong one:

L-6 shows the loop working as designed. It was a style rule learned from nits, it recurred 18 of 44 times, it prevented almost nothing, and it was retired. Memory that doesn't pay for itself gets removed.

On cost, injection averages +0.9K tokens and $0.0023 per session, which came to $2.41 over eight weeks. L-14 alone costs $0.002 per session and prevents about one review round-trip every eight sessions. A fraction of a cent of context replaces part of an engineer's review cycle.

10. Beyond code review: every signal your org already has

Code review is the first sensor, not the only one. The same loop works for any signal that an agent-written change caused a problem, and on the upgraded Entelligence plan these sources plug into the World Model today:

Every source goes through the same steps. It becomes an event, linked to code, linked to the session that wrote it, clustered into a learning, gated by a human, delivered by the CLI and measured by holdout. The loop doesn't care where the signal comes from.

Over the long term, the goal is a production-aware memory for every coding agent in the org, so the incident your on-call engineer fixed at 3 a.m. is one no agent in your company will cause again.

11. Open source, and where this goes

The World Model visualization is open source at Entelligence-AI/agentplay. Point it at your own agent sessions to see the patterns, people and themes in your graph and how they connect.

We're open-sourcing the view because we want the whole industry looking at agent traces this way. Agents will keep getting smarter, but they won't pick up the memory of your codebase on their own: the contracts, conventions, incidents and decisions that make your code yours. We're building the layer that captures that memory, delivers it and proves it works.