What loop would keep my software improving with LLMs?

If I squeeze what I have so far into one line, it is this. A production-grade software self-improvement loop does not hold without human intent inside it.

Why this question

Plenty of teams now build software with LLMs. What they build runs. But it carries more issues than you would expect, and the edge cases are where most of them sit.

I am in the same spot. I run harness-maker and use it to drive code work across several projects. One day a thought came to me. What if an LLM harness read my code line by line every night, reviewing and testing and improving it while I slept? How much would the quality go up?

The project I had in mind is my own embedded automated test farm. Python, 51,674 LOC, an imx8mm board pool, and a nightly hardware-test cron that already runs at 23:00 KST. I pictured a code-improvement cron running beside the hardware-test cron, on the same codebase.

After that the question narrowed to one thing. If I want to keep improving this software with LLMs until the issue count reaches zero, what do I need?

Current hypothesis

When I step back, the core is one thing. The self-improvement loop has to close. This is the final picture of the loop as I see it now.

The loop turns on input, state, and output. Part of one cycle’s output flows back in as the next cycle’s input. What gets carried forward should be code artifacts. Shallow task state is not enough; merged PRs, rejected PRs, and test logs are what move on.

In this loop, spec is set only once. A human defines it and a human locks it. That is the oracle. After that, each repeating cycle starts at ultra-audit and does not go back to spec. The human reviews twice in a cycle: once over the improvements ultra-audit surfaces, deciding what to fix, and once over the PR at the end. The human holds the intent, and between those two reviews the code does the running.

The middle can be automated because spec is the oracle. Since spec is the standard for “what is correct,” both the audit and the gates can judge objectively, with no human in the middle. Without a spec, a human has to step in at every stage, and that step is where the bottleneck forms.

The heart of the loop is the audit. An ordinary audit falls short, I think; it has to be an ultra-audit. Claude Code’s dynamic workflow fans out a swarm of sub-agents and audits hard, module by module. (The two workflows I ran while writing this are the example: more than 30 agents digging in parallel in a single run.) Once ultra-audit pulls up issues and improvements module by module against spec, the human can read that result and pick what to fix, instead of reading code line by line. This is where the human bottleneck shrinks the most.

The rest are the parts that fill each slot of the loop. For each part, the design test I used was where it takes weight off the human.

The panel at the audit slot forces multi-vendor to block spec gaming. The default is 2 vendors (Claude as main, Codex as sub). Several copies of the same vendor are hard to call a panel. harness-maker is open source, so forcing vendor diversity on everyone is not easy. Even so, the default setting starts at 2 vendors.

At the gate slot sit the 5 rubric items that decide an A+ PR.

1. Static analysis (clang-tidy and such) with 0 issues, and 0 lint issues
2. Coding style stays consistent (camel, etc.)
3. Meets the spec’s criteria
4. Tests pass (existing plus new)
5. No issues from the security / UI / UX / performance / concurrency / side-effect / domain reviewers

The category-5 reviewers map directly onto the specialized sub-agents defined in the global CLAUDE.md (concurrency-reviewer, performance-reviewer, side-effect-reviewer, and so on). On top of these 5 gates I put one more layer, the TC quality gate. The idea is to force mutation and check whether the tests react to code changes. It looks at spec coverage too. Even at 100% spec coverage, if the mutation score is low the PR is rejected. The clearer the rubric, the less a human has to look at directly during PR review.

The operating cadence is set by sprint. It does not run every day. A PR review takes 5 minutes at best per PR, and spending 2 hours a day on review, every day, is not a life that lasts. Running it daily also wears down its own effect. Trivial PRs get auto-merged on static conditions and leave the human queue entirely. I have not designed that part yet, so it comes back under open questions. All of it points at one thing: keeping the human’s two touchpoints light.

The loop itself is wired from skills I built myself. sw_improve runs the self-improvement cycle and ultra-audit handles its audit. Neither is a skill that ships inside harness-maker. They are my skills, running on top of the review agents and gates that harness-maker lays down (its production preset). I am dogfooding this on the target project for real. The first picture was reviewing 30 PRs every morning, and it changed twice during Round 1.

So if I were to bolt on a self-improvement loop, I think the order goes like this. Set spec as the oracle once and lock it, start each repeat at ultra-audit, and let the human review only that audit’s output and the PR. Automate the space between with gates. The aim is not to remove the human. It is to gather the bottleneck into those two reviews so the human can give attention to intent alone. Looking back, making this loop efficient came down to how far you can shrink the human bottleneck. I think that is where companies will dig in next.

Evidence so far

Evidence for this hypothesis has gathered on two levels. One is industry evidence for the larger claim that human intent closes the loop. The other is the reasoning behind each design decision.

Human intent closes the loop (industry already does it this way)

Going from an issue to an automatic PR already runs in production. Even so, confirming intent is still left to a human. GitHub Copilot Autofix is one case. When code scanning finds an issue, a PR goes up automatically within 30 seconds, and it runs tests before it goes up. But the GitHub docs state that it remains your responsibility to ensure the suggested change maintains the intended behavior (GitHub Copilot Autofix docs). The loop runs on its own, but confirming intent still belongs to the human.

Research points to the same split. AI catches the mechanical things (naming, tests, error handling, security patterns), and the human concentrates on intent (How Humans Review AI-Generated PRs). The numbers hint at why. AI-generated code is over 95% syntactically correct, yet only 55% is secure by default (Security-related AI-Generated PRs). It runs, and still drifts on intent and safety. However fast AI moves, I expect this split to hold for a while. And if AI does take over even this part, will there be anything left for the human to do?

The rationale for each design decision

The first thing I worried about was spec gaming. It is not a hypothetical risk that might show up someday. It is the base rate. “The case where the LLM reads the spec loosely, decides it improved, and just files a PR” is an already-measured phenomenon. Both the METR 2025 report and Palisade Research report that reward hacking happens more often the more a model reasons (Reward Hacking Benchmark).

Forcing multi-vendor on the panel came from the same worry. Three of the same vendor is not a panel. What I wrote down at this point was: “Obviously B. The training data and the training method are completely different.” opus/sonnet/haiku are, in the end, three copies of the same sheet. Mozilla AI’s Star Chamber pattern shows this as a case used in production (Star Chamber). There it is 3 models by default, sorted by confidence into consensus / majority / individual.

The clearer the A+ rubric at the gate, the higher reviewer agreement goes. At first this was only a hunch, “You have to spell out clear criteria and examples for an A+,” and then it turned out to be something you can measure. On the MultiChallenge benchmark, giving a rubric moved LLM-as-judge alignment from 37.3% to 93.95% (Snorkel rubric design). Grade labels alone were not enough; what moved the result was mostly the checklist criteria.

There is a reason the TC quality gate sits apart, as the 6th on top of the 5. In my own words, “Forcing mutation to test is the TC quality gate.” Spec coverage and mutation score have to be read together. Even at 100% coverage, a 15% mutation score gets the PR rejected (arxiv 2510.23350).

I set Sprint 1’s KPI as value, not volume, because of this: “Finding and fixing the important issues has far higher ROI than volume. Fixing a lot is not what matters. Early on you need a lot of test code to push spec and mutation score up.” One critical issue means more than 2,500 lines of tooling changes.

I also lean on the finding that a self-improving agent’s ROI is in tooling, not the algorithm. In the SWE-Bench Verified case that went from 17% to 53%, the gain came from work on tools: smart edit, an AST symbol locator, a context summarizer (A Self-Improving Coding Agent). My sw_improve skill is no different. The skill needs good tools inside it before it pays off.

Open questions

Round 1 left 6 areas I could not solve or still cannot get a grip on, and Round 2 added one more.

The one that confuses me most is how to read the panel consensus trend. When the consensus rate climbs from 70% to 95%, it is tempting to read that as a “maturity signal.” Reading it only that way is a mistake. It is a good signal only while diversity holds. Otherwise it can be a spec-gaming signal, the panel collapsing into a single voice. So I need a diversity metric measured apart from the consensus rate.

I have not given sprint success criteria an explicit definition either. I accepted the reframe that “week 1 is the infrastructure phase.” But what numbers will actually judge sprint 1 as success or failure is still unclear to me. Is it N spec items, or N mutation-resistant tests? And what threshold?

How to write spec objectively is still open too. I have decided the pattern up to this point: draft the spec through an LLM deep interview, have a human review it, then lock it (“after that, no changes without human approval”). How to write a spec so the LLM cannot “decide on something plausible” is the part I have not worked out.

UI/UX is hard to pin down with spec and tests, so all I decided was to split it into its own track. How to handle it in practice is still undecided.

Trivial PR starvation is unsolved as well. Seeing all 30 PRs a day is too much. Trivial PRs need a separate track, where they go to a junior or second-tier reviewer or flow to auto-merge after a 2-reviewer consensus, and I have not designed it yet.

Generalizing has a limit. What I wrote about it was, “This process ends up being a system a general harness cannot do 100%.” The panel/gate/utility structure generalizes; spec/tests differ from project to project. Where harness-maker provides defaults and where project tuning begins is a line I have not drawn yet.

The last question is that human review is the bottleneck in the end. The trivial auto-merge or 2-person peer review I mentioned above can shrink the bottleneck. But nothing guarantees that a human will do their due diligence on spec and PR. People differ, and I think this is the biggest variable in whether the loop succeeds.

What to check next

What I most want from Round 2 is to move from the abstract to the concrete. I want to design, in detail, the components I actually need and the workflow that actually runs.

The deliverables I have in mind go like this. First the sw_improve skill’s architecture, one flow that shows how the gates, the reviewer panel, utilities and the sprint cycle move. Then a checklist for the 5 A+ rubric criteria, with examples attached to each item. The TC quality gate needs a measurement method: a spec coverage matrix, plus a decision on which tool measures mutation score. Sprint success criteria need concrete numbers, meaning the infrastructure-phase metric threshold. The panel diversity metric should read consensus rate and agreement diversity together. And a trivial-PR routing policy is still left.

And one new item came up this round. I am integrating this process into the embedded automated test farm I run right now. I plan to measure for myself how much the software actually improves and how big the human bottleneck really is. What I am most curious about is satisfaction. Will the quality engineers (SQE) feel the reassurance of having software tested for them on a regular schedule, or will they feel replaced, as if their work were disappearing? Numbers alone will not show much of this, so I think I have to live through it to know.

Search keywords: mutmut python mutation testing, pytest spec coverage tools, sprint kpi infrastructure phase, LLM agent panel agreement diversity metric, code review trivial auto-merge consensus.