What loop do I need to keep improving my software with LLMs?

Current one-line takeaway: A production-grade software self-improvement loop does not hold without human intent inside it.

Why this question

Plenty of teams now build software with LLMs. It runs. But it has more issues than you would expect, especially at the edges.

I am in the same spot. I run harness-maker and use it to drive code work across several projects. Then this thought hit me: what if an LLM harness analyzed my code line by line every night, reviewing and testing and improving it overnight. How much would quality actually go up.

The project I have in mind is my own embedded automated test farm. Python, 51,674 LOC, an imx8mm board pool, already running a nightly hardware-test cron at 23:00 KST. So the picture is a code-improvement cron running next to the hardware-test cron on the same codebase.

So the question narrows to one thing. To keep improving this software with LLMs until the issue count hits zero, what do I need.

Current hypothesis

Step back and the core is one thing. You have to close the self-improvement loop. Here is the final picture of that loop.

The loop turns on input, state, and output. Part of one cycle’s output flows back as the next cycle’s input. And what you carry into the next input is not shallow task state but code artifacts: merged PRs, rejected PRs, test logs.

In this loop, spec is set only once. A human defines it and locks it. That is the oracle. From then on the repeating cycle starts not at spec but at ultra-audit. The human reviews twice in a cycle: the improvements ultra-audit surfaces (what to fix), and the PR at the end. The human holds the intent, and between those two reviews the code runs.

What makes the middle automatable is that spec is the oracle. Because spec is the standard for “what is correct,” both the audit and the gates judge objectively, with no human in the middle. Without spec, a human has to step in at every stage, and that is the bottleneck.

The heart of the loop is the audit. Not just any audit, it has to be an ultra-audit. Claude Code’s dynamic workflow fans out a swarm of sub-agents and audits hard, module by module. (The two workflows I ran while writing this are the example: more than 30 agents digging in parallel in a single run.) Once ultra-audit pulls up issues and improvements module by module against spec, the human reviews that result and picks what to fix instead of reading code line by line. The human bottleneck shrinks the most right here.

The rest are the parts that fill each slot of this loop. And the design test for each part is where it lightens the human load.

The panel at the audit slot forces multi-vendor to block spec gaming. The default is 2 vendors (Claude as main, Codex as sub). Several copies of the same vendor are not a panel. harness-maker is open source, so forcing vendor diversity on everyone is hard, but the default setting starts at 2 vendors.

At the gate slot sit the 5 rubric items that decide an A+ PR.

1. Static analysis (clang-tidy and such) with 0 issues, and 0 lint issues
2. Coding style stays consistent (camel, etc.)
3. Meets the spec’s criteria
4. Tests pass (existing plus new)
5. No issues from the security / UI / UX / performance / concurrency / side-effect / domain reviewers

The category-5 reviewers map directly onto the specialized sub-agents defined in the global CLAUDE.md (concurrency-reviewer, performance-reviewer, side-effect-reviewer, and so on). On top of these 5 gates sits one more, the TC quality gate. The point is to force mutation and verify the tests are sensitive to code changes. It looks at spec coverage too. Even at 100% spec coverage, if the mutation score is low the PR is rejected. The clearer the rubric, the less a human has to look at directly during PR review.

The operating cadence runs by sprint, not by day. A PR review is 5 minutes at best per PR, and spending 2 hours a day on review every day is not sustainable. On top of that, running it daily loses its own effect. Trivial PRs get auto-merged on static conditions and pulled out of the human queue entirely (see open question 5). All of it serves one purpose: keeping the human’s two touchpoints light.

The loop itself is wired from skills I built myself. sw_improve runs the self-improvement cycle and ultra-audit handles its audit. Neither is a skill that lives inside harness-maker. They are my skills running on top of the review agents and gates that harness-maker lays down (its production preset). I dogfood this on the target project for real. The first picture was reviewing 30 PRs every morning, but it got updated twice during Round 1.

So if you are going to bolt on a self-improvement loop, the order is this. Set spec as the oracle once and lock it, start the repeat at ultra-audit, and let the human review only that audit’s output and the PR. Automate the space between with gates. It is not about removing the human, it is about herding the bottleneck into those two reviews so the human can focus on intent alone. In the end, making this loop efficient comes down to how much you shrink the human bottleneck. That is exactly where companies will dig in next.

Evidence so far

Evidence has gathered on two levels. One is industry evidence for the big claim that human intent closes the loop. The other is the rationale for each design decision.

Human intent closes the loop (industry already does it this way)

Issue to automatic PR is already production. Even so, confirming intent is the human’s job. Look at GitHub Copilot Autofix. When code scanning finds an issue, a PR goes up automatically within 30 seconds. Naturally it runs tests before it goes up. But the GitHub docs state that it remains your responsibility to ensure the suggested change maintains the intended behavior (GitHub Copilot Autofix docs). The loop is automatic, but confirming intent is the human’s.

Research points to the same split. AI catches the mechanical things (naming, tests, error handling, security patterns), and the human concentrates on intent (How Humans Review AI-Generated PRs). The numbers explain why. AI-generated code is over 95% syntactically correct, but only 55% is secure by default (Security-related AI-Generated PRs). It runs, but it drifts on intent and safety. I suspect this split holds for a while, however fast AI moves. And if AI takes over even up to here, maybe it is a time when the human has nothing left to do.

The rationale for each design decision

Spec gaming is the base rate, not a hypothetical risk. “The case where the LLM reads the spec loosely, decides it improved, and just files a PR” is an already-measured phenomenon. Both the METR 2025 report and Palisade Research report that reward hacking grows more frequent the more a model reasons (Reward Hacking Benchmark).

Three of the same vendor is not a panel. “Obviously B. The training data and the training method are completely different.” opus/sonnet/haiku are three copies of the same sheet. Mozilla AI’s Star Chamber pattern shows this as a production reference (Star Chamber). 3-model default, confidence-based classification (consensus / majority / individual).

The clearer the A+ rubric, the higher reviewer agreement. “You have to spell out clear criteria and examples for an A+.” On the MultiChallenge benchmark, providing a rubric jumped LLM-as-judge alignment from 37.3% to 93.95% (Snorkel rubric design). Grade labels alone are not enough; the checklist criteria are the primary driver.

The TC quality gate is the 6th on top of the 5. “Forcing mutation to test is the TC quality gate.” You have to look at spec coverage and mutation score together. Even at 100% coverage, a 15% mutation score gets the PR rejected (arxiv 2510.23350).

Sprint 1’s KPI is value, not volume. “Finding and fixing the important issues has far higher ROI than volume. Fixing a lot is not what matters. Early on you need a lot of test code to push spec and mutation score up.” One critical issue means more than 2,500 lines of tooling changes.

A self-improving agent’s ROI is tooling, not the algorithm. In the SWE-Bench Verified case that improved from 17% to 53%, the source of the gain was tooling: smart edit, an AST symbol locator, a context summarizer (A Self-Improving Coding Agent). Same for my sw_improve skill. The skill itself needs good internal tools to pay off.

Open questions

6 areas left unsolved or still hand-wavy from Round 1, plus one more from Round 2.

1. The two-sided reading of the panel consensus trend. Reading a rise in the consensus rate from 70% to 95% one way, as a “maturity signal,” is wrong. It is a good signal only on the condition that diversity holds. Otherwise it can be a spec-gaming signal, the panel collapsing into a single voice. I need a diversity metric measured separately from the consensus rate.

2. An explicit definition of sprint success criteria. I accepted the reframe that “week 1 is the infrastructure phase,” but it is still not clear what numbers actually judge sprint 1 as success or failure. N spec items? N mutation-resistant tests? What threshold?

3. How to write spec objectively. The pattern is decided up to drafting the spec via an LLM deep interview, having a human review, then locking it (“after that, no changes without human approval”). But the concrete pattern for writing spec so the LLM cannot “decide on something plausible” is still open.

4. The UI/UX track. It is hard to pin down with spec and tests, so I only decided to split it into its own track. How it is actually handled is undecided.

5. Trivial PR starvation. Seeing all 30 PRs a day is too much. Trivial PRs need a separate track: route them to a junior or second-tier reviewer, or flow them to auto-merge after a 2-reviewer consensus. Still undesigned.

6. The generalizability limit. “This process ends up being a system a general harness cannot do 100%.” The panel/gate/utility structure generalizes; spec/tests are project-specific. Where harness-maker provides defaults and where project tuning begins is an undecided boundary.

7. Human review is the bottleneck in the end. Trivial auto-merge or 2-person peer review can shrink the bottleneck (see 5). But there is no guarantee a human does their due diligence on spec and PR. It varies by individual, and this is the biggest variable that decides whether the loop succeeds.

What to check next

Round 2’s core goal is abstract to concrete. I want to design, in detail, the components I actually need and the workflow that actually runs.

Concrete deliverable candidates:

• The sw_improve skill’s architecture (the gates, the reviewer panel, utilities, the sprint-cycle flow)
• A checklist for the 5 A+ rubric criteria (with examples attached to each item)
• A measurement method for the TC quality gate (a spec coverage matrix plus which tool for mutation score)
• Concrete numbers for sprint success criteria (the infrastructure-phase metric threshold)
• A panel diversity metric (consensus rate and agreement diversity read together)
• A trivial-PR routing policy

And one new item this round. I am integrating this process into the embedded automated test farm I run right now. I plan to measure directly how much the software actually improves and how big the human bottleneck really is. What I am most curious about is satisfaction. Will the quality engineers (SQE) feel the reassurance of having software tested for them on a regular schedule, or will they feel replaced, like their work is disappearing. This part is hard to read from numbers alone, so I have to live through it.

Search keywords: mutmut python mutation testing, pytest spec coverage tools, sprint kpi infrastructure phase, LLM agent panel agreement diversity metric, code review trivial auto-merge consensus.