First off, after that little break here on the blog: Why is everyone so obsessed with code review?
I’ve been having versions of this conversation with engineering teams for weeks. Agents produce more code, the review queue grows, and suddenly everyone is looking for a way to get through it. Fair enough. Someone has to take responsibility for what gets shipped.
But look at what we’re spending our attention on. Imagine a team facing 1,000 merge requests a week. Before agents, that same team could spend 7 days building the wrong thing and put it into a single merge request. The queue looked manageable. The product problem was already there.
AI has a way of amplifying the culture it arrives in, including the good parts. A team that experiments and learns gets more room to do both. An engineering-driven organization that has neglected product work can now produce the consequences of that neglect much faster. And spend an extraordinary amount of time reviewing them.
That matters when we talk about a software factory, because a product team gets to shape much more than the rate at which code comes out.
Put your judgment to work
If humans keep correcting the same mistakes in machine-produced work, we have something to build into the factory. Backpressure is fast, inexpensive feedback that lets the agent correct a problem during its working cycle, such as a compiler error or a failing focused test.
If the team has agreed to keep two modules separate, a quick architecture check can tell the agent when its change introduces a dependency between them. The same idea applies to accessibility. A form field missing an accessible label gives the agent something concrete to fix as soon as the check runs.
At a review gate, specialized agents can examine the change from different angles. A security reviewer tries to find vulnerabilities. For UX, the question might be whether someone can complete the intended task, while a UI reviewer checks the implementation against the design system. Give these reviewers their own instructions and a fresh context so they can challenge the result independently of the agent that built it. The change waits while their findings are resolved.
Humans have to make their judgment usable by machines so machines can review machine work and correct it before it reaches us.
Otherwise, we’ll still have 100 people standing at the factory line inspecting nonsense that could have been caught earlier.
Compiler errors and failing tests are the straightforward part. Give those signals back to the agent while it works and let it fix the problem. Product judgment takes more effort. Could someone actually use this to make the intended decision? Does the interaction fit the situation? Has the agent built a plausible generic answer to a very specific problem?
HumanLayer describes a practical version in Harness Engineering for Coding Agents. They found that running the full test suite flooded the agent’s context with passing results. Their approach became: “success is silent, and only failures produce verbose output.” Focused checks keep useful feedback available without making every correction expensive.
This is what I mean by Outcome Grading: turning product and domain judgment into a form we can evaluate repeatedly. Good and bad examples help. So do realistic scenarios and explicit reasons for rejecting a result. The evaluator needs its own checks, too. A confidently wrong judge is perfectly capable of keeping a factory busy.
Right now, we’re developing these evaluations with teams and bring rubrics, structured criteria for judging results, as a starting point. The team’s knowledge of its product has to shape them. A solely engineering-driven culture can encode poor product assumptions just as efficiently as good ones. Customer contact and human disagreement still matter.
Spend the time together in the factory
Spec-driven development keeps appearing as the proposed cure for all of this. Write a sufficiently good specification and the agent will know what to do.
A spec is one way to express intent. Useful! I have no objection to specs. I spend a lot of time arguing against the idea that we can finish understanding the work before we build it.
And if we have AI available, why should humans spend their most valuable time turning a rich discussion into a neat requirements document by hand?
Have the discussion! Challenge each other. By any means, we all spend so much time in front of our agents these days, use that little time with your fellow humans to collaborate. Let the person who knows the domain explain why an apparently obvious solution would be a disaster. With everyone on board, record it. Hand the conversation to your agent and use its ability to recognize patterns to help develop the specification.
Don’t compress away your ways of thinking before the machine gets to work with it. The hesitation, the example someone almost left out, the disagreement about what matters: these can carry more intent than the tidy sentence everyone finally agrees to put in a ticket. Keep that source material available as the spec evolves.
AI can help us structure what came out of human collaboration. We should spend the time it saves on more of that collaboration. It’s where people discover what they mean, challenge each other’s assumptions, and decide what they are willing to take responsibility for.
Building is part of specifying
As a software engineer, I care a lot about contact with the material. Which is software! Running the software, shape it in terms of architecture, trying an interaction and realizing that the thing I confidently described half an hour ago makes very little sense.
What a sad vision of our profession, to be allowed to write specs for the rest of your career while the interesting discoveries happen somewhere behind a curtain.
Software changes our understanding as we build it. A prototype can expose a missing concept, and during integration we may discover that two people have been using the same word for different things. Or someone tries an interaction and it feels awkward. Those discoveries belong in the specification, and sometimes they change the whole direction.
Drew Breunig arrived at the same problem after publishing a library as a spec and tests: “Implementing the code helps us improve our spec.” His Spec-Driven Development Triangle keeps specs, tests, and code evolving together. Even an experiment in distributing software without code led back to learning through implementation.
A software factory should make that contact easy for the entire product team. Product people can explore working variants, designers can try interactions in the system, and engineers can investigate the constraints that become visible along the way. People still bring different expertise. They can now do more of their thinking around the same running software.
That also gives us a much more interesting use for cheaper implementation: try several serious approaches before committing to one. Put them close enough to real data and system constraints that comparing them teaches us something. A working prototype still needs contact with users to tell us whether the problem deserves solving.
The amount we delegate can vary with the task. Sometimes I want an agent to work for hours against a well-understood goal. Sometimes I want to stay close and change direction every few minutes. I described this range of collaboration in The Elastic Loop. The factory has to accommodate that judgment.
You build the factory while it builds the software
The factory itself becomes another place where engineers need material contact. We have to see how it works, understand where it fails, and be able to change it.
I approach the software factory as an operating model: how the team organizes its work with agents, makes decisions, and takes responsibility for the result. It includes the technical environment and the way product knowledge reaches the work. It also includes who steps in when the available evidence is insufficient.
You build this through actual product work.
- When an agent repeatedly guesses a domain rule, improve the context it can find.
- When a review catches the same class of failure again, work out how to catch it during the next run.
- When an evaluation accepts something the team finds useless, investigate the evaluation.
This work needs time and ownership alongside feature work. A team whose entire capacity is allocated to the next batch of features will struggle to improve the system producing them.
And responsibility for the resulting software remains. For a consequential change, the team should be able to explain the architectural decision, the assumptions that have to hold, and what to do when it fails. Passing an evaluation gives us evidence about behavior. Maintaining enough understanding to operate and change the system is continuing work for the team. So, in a sense, you’ll have a separate backlog for your factory.
Let the work tell you what to improve
Once several people and teams are working with agents, individual observations become hard to connect.
- One person keeps repairing missing setup context.
- Another prompts fixes for weak results after the agent says it’s done.
- A third has found a useful checking pattern that everyone else could benefit from.
Loop Intelligence helps make those patterns visible across actual work. We look at the path from the task to its result, including the feedback and corrections along the way, and derive recommendations for improving the factory. A missing tool connection might matter more than another prompting workshop. If the same product mistake keeps coming back, we may need a new evaluation scenario, while a useful procedure could become a shared skill.
I developed this idea in When everyone has AI and the company still learns nothing. Here, the product team puts that learning into practice by making useful discoveries available to the next person and the next task.
For most teams, we start with recommendations people can inspect, curate, and act on. The team decides what to improve, and the next runs give us evidence about whether that change helped.
Right now I am careful with the promise of a self-improving factory. You need an operating model that practices evaluation-driven development from the beginning to make that credible. Otherwise, how would the factory know that its own changes were improvements? For many teams, getting useful recommendations and building the habit of acting on them is already substantial progress.
The evaluations also have to respect how product development works. Changing direction because a prototype taught us something is useful exploration. Repairing an error against an already agreed requirement is a different kind of work. If our metrics punish both as rework, we’ll teach the team to avoid discovery. We address distinctions like this in the rubrics we develop together.
Start with a piece of product work
Pick something the team actually needs to understand or deliver. Discuss it together, let agents help make the intent explicit, and build enough to test your assumptions. Make the technical and product feedback available during the work. Then inspect both what you built and how you got there.
Give the recurring problems a place in the factory’s own backlog. Fix something there, and check whether the next piece of work benefits.
I want a future in which product teams spend more time exploring possibilities together, engineers can reach into both the software and its factory, and what we learn survives the session in which we learned it. Agents give us much more room to work that way. We have to build the operating model that makes use of it.
There’s plenty to do beyond clearing the review queue!