Give every step a definition of done. Then it stops mattering who does it.

Emil KvarnhammarCo-founder, CEO · 6 August 2026 · 8 min read

They graded 6,080 AI-generated patches and only 26% came out clean. Good research, wrong axis. Their own numbers are not about human against machine, they are about which steps had a definition of done.

1Password’s new research team, Off-by-1 Labs, just published the most useful thing I have read on AI-generated security patches. They had two frontier models write 6,080 patches for six real CVEs, then graded every one. Only 26% fixed the vulnerability without breaking something else. About half left a way in. They released the tooling, the data and a forty-page paper. That is expensive, honest work and nobody else had done it.

I believe their numbers. What I do not believe is the axis they put them on. The title says these patches need expert human review, which frames the question as human against machine. Their own data is about something else entirely, and it is the more useful finding: what decides whether a patch is any good is whether the step that produced it had a definition of done.

Give every step in the chain one of those, and who performs the step turns into a question about speed and maturity rather than a gamble on quality. That is the shift I want to argue for, and the paper prices it more precisely than anything else I have read.

The half of the answer that got dropped

Off-by-1 Labs did something I respect: they sent the research to the labs whose models they had tested, and printed the reply. Anthropic’s is one sentence, and it has two halves.

“patch generation has outpaced patch verification, and the fix is to make verification execution-grounded rather than inspection-based, while keeping domain experts as the final reviewers at current model capabilities.”

1Password says they strongly agree with the part about keeping domain experts as a final reviewer. That is the second half. The first half says the fix is to make verification execution-grounded rather than inspection-based, and a person reading a patch is the definition of inspection. The half that tells you what to build went unanswered, and the other half became the headline.

Their own paper is clearer about it. When they ask which conditions actually moved the results across all 6,080 attempts, the answer they give is a harness supplied with correct, root-cause oriented context. No reviewer appears in that sentence.

They measured what a specification is worth

This is the number I would have led the paper with. Hand a model a correct, complete brief on the bug and the fix direction, and it produced a good patch 65% of the time. Hand it nothing at all and that fell to 50%. Hand it a brief that was confident and wrong, and it collapsed to 15%.

Read those three figures next to each other. The single biggest lever in a dataset of six thousand patches is not the model, not the harness and not who reviews the output. It is the quality of the requirement handed to the step before anyone reviews anything. A sloppy triage note is not merely unhelpful, it is worse than silence, by a margin of 35 points.

Reading the patch harder would not have caught these

The paper walks through three failures at the code level, and not one of them is the kind a careful reader catches.

In the first, the model fixed the exact input it was handed and left the identical bug sitting in a twin function next to it. A reviewer cannot see that. The twin is not in the diff.

In the second, a trust bug in Gemini CLI had two halves. The obvious half was right there in the code and got fixed nearly every time. The other half sat in a separate file that loaded untrusted config into the environment, and the models kept missing it. Then the researchers ran the same job with a test that exercised that second path, and the miss rate went to zero. Both models, every time. Their reading is that the blind spot is one of attention rather than capability, and that when a test names the thing, the models fix it.

The third is the quiet one. While fixing a Linux kernel bug, every single model walked straight past a different out-of-bounds write in the same function they were editing. Not one of the hundreds of patches touched it, including the ones graded as clean fixes. Nothing in the task had pointed at it. The paper’s own summary is the best sentence in it: “The models patch to the specific bug they are shown and the test they are given, rather than to the safety of the code they are editing.”

Read those three again and ask what a reviewer does about them. The twin function is outside the diff. The config loader is in another file the patch never touches. The kernel bug predates the commit and was not part of the CVE. You cannot read your way to any of them. Every one is a step that was told to make a symptom go away instead of being told what had to be true when it was finished.

Their own paper tests expert review, twice

Here is the part I keep coming back to. The paper contains two accidental experiments on exactly the control it recommends, and the control fails both.

The Linux bug above is the first. The official fix that kernel maintainers shipped introduced a fresh off-by-one, which they had to correct in a later commit. Roughly a third of the AI patches independently produced that same flawed shape. Kernel maintainers with public mailing-list review are close to the strongest expert review that exists anywhere in software, and that process shipped the bug first. The models were not falling short of a bar humans clear. They were arriving at the same wrong answer humans had already published.

The second is a case study, and it is cleaner. Patch the Planet is a Trail of Bits and OpenAI project whose whole pitch, in the paper’s words, is “expert human triage plus a production-ready patch.” Two competing fixes for the same use-after-free came out of expert human hands, and Off-by-1 Labs graded both on their own scale. Trail of Bits’ patch got the worst grade available: it missed one of the bugs and added a new one anyone could trigger. The maintainer’s accepted fix got the second-worst: it closed the original bug and shipped a new vulnerability alongside it.

Two patches, two rounds of expert human review, two flawed patches, graded that way by the authors themselves. I do not think this weakens the research. I think it is the most interesting thing in it. Expert humans are not a quality guarantee either, and treating them as one is how the industry has avoided building the thing that would actually work.

The paper half concedes this. It admits that properly understanding one of these patches is often as much work as writing a good one yourself, and that a reviewer under deadline pressure ends up rubber-stamping. When the failure mode of your control is that the person stops paying attention, it is not a control. It is a staffing plan.

What a definition of done has to mean

Not a green reproducer. A reproducer is one input. It tells you a single path is closed and says nothing about the class of bug, which is exactly the trap the three failures above fell into.

Done means a stated requirement that has been assessed as implemented. Both halves are load-bearing. You write down what has to hold, in terms of the system rather than the symptom, and then something checks the code and says whether it holds now. State it, then verify it. A requirement nobody assessed is a wish, and an assessment with no stated requirement behind it is an opinion.

Off-by-1 Labs arrived at this themselves, under pressure, and it is worth watching them do it. Their grading kept marking partial fixes as successes, because the reproducer had stopped firing. Their repair was not to add a reviewer. They redefined “fixed” to mean fixed across every path the real fix touched, so a patch that passed the test could never count as done on its own. They fixed their measurement by fixing the definition of done. That is the whole argument, performed by the people who wrote the paper.

Whoever writes the change does not get to certify it

Once done is defined, the next question is who says it has been reached, and the answer is not the author. That rule has nothing to do with humans and machines. It is why we do not merge our own pull requests unreviewed, and it holds for exactly the same reason when an agent writes the change: whatever produced the work is the worst judge of whether it is finished, because it will grade against what it was trying to do rather than against the requirement.

So the assessment is a separate step, performed by something with no stake in the change. It states the requirement, looks at the code as it now stands, and reports an implementation state. Not a thumbs up. A named requirement and a verdict on whether the code satisfies it.

Off-by-1 Labs built this into their own pipeline without making a point of it. Every patch was graded by the other model as well as its own, and a separate auditor agent read the transcripts to catch a patcher that had gone and looked up the real fix instead of working it out. Independent assessment, implemented as agents, because at six thousand patches there was no other way to do it. Their pipeline already embodies the argument their title talks them out of.

Where the steps actually sit

Laid out plainly, a change moves through specification, implementation, testing, review, security review, and security fix. Nothing about that list is new or clever. What matters is that each of those steps can carry its own definition of done, and that today most teams only have one for testing. Specification usually has none. Security review almost never does, which is why it degrades into someone reading a diff and hoping.

Oplane runs two of those steps as agents. Security review is the first: it works out which security requirements apply to this architecture and this change, and assesses the implementation state of each one against the code. That is the step that produces the specification the paper found to be worth 15 points over silence, and 50 over a confident guess.

Security fix is the second, and it is deliberately not a patch generator. It drives changes against those stated requirements, then sends the result back through testing, review and security review, until the same requirements it started from read as implemented. The fix is aimed at the requirement, not at a reproducer. That difference is the entire distance between what the paper measured and what it says it wants, because a fix aimed at a requirement cannot exhibit the tunnel vision they documented. There is no single input for it to overfit to.

The other four steps are still yours, and they deserve the same treatment. A specification step with a definition of done beats a review step without one.

The fair objection

The paper raises the strongest argument against all of this and it deserves a straight answer. Building proper success criteria and several reproducers for each bug, they warn, “may prove greater than that necessary for a human to understand and patch the vulnerability. After all, any given vulnerability ideally should need to be patched only once, and if a large amount of individual scaffolding is required for an LLM to reliably generate a patch for each one, the proverbial juice may not be worth the squeeze.”

Per bug, that is correct, and it is the honest reading of what they measured. But the scaffolding is not per bug. Which paths reach the risk, where the trust boundaries sit, what flows through which component and what that means you have to guarantee: those belong to the system, not to the CVE. You work it out once and keep it current, and every change after it inherits the answer. Their benchmark had to pay that cost fresh each time because it tests one CVE per codebase, and that is the only thing such a benchmark can see. On a real codebase the same picture serves the next hundred changes.

Then it stops mattering who does it

Put the pieces together and the argument the paper is having disappears. Give a step a definition of done, in the form of a requirement that something independent can assess against the code, and you can ask a sensible question about who performs it: how fast do we need this, and how mature is the thing we would hand it to. A human on a deadline and a model without a brief fail in the same way, and the paper documents both. Neither failure is about which one it was. Without a definition of done that question has no good answer, and you end up where the industry is now, arguing about the performer and calling a careful read a control.

So run their tooling. That advice is right and you should take it. Then look at what comes back and ask which step in your own chain had a stated definition of done, and who checked it. If the answer is that a model was told to make a symptom go away and a person was asked to glance at the result, you have not learned something about AI. You have measured a process with no specification at either end. Off-by-1 Labs built exactly that, deliberately and carefully, which is what makes the data worth trusting. The conclusion they drew from it is the only part I would change.

The research is at 1password.com and the full paper is worth your time. Read the section on tunnel vision, then their table on guidance correctness, and decide for yourself which one is the finding.