A fix round with nothing to fix has to invent an edit to pass since-verify:changed #165
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
A fix round must change the code (
agent.fixcompletes onsince-verify:changed, #139). But sometimes the right answer to a failing judge is to change nothing.Seen on the Sonnet 5.5 run on soundcheck#71 (thorough): the judge's first round failed the e2e criteria because no Postgres was available.
agent.fixrightly refused to stub the database, explained why inrepair.md, and changed nothing. Its attempt failedsince-verify:changed. The retry was then pushed into a pointless edit to pass the check. That cost an extra attempt and a commit nobody wanted, and it adds a reason to the record that isn't true.The judge's
unverifiableverdict now keeps most of these cases out of fix rounds (6fe3e8e). But anything that reaches a fix round with nothing to fix still has to invent an edit.What to decide:
repair.mdsays why. One option is ano-changesection the predicate accepts. Another is that a round with no change sends the run back to the judge or on to a person, rather than to a retry.Shipped as
fix-round-may-change-nothing..osf/fix/r<N>/repair.md, has a## No changesection that says why. A new completion check,fix-made, replacessince-verify:changedonagent.fixandagent.fix.r2.Verified. The fast and full lanes passed on the integrated batch (3,388 backend tests, 681 vitest, 249 browser specs with the slow and live ones, 29 of 29 simulated runs), and it is live on production as
57971c2(deploy run 138, self-checkbraid-selfcheck: ok). Archived ina6ae0dc. Two new simulated runs cover it:fix-changes-nothing(the judge then passes and the run merges) andfix-changes-nothing-same-failure(it stops atscript.confirm-checks-passed).Not measured yet: how often real runs use
## No change, and whether agents use it to dodge a real fix. That needs real-model runs.