A fix round with nothing to fix has to invent an edit to pass since-verify:changed #165

Closed
opened 2026-10-09 03:27:07 -04:00 by cmoriarty · 1 comment
Owner

A fix round must change the code (agent.fix completes on since-verify:changed, #139). But sometimes the right answer to a failing judge is to change nothing.

Seen on the Sonnet 5.5 run on soundcheck#71 (thorough): the judge's first round failed the e2e criteria because no Postgres was available. agent.fix rightly refused to stub the database, explained why in repair.md, and changed nothing. Its attempt failed since-verify:changed. The retry was then pushed into a pointless edit to pass the check. That cost an extra attempt and a commit nobody wanted, and it adds a reason to the record that isn't true.

The judge's unverifiable verdict now keeps most of these cases out of fix rounds (6fe3e8e). But anything that reaches a fix round with nothing to fix still has to invent an edit.

What to decide:

  • Let a fix round finish with no change when its repair.md says why. One option is a no-change section the predicate accepts. Another is that a round with no change sends the run back to the judge or on to a person, rather than to a retry.
  • Keep the guard #139 added against a fix agent that claims a fix it never made. Whatever the round says, the judge's next round re-runs the lanes.
A fix round must change the code (`agent.fix` completes on `since-verify:changed`, #139). But sometimes the right answer to a failing judge is to change nothing. **Seen on the Sonnet 5.5 run on soundcheck#71 (thorough):** the judge's first round failed the e2e criteria because no Postgres was available. `agent.fix` rightly refused to stub the database, explained why in `repair.md`, and changed nothing. Its attempt failed `since-verify:changed`. The retry was then pushed into a pointless edit to pass the check. That cost an extra attempt and a commit nobody wanted, and it adds a reason to the record that isn't true. The judge's `unverifiable` verdict now keeps most of these cases out of fix rounds (6fe3e8e). But anything that reaches a fix round with nothing to fix still has to invent an edit. **What to decide:** - Let a fix round finish with no change when its `repair.md` says why. One option is a `no-change` section the predicate accepts. Another is that a round with no change sends the run back to the judge or on to a person, rather than to a retry. - Keep the guard #139 added against a fix agent that claims a fix it never made. Whatever the round says, the judge's next round re-runs the lanes.
Author
Owner

Shipped as fix-round-may-change-nothing.

  • A fix round may finish without changing the code when its note, .osf/fix/r<N>/repair.md, has a ## No change section that says why. A new completion check, fix-made, replaces since-verify:changed on agent.fix and agent.fix.r2.
  • Such a round counts as made: the next round runs the lanes again on the same code, and the judge looks again, given that section. If it fails the same way, the loop stops for a person and the closing check quotes the reason. #139's guard against a fix that claims a change it never made is kept.
  • The fix brief says when to change nothing: never stub a missing service, never edit only so that something changed.

Verified. The fast and full lanes passed on the integrated batch (3,388 backend tests, 681 vitest, 249 browser specs with the slow and live ones, 29 of 29 simulated runs), and it is live on production as 57971c2 (deploy run 138, self-check braid-selfcheck: ok). Archived in a6ae0dc. Two new simulated runs cover it: fix-changes-nothing (the judge then passes and the run merges) and fix-changes-nothing-same-failure (it stops at script.confirm-checks-passed).

Not measured yet: how often real runs use ## No change, and whether agents use it to dodge a real fix. That needs real-model runs.

Shipped as `fix-round-may-change-nothing`. - A fix round may finish without changing the code when its note, `.osf/fix/r<N>/repair.md`, has a `## No change` section that says why. A new completion check, `fix-made`, replaces `since-verify:changed` on `agent.fix` and `agent.fix.r2`. - Such a round counts as made: the next round runs the lanes again on the same code, and the judge looks again, given that section. If it fails the same way, the loop stops for a person and the closing check quotes the reason. #139's guard against a fix that claims a change it never made is kept. - The fix brief says when to change nothing: never stub a missing service, never edit only so that something changed. **Verified.** The fast and full lanes passed on the integrated batch (3,388 backend tests, 681 vitest, 249 browser specs with the slow and live ones, 29 of 29 simulated runs), and it is live on production as 57971c2 (deploy run 138, self-check `braid-selfcheck: ok`). Archived in a6ae0dc. Two new simulated runs cover it: `fix-changes-nothing` (the judge then passes and the run merges) and `fix-changes-nothing-same-failure` (it stops at `script.confirm-checks-passed`). Not measured yet: how often real runs use `## No change`, and whether agents use it to dodge a real fix. That needs real-model runs.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#165
No description provided.