The fix loop gives up after three rounds, so an unattended run fails overnight while it is still making progress: keep going by default, with a limit in Settings #178

Closed
opened 2026-10-16 18:34:44 -04:00 by cmoriarty · 1 comment
Owner

From the user, 2026-10-16: run 66 ended failed with "the fix loop stopped (cap): the judge found 1 problem … after 2 fix rounds", and run 67 is heading the same way.

I think by default, the system should be able to keep trying indefinitely. It is the worst when you let something run all night, and wake up to a senseless failure like that. The user should have the ability in the settings page to limit how many tries a VERIFY & FIX section gets.

Today

  • osf.pipeline.fixloop.ROUNDS = 3: a check, then up to two fix rounds.
  • The rounds are fixed nodes in every built-in pipeline's graph: script.run-tests, .r2, .r3, with agent.judge[.rN] and agent.fix[.rN].
  • When round 3 is still red, state() returns stop/cap, and script.confirm-checks-passed fails the run.
  • Two other stops are not the bound: no_progress (a round failed exactly as the one before it) and ci (the project's CI could not finish).

Wanted

  • By default, the loop keeps going while rounds are making progress: there is no fixed number of rounds.
  • Settings gets a "fix rounds per run" limit: unlimited by default, or a number. A run records the value it was started with.
  • Runs and the spine stay readable as rounds go past 3.

Questions for the design

  • How rounds are made: the graph is static today, so rounds past 3 need materializing on demand, as fan-outs are.
  • What happens to no_progress when there is no limit. Stopping on "the same failure as last time" is still the right guard against an all-night loop that changes nothing. Or should it change approach, such as a fresh session, before it stops?
  • Whether a stop should open a gate for a person ("continue for N more rounds") rather than fail the run.
From the user, 2026-10-16: run 66 ended `failed` with "the fix loop stopped (cap): the judge found 1 problem … after 2 fix rounds", and run 67 is heading the same way. > I think by default, the system should be able to keep trying indefinitely. It is the worst when you let something run all night, and wake up to a senseless failure like that. The user should have the ability in the settings page to limit how many tries a VERIFY & FIX section gets. **Today** - `osf.pipeline.fixloop.ROUNDS = 3`: a check, then up to two fix rounds. - The rounds are fixed nodes in every built-in pipeline's graph: `script.run-tests`, `.r2`, `.r3`, with `agent.judge[.rN]` and `agent.fix[.rN]`. - When round 3 is still red, `state()` returns `stop`/`cap`, and `script.confirm-checks-passed` fails the run. - Two other stops are not the bound: `no_progress` (a round failed exactly as the one before it) and `ci` (the project's CI could not finish). **Wanted** - By default, the loop keeps going while rounds are making progress: there is no fixed number of rounds. - Settings gets a "fix rounds per run" limit: unlimited by default, or a number. A run records the value it was started with. - Runs and the spine stay readable as rounds go past 3. **Questions for the design** - How rounds are made: the graph is static today, so rounds past 3 need materializing on demand, as fan-outs are. - What happens to `no_progress` when there is no limit. Stopping on "the same failure as last time" is still the right guard against an all-night loop that changes nothing. Or should it change approach, such as a fresh session, before it stops? - Whether a stop should open a gate for a person ("continue for N more rounds") rather than fail the run.
Author
Owner

Shipped in 35c556b (change fix-loop-keeps-going, commit b3802d2), live since 00:34Z on 2026-10-17, and archived after the verification run (run_01M53R7621CJW6PCZG7S6R94HS, thorough on cmoriarty/braid-combined-test, vLLM, succeeded and merged at 03:49Z).

  • No fixed number of rounds by default. Settings → Runs → Fix rounds per run is empty (unlimited) unless set, from 1 to 20. A run keeps the value it was started with, and a run from before the setting keeps two.
  • Rounds past the graph's three are made on demand. They are agent.fix.r<n-1>, script.run-tests.r<n> and agent.judge.r<n>, copies of the last round renumbered. After a restart they are read back from the step rows, and the closing check waits while the loop makes one.
  • A stop opens a gate rather than failing. On the limit or on no progress, the closing check opens "Fix loop stopped, round n". Try 3 more rounds raises the limit, or waives that round's no-progress stop. Stop here fails the check as before. Auto-approve leaves the gate open, so an unattended run waits for you.
  • Simulated runs against the real scheduler:
    • five rounds to a merge;
    • a limit of 2, stopped at the gate and carried on with Try 3 more rounds;
    • no progress, stopped at the gate and answered Stop here.
  • On production: the verification run's loop read the run's limit (.osf/fix/limit.json = no limit) and passed in round 2. No extra round or gate was needed there yet.

Closing.

Shipped in 35c556b (change `fix-loop-keeps-going`, commit b3802d2), live since 00:34Z on 2026-10-17, and archived after the verification run (`run_01M53R7621CJW6PCZG7S6R94HS`, thorough on cmoriarty/braid-combined-test, vLLM, succeeded and merged at 03:49Z). - **No fixed number of rounds by default.** Settings → Runs → **Fix rounds per run** is empty (unlimited) unless set, from 1 to 20. A run keeps the value it was started with, and a run from before the setting keeps two. - **Rounds past the graph's three are made on demand.** They are `agent.fix.r<n-1>`, `script.run-tests.r<n>` and `agent.judge.r<n>`, copies of the last round renumbered. After a restart they are read back from the step rows, and the closing check waits while the loop makes one. - **A stop opens a gate rather than failing.** On the limit or on no progress, the closing check opens "Fix loop stopped, round n". **Try 3 more rounds** raises the limit, or waives that round's no-progress stop. **Stop here** fails the check as before. Auto-approve leaves the gate open, so an unattended run waits for you. - **Simulated runs against the real scheduler:** - five rounds to a merge; - a limit of 2, stopped at the gate and carried on with Try 3 more rounds; - no progress, stopped at the gate and answered Stop here. - **On production:** the verification run's loop read the run's limit (`.osf/fix/limit.json` = no limit) and passed in round 2. No extra round or gate was needed there yet. Closing.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#178
No description provided.