Browser checks run out before every criterion is seen, and the chips hide it: "pass · 19 of 19" with 3 unclear, "nothing found" after 3 fixes #354

Closed
opened 2026-11-01 02:26:16 -05:00 by cmoriarty · 1 comment
Owner

What happens
Run 166 (vLLM, stack 126) built cleanly in 93 minutes. Its browser checks still ran out before the judge had seen every criterion, and two chips hid it.

  • The browser checker runs out of turns.
    • Each browser check gets 40 turns (BROWSER_STEPS), and its brief says to stop at about 25 actions (BROWSER_ACTIONS).
    • A key press is one action, so a keyboard-only flow uses them up fast.
    • In apply, the build agent's first check stopped three steps short ("Maximum steps for this browser-check agent have been reached"). It sent a second check to finish, which took 4.0 minutes.
    • In the Visual check, the fourth check ("Keyboard-only: expense flow") made 24 key presses in its 40 turns. It stopped before its last two steps.
  • The four-check cap then stops the judge.
    • Its fifth check ("Finish keyboard check at Reset") was refused: "This session has had 4 browser checks. Do not start another".
    • The cap came from #338, after a Free judge sent seven checks in 25 minutes. On Free the judge now drives the browser itself. The cap still applies to every session that sends browser checks, on every backend.
    • The judge settled the two cut-off steps from earlier checks, after a 5-minute think.
    • It never looked at three criteria and marked them unclear: blocking the removal of a person who has expenses, Reset with Cancel, and Reset with OK. Its evidence for each says the browser check budget (4 checks) was exhausted.
    • Apply's own checks had seen all three work. The app was fine; the judge just never saw them.
  • The Visual check's chip says "pass · 19 of 19". When nothing failed, judge_verdict counts every criterion, so 16 passes and 3 unclear read as 19 of 19.
  • The code review's chip says "nothing found". code_review_verdict counts only findings left open. The reviewer found and fixed three minor issues:
    • the stored-state loader accepted a duplicate sharer id;
    • it also accepted names that differ only in case;
    • app.ts left ids unescaped in four attributes.

What we want

  1. A check that was cut short can be finished. Some shapes:
    • A check that ends UNCLEAR because it ran out of turns doesn't count toward the cap, once. The judge can then send one continuation from the state it left. A spiral like #338's, where checks finish and disagree, still stops at four.
    • Or the cap becomes a profile setting: four on Free, higher on local backends, where a check costs only time.
  2. A keyboard-only flow fits in one check. For example:
    • a browser action that presses a sequence of keys (Tab Tab Tab Enter) in one call, counted as one action;
    • or a judge brief that counts each key press against the 25 and splits keyboard flows before sending them.
  3. The chips say what happened.
    • Visual check: "pass · 16 of 19 · 3 unclear", not shown as a plain pass.
    • Code review: "3 fixed" when everything it found was fixed, with the panel still listing them.

Done when

  • In a sim where the fourth check runs out of turns, the judge can send one more check to finish it, and #338's spiral still stops at four.
  • A keyboard-only check of about 25 key presses finishes inside one browser call.
  • A report like run 166's (16 pass, 3 unclear) reads "pass · 16 of 19 · 3 unclear", and a code review like run 166's reads "3 fixed".

Where to look:

  • BROWSER_CHECKS_MAX and its refusal in src/osf/agent/braid_guards.js (lines 63–68 and 128);
  • BROWSER_STEPS, BROWSER_ACTIONS and the browser agent's brief in src/osf/agent/server.py (373–374 and 455–475);
  • the judge's guidance on sizing checks in src/osf/engine/prompts.py (about 4985–5020);
  • judge_verdict in src/osf/pipeline/fixloop.py (626);
  • code_review_verdict and _weighed in src/osf/pipeline/review.py (about 2320–2372).

Related: #338 (the cap), #315 (the judge's panel), #287 (review chips), #254 (apply plays the app before it's done), #350 (the backend test run 166 belongs to).

Found while reviewing run 166 (braid#350's 8-bit vLLM run), 2026-11-01. Requested by the owner.

**What happens** Run 166 (vLLM, stack 126) built cleanly in 93 minutes. Its browser checks still ran out before the judge had seen every criterion, and two chips hid it. - **The browser checker runs out of turns.** - Each `browser` check gets 40 turns (`BROWSER_STEPS`), and its brief says to stop at about 25 actions (`BROWSER_ACTIONS`). - A key press is one action, so a keyboard-only flow uses them up fast. - In apply, the build agent's first check stopped three steps short ("Maximum steps for this browser-check agent have been reached"). It sent a second check to finish, which took 4.0 minutes. - In the Visual check, the fourth check ("Keyboard-only: expense flow") made 24 key presses in its 40 turns. It stopped before its last two steps. - **The four-check cap then stops the judge.** - Its fifth check ("Finish keyboard check at Reset") was refused: "This session has had 4 browser checks. Do not start another". - The cap came from #338, after a Free judge sent seven checks in 25 minutes. On Free the judge now drives the browser itself. The cap still applies to every session that sends `browser` checks, on every backend. - The judge settled the two cut-off steps from earlier checks, after a 5-minute think. - It never looked at three criteria and marked them `unclear`: blocking the removal of a person who has expenses, Reset with Cancel, and Reset with OK. Its evidence for each says the browser check budget (4 checks) was exhausted. - Apply's own checks had seen all three work. The app was fine; the judge just never saw them. - **The Visual check's chip says "pass · 19 of 19".** When nothing failed, `judge_verdict` counts every criterion, so 16 passes and 3 `unclear` read as 19 of 19. - **The code review's chip says "nothing found".** `code_review_verdict` counts only findings left open. The reviewer found and fixed three minor issues: - the stored-state loader accepted a duplicate sharer id; - it also accepted names that differ only in case; - `app.ts` left ids unescaped in four attributes. **What we want** 1. **A check that was cut short can be finished.** Some shapes: - A check that ends `UNCLEAR` because it ran out of turns doesn't count toward the cap, once. The judge can then send one continuation from the state it left. A spiral like #338's, where checks finish and disagree, still stops at four. - Or the cap becomes a profile setting: four on Free, higher on local backends, where a check costs only time. 2. **A keyboard-only flow fits in one check.** For example: - a browser action that presses a sequence of keys (`Tab Tab Tab Enter`) in one call, counted as one action; - or a judge brief that counts each key press against the 25 and splits keyboard flows before sending them. 3. **The chips say what happened.** - Visual check: "pass · 16 of 19 · 3 unclear", not shown as a plain pass. - Code review: "3 fixed" when everything it found was fixed, with the panel still listing them. **Done when** - In a sim where the fourth check runs out of turns, the judge can send one more check to finish it, and #338's spiral still stops at four. - A keyboard-only check of about 25 key presses finishes inside one `browser` call. - A report like run 166's (16 pass, 3 unclear) reads "pass · 16 of 19 · 3 unclear", and a code review like run 166's reads "3 fixed". **Where to look:** - `BROWSER_CHECKS_MAX` and its refusal in `src/osf/agent/braid_guards.js` (lines 63–68 and 128); - `BROWSER_STEPS`, `BROWSER_ACTIONS` and the `browser` agent's brief in `src/osf/agent/server.py` (373–374 and 455–475); - the judge's guidance on sizing checks in `src/osf/engine/prompts.py` (about 4985–5020); - `judge_verdict` in `src/osf/pipeline/fixloop.py` (626); - `code_review_verdict` and `_weighed` in `src/osf/pipeline/review.py` (about 2320–2372). Related: #338 (the cap), #315 (the judge's panel), #287 (review chips), #254 (apply plays the app before it's done), #350 (the backend test run 166 belongs to). *Found while reviewing run 166 (braid#350's 8-bit vLLM run), 2026-11-01. Requested by the owner.*
Author
Owner

Shipped in PR #356 (merge c00c62f1, live on production c00c62f).

  • A cut-short check can be finished: a browser check that ran to its 40 turns no longer counts toward the session's four, once per session. A sixth check is still refused, so #338's spiral still stops. The new sim browser-check-out-of-turns covers it: the fourth check runs out of turns, the fifth finishes, the sixth is refused. It fails without the guard change and passes with it.
  • Keyboard flows: I didn't add a multi-key action. The Playwright server presses one key per call, and its code-running tool stays out for safety. Instead, the judge's and build's briefs count each key press as an action, and split a keyboard flow of more than about 15 presses. The browser subagent is told a key press takes a turn.
  • Chips:
    • Visual check: pass · 16 of 19 · 3 unclear, in amber.
    • Code review: 3 fixed, or e.g. 1 note · 1 fixed.
    • Tests in tests/pipeline/test_review_verdicts.py.
  • Not run on production: on Free the judge drives the browser itself, with no subagents, so the continuation can't happen there. The sim is the evidence. A vLLM run waits on trogdor.
**Shipped** in PR #356 (merge c00c62f1, live on production c00c62f). - **A cut-short check can be finished:** a `browser` check that ran to its 40 turns no longer counts toward the session's four, once per session. A sixth check is still refused, so #338's spiral still stops. The new sim `browser-check-out-of-turns` covers it: the fourth check runs out of turns, the fifth finishes, the sixth is refused. It fails without the guard change and passes with it. - **Keyboard flows:** I didn't add a multi-key action. The Playwright server presses one key per call, and its code-running tool stays out for safety. Instead, the judge's and build's briefs count each key press as an action, and split a keyboard flow of more than about 15 presses. The browser subagent is told a key press takes a turn. - **Chips:** - Visual check: `pass · 16 of 19 · 3 unclear`, in amber. - Code review: `3 fixed`, or e.g. `1 note · 1 fixed`. - Tests in `tests/pipeline/test_review_verdicts.py`. - **Not run on production:** on Free the judge drives the browser itself, with no subagents, so the continuation can't happen there. The sim is the evidence. A vLLM run waits on trogdor.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#354
No description provided.