A thorough run loses hours to what Braid could tell or do for it (run 62): the last call, the judge, tags, briefs, browsers, run ids, parallel applies, effort #170
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Run 62 built a new project (a browser golf game on plasma-golf) with the thorough pipeline overnight, on qwen3.8-27b with reasoning effort
high. Planning took 40 min. The eight apply groups took 376 min, one after another. write-spec-tests took 60 min (its limit), the code review 66, and the lanes 2 (unit 138/138, e2e 5/5). The judge then ran its full 2 hours without writing a single criterion.Of the 719 agent minutes, the model was generating for 74%, 77% of what it generated was reasoning, and in 86% of its turns no other request was running on trogdor. The tasks below are what Braid itself could have done better: briefs, tools and infrastructure. The numbers come from the run's step transcripts.
Out of scope: the context settings (
agent_context, compaction). Long sessions that slow down a little are preferred over short ones that can never finish.1. The last call makes the model re-read the whole session
Every step that reached its last call in run 62 then sat silent for minutes before its next turn:
A turn at that size normally starts in 3 to 25 s. In run 40 (effort
medium), the turn after a last call started in 2.6 s.Most likely cause. The last call names the
nothinkvariant, which sendschat_template_kwargs: {enable_thinking: false}. At efforthigh, Braid sends no effort, and Qwen3.8's template opens the system message with its "Reasoning effort is set to xhigh" line (seeosf/agent/server.py). If that line, or anything else near the start, renders differently with thinking off, the prompt differs from its first tokens and vLLM's prefix cache can't be used. So all 250–330k tokens are read again.Ruled out with opencode 1.18.34 and fakeinfer: opencode keeps an injected prompt at the end and leaves the earlier messages byte-identical, on an idle session and mid-turn, whatever the prompt's message id. Nothing was dropped from the history either: input tokens barely moved across the stall (247,552 → 248,522).
/tokenize. Render one request with no kwargs, with{enable_thinking: false}, and with{reasoning_effort: low}(thebriefvariant), at efforthighand atmedium, and find where the token streams first differ./tokenizeshows it leaves the prompt's start unchanged.chat_template_kwargsas the request before it.agent.judge,agent.judge.r2andagent.judge.r3, in every pipeline that has them.2. Nothing limits what the judge checks
The judge is told to judge "each scenario a person could see in a browser". For a game, that's nearly all 136 scenarios, and it isn't told which ones the lanes' passing tests already cover. It never used
browser. Instead it wrote about a dozen scripts of its own under itsTMPDIR: Playwright drivers, and a rolldown bundle of the app's physics to check trajectories offline. After 104 minutes itsjudge.jsonstill had no criteria, although the brief says to write them as it goes.browser; scripts of its own only bring the page to a state.propose.specfans out.TMPDIR.3. Braid never matches the test tags to scenarios
scenario_coverage(osf/pipeline/apply.py) counts a// spec:tag only when it is the exact scenario name. The apply brief requires a tag but never says what it should look like, so every apply group wrote slugs such asball-physics/cup-capture.Braid counted 0 of 136 scenarios covered; matching slugs gives 88. As a result, write-spec-tests was told all 136 lacked tests (its agent worked out the 48 on its own), and the judge had nothing to lean on. Related: #138.
// spec: <capability>/<scenario>, with the scenario's name exactly as its#### Scenario:heading has it.scenario_coveragealso accepts a normalised slug: lower case, with runs of other characters turned into-.4. Specs are cut off at 8,000 characters in briefs
MAX_ARTIFACT_CHARS(osf/engine/prompts.py) truncated 4 of the 6 delta specs in every apply, review, write-spec-tests and judge brief. Agents re-read pasted files 31 times as a result.Each apply brief was 84k characters, 43k of them all six specs, even for the renderer group, which needed two.
5. write-spec-tests planned everything before writing anything
It spent 44 of its 60 minutes in six long thoughts (4.5 to 12.4 min each) before its first write. Its only file came at 55.8 min, after the last call, and was never run. That left 4 failing tests for the code review to diagnose; one of them did catch a real bug, a bounce the spec forbids.
propose.specdoes: tests only, each in files of its own.6. The project's Playwright can't get the browser it needs
The image's Chromium (build 1243) matches Braid's Playwright MCP (#157) and serves the
browsersubagent well. The project's own@playwright/testis a different matter: 1.64 here, which needs build 1248.PLAYWRIGHT_BROWSERS_PATHpoints at the image's/opt/ms-playwright, which only root can write to, sonpx playwright installfails with EACCES.apply[8] lost about 14 minutes to this: the install timed out at 300 s, then failed after 473 s, then succeeded in 31 s into a cache inside the run. The judge later pinned a browser executable by hand. On a laptop this never comes up, because
~/.cache/ms-playwrightis writable and the project installs its own build once.script.dependencies(andscript.dependencies.tests) installs the browsers the project's Playwright wants, right after the packages, whenever its manifest depends on@playwright/testorplaywright.7. Agents mistype the run id
In absolute paths the model wrote
…288HGSE4for…288HGZHGSE4. That happened in 9 tool calls across 6 steps, and caused 9 of the 13 permission rejections. The rejection message doesn't name the path, so agents blamed a long timeout or "a temporary glitch", and the judge shortened its timeouts because of it.The id is a ULID: sortable and unique without coordination, but it puts 16 random characters into every path an agent sees (worktree,
HOME,TMPDIR, screenshots).$TMPDIRworks.runs/62/andosf/62-…. The database and API keep the ULID, and existing runs keep their folders.8. Report length limits aren't in the briefs
The code review's first attempt made 59.5 minutes of real fixes, then failed only because
findings[].whatmust be at most 800 characters andtests.commandat most 400. Its brief says neither. The second attempt spent 6.7 min in a new session shortening the text.osf/pipeline/nodes.pyso they cannot drift. That includes the code review and the judge (criterionat most 600,evidenceat most 4,000).9. Two quirks of this environment that briefs don't mention
127.0.0.1. Herelocalhostresolved to::1, and Playwright's webServer waited out its 240 s.10. Apply groups in parallel, safely
Groups run one at a time on purpose (
_earlier_siblings_allowinosf/engine/scheduler.py). When they once started together in one worktree, group 2 found group 1's work missing and redid it in the same files. Run 62 then spent 376 min on a single stream, though a stream barely slows when others run beside it: 47 tok/s alone, 47 with one more, 41–43 with two or three. Related: #128.tasks.mdgroups declare which groups they depend on. The tasks step writes this, the review checks it, andscript.budgetrecords it. In run 62, physics (3) and the renderer (5) needed only groups 1 and 2.11. Reasoning effort: Auto
Effort is a single setting each for implementation and planning; run 62 had both at
high. Thoughts of two minutes or more (59 of them) held 67% of its 408 minutes of reasoning.highfor planning, apply and fix;mediumfor the reviews, code review, write-spec-tests and the judge;lowfor summaries.agent.fixforking an apply step) uses the effort of the session it forks. A change re-renders the start of the prompt (task 1).Done when
Every task above is done or split off. Then a thorough run on a new project, compared with run 62 by model turns and generated tokens (wall clocks only roughly, since trogdor's speed changes), shows:
Shipped on main through ten OpenSpec changes, archived. What each section got, with the numbers measured on the way:
1. The last call
/tokenize(tools/template_probe.py): only a request with the session's own settings keeps the start of the prompt.{enable_thinking: false}changes it at token 3 at effort high (token 359 at medium), and dropping the tools changes it at token 41.prompts-keep-the-prefix). The handoff note is now written to a file with the session's own tools. Live on trogdor at about 85k tokens of context, the turn after a last call started in 1.1 s, against 64.0 s with the oldnothinkvariant.tests/unit/test_prompt_prefix.py(pinned opencode against fakeinfer) holds that every later request carries the first one'schat_template_kwargs, tools and system message; a simulated run covers the handoff note.agent.judge,.r2,.r3).2. What the judge checks
verify.jsonnow lists them).Bcriteria for each page's look, layout, readable text and errors on the page or in the console. Up to 60 criteria, ids up to 200 characters.judge.jsonper round, and with the covered scenarios gone run 62's judge would have had at most 48 criteria (the change's design).TMPDIR.3. Tags
#### Scenario:heading rule.scenario_coveragealso matches a normalised slug. On run 62's 136 scenarios: 0 matched exactly, 88 by slug.4. Specs in briefs
Specs:line) and the whole task list, and names the other specs by requirement and scenario. No brief cuts a spec silently: a cut names the path to read the rest.5. write-spec-tests
.osf/test/scenarios.json), one at a time; its brief says to write and run one test file before the next. It may change only tests.6. The project's Playwright browsers
osfd doctorchecks the cache is writable.7. Run ids
$TMPDIR.runs/<n>/and its branchosf/<n>-<slug>, from its ledger number at admission; older runs keep theirs.8. Report limits
9. This machine's quirks
127.0.0.1, and that WebGL here runs in software, so frame rates measured here are not real.10. Apply groups in parallel
tasks.mdgroups declareDepends on:, the review checks it,.osf/budget.jsonrecords it, and a group waits only for its dependencies: a failed group now stops only its dependents.PREVIEW_PORT, which only opencode 2's per-session environment can change.11. Effort Auto
efforts:map names (built-ins:highfor exploring, proposing, applying, reconciling, fixing and revising,lowfor the summary,mediumotherwise), and the page lists the map. A session keeps one effort, its subagents' included, and a fork keeps its parent's.mediumandlow, and thought 289, 143 and 94 characters on the same question. That check found that opencode fills an entry's missing options from the base entry, which sent the@highsteps out atmedium; the base now sends no effort, andtests/unit/test_effort_entries.pyholds it.Verification
Fast lane, full lane and all 16 simulated runs pass; the few browser specs and the one real-opencode test that timed out under the machine's load pass alone. The full lane also turned up two flaws in the simulated runs' own stack, fixed in
fe45309: a port range taken from the next ephemeral port, and a settings save on the readiness poll's 2 s timeout.Production serves
3fe507fsince 08:06:35Z.osfd doctorpasses, the Playwright cache holds the image's links, and the eight earlier runs kept their numbers, folders and branches. Nine changes are archived (ea9142a);short-run-namesfollows once the next run shows itsruns/<n>/folder andosf/<n>-…branch.Still to do for this issue: the thorough run on a new project, compared with run 62. It stays open until then.
validate --all --strictfails on 42 specs #156Comparison run: run 65, stopped after the reviews
Run 65 took run 62's brief on
thoroughat effort Auto, against a fresh empty repository, with every change of this issue deployed. It is stopped at the user's request after the planning and review phase (16 agent steps, all succeeded), before the apply groups, the last call and the judge. It is kept and can be resumed.On the 14 agent steps both runs have (explore, proposal, four of the specs, design, tasks, the four reviews):
What this shows:
high, the four reviews and the second review atmedium. Run 62 sent everything at the run's one effort.hole-content.Not yet exercised:
These are left for the single verification run that will check the remaining changes together. Parallel apply groups moved to #172. This issue stays open until that run has checked them.
Run 66: the checks run 65 didn't reach
Run 66 (
run_01M52JT93Q9X1VH3GD05D2HMZY) is athoroughrun on a new project: a link shortener on Postgres, in a fresh public repository, cmoriarty/braid-combined-test. It ran on production (57f9a21) on trogdor at effort Auto. It succeeded, through the pull request, merge and archive. The #171 session's local quick-fix run shared trogdor from 15:50 to 15:59Z; the timing checks below fall outside that window.script.scenariosread "29 of 31 scenarios have a tagged test; 2 have none, in 2 capabilities". The step fanned out over just those 2, one after the other (2.7 and about 21 min). The pull request then reads "All 31 scenarios … have a tagged test".ms-playwright. Even so,npx playwright install chromiumdownloaded nothing in every round, and the e2e lane ran in 14 s.osf.step.last_callwas sent.Two of the five, the last call and the refusal, still have only their unit and simulated tests. A real run reaches them only by accident: a step near its limit, an agent reaching outside its folder. This issue can close on the rest, or stay open until a run happens to hit them.
Closing on the evidence of runs 65, 66 and the verification run (
run_01M53R7621CJW6PCZG7S6R94HS, thorough on cmoriarty/braid-combined-test, vLLM, succeeded and merged at 03:49Z).handoff-note.Please reopen if either misbehaves on a real run.
unclearcontradiction, each judge writing its own validator, reviewers re-reading pasted text #386