A thorough run loses hours to what Braid could tell or do for it (run 62): the last call, the judge, tags, briefs, browsers, run ids, parallel applies, effort #170

Closed
opened 2026-10-13 11:11:33 -04:00 by cmoriarty · 4 comments
Owner

Run 62 built a new project (a browser golf game on plasma-golf) with the thorough pipeline overnight, on qwen3.8-27b with reasoning effort high. Planning took 40 min. The eight apply groups took 376 min, one after another. write-spec-tests took 60 min (its limit), the code review 66, and the lanes 2 (unit 138/138, e2e 5/5). The judge then ran its full 2 hours without writing a single criterion.

Of the 719 agent minutes, the model was generating for 74%, 77% of what it generated was reasoning, and in 86% of its turns no other request was running on trogdor. The tasks below are what Braid itself could have done better: briefs, tools and infrastructure. The numbers come from the run's step transcripts.

Out of scope: the context settings (agent_context, compaction). Long sessions that slow down a little are preferred over short ones that can never finish.

1. The last call makes the model re-read the whole session

Every step that reached its last call in run 62 then sat silent for minutes before its next turn:

Step Context Wait for the next turn's first token
code review 247k 4.5 min
write-spec-tests 272k about 4 min, until it timed out before the turn began
judge 329k 6+ min, while vLLM's KV usage climbed and no tokens came out

A turn at that size normally starts in 3 to 25 s. In run 40 (effort medium), the turn after a last call started in 2.6 s.

Most likely cause. The last call names the nothink variant, which sends chat_template_kwargs: {enable_thinking: false}. At effort high, Braid sends no effort, and Qwen3.8's template opens the system message with its "Reasoning effort is set to xhigh" line (see osf/agent/server.py). If that line, or anything else near the start, renders differently with thinking off, the prompt differs from its first tokens and vLLM's prefix cache can't be used. So all 250–330k tokens are read again.

Ruled out with opencode 1.18.34 and fakeinfer: opencode keeps an injected prompt at the end and leaves the earlier messages byte-identical, on an idle session and mid-turn, whatever the prompt's message id. Nothing was dropped from the history either: input tokens barely moved across the stall (247,552 → 248,522).

  • Confirm on trogdor through vLLM's /tokenize. Render one request with no kwargs, with {enable_thinking: false}, and with {reasoning_effort: low} (the brief variant), at effort high and at medium, and find where the token streams first differ.
  • Fix: a prompt sent into a running session must not change the start of the prompt. That covers the last call, the cut-off re-prompt, the handoff, and the console's steer, queue and redirect. Keep the session's own template settings, and ask for brevity in the text. Name a variant only where /tokenize shows it leaves the prompt's start unchanged.
  • A test, or a sim scenario, showing that the request after a last call carries the same chat_template_kwargs as the request before it.
  • Raise the judge's limit from 2 h to 3 h: agent.judge, agent.judge.r2 and agent.judge.r3, in every pipeline that has them.

2. Nothing limits what the judge checks

The judge is told to judge "each scenario a person could see in a browser". For a game, that's nearly all 136 scenarios, and it isn't told which ones the lanes' passing tests already cover. It never used browser. Instead it wrote about a dozen scripts of its own under its TMPDIR: Playwright drivers, and a rolldown bundle of the app's physics to check trajectories offline. After 104 minutes its judge.json still had no criteria, although the brief says to write them as it goes.

  • The judge's brief carries the lanes' results: which lanes passed, and which scenarios their passing tests cover (by tag, see task 3).
  • It judges only what no passing test covers, plus what needs a person's eyes: the look, the layout, readable text, errors on the page.
  • It looks through browser; scripts of its own only bring the page to a state.
  • Consider one judge per capability when there are many scenarios, the way propose.spec fans out.
  • A retried judge is told what the earlier attempt left in its TMPDIR.

3. Braid never matches the test tags to scenarios

scenario_coverage (osf/pipeline/apply.py) counts a // spec: tag only when it is the exact scenario name. The apply brief requires a tag but never says what it should look like, so every apply group wrote slugs such as ball-physics/cup-capture.

Braid counted 0 of 136 scenarios covered; matching slugs gives 88. As a result, write-spec-tests was told all 136 lacked tests (its agent worked out the 48 on its own), and the judge had nothing to lean on. Related: #138.

  • The apply brief states the tag format, as write-spec-tests' brief already does: // spec: <capability>/<scenario>, with the scenario's name exactly as its #### Scenario: heading has it.
  • scenario_coverage also accepts a normalised slug: lower case, with runs of other characters turned into -.

4. Specs are cut off at 8,000 characters in briefs

MAX_ARTIFACT_CHARS (osf/engine/prompts.py) truncated 4 of the 6 delta specs in every apply, review, write-spec-tests and judge brief. Agents re-read pasted files 31 times as a result.

Each apply brief was 84k characters, 43k of them all six specs, even for the renderer group, which needed two.

  • An apply group's brief pastes, in full, the delta specs of the capabilities its tasks touch, and lists the others by requirement and scenario name.
  • No brief cuts a spec off silently: it either pastes the whole spec, or gives the names and the path to read.

5. write-spec-tests planned everything before writing anything

It spent 44 of its 60 minutes in six long thoughts (4.5 to 12.4 min each) before its first write. Its only file came at 55.8 min, after the last call, and was never run. That left 4 failing tests for the code review to diagnose; one of them did catch a real bug, a bounce the spec forbids.

  • The brief says to work one test file at a time: read what that file needs, write it, run it, then move to the next.
  • Fan it out per capability, the way propose.spec does: tests only, each in files of its own.

6. The project's Playwright can't get the browser it needs

The image's Chromium (build 1243) matches Braid's Playwright MCP (#157) and serves the browser subagent well. The project's own @playwright/test is a different matter: 1.64 here, which needs build 1248. PLAYWRIGHT_BROWSERS_PATH points at the image's /opt/ms-playwright, which only root can write to, so npx playwright install fails with EACCES.

apply[8] lost about 14 minutes to this: the install timed out at 300 s, then failed after 473 s, then succeeded in 31 s into a cache inside the run. The judge later pinned a browser executable by hand. On a laptop this never comes up, because ~/.cache/ms-playwright is writable and the project installs its own build once.

  • script.dependencies (and script.dependencies.tests) installs the browsers the project's Playwright wants, right after the packages, whenever its manifest depends on @playwright/test or playwright.
  • They go into one writable cache on the state volume, shared by all runs, so each version is downloaded once. The cache also holds links to the image's builds, so the MCP server still finds its own. Agents' shells, the lanes and the MCP server all get this path. Playwright locks the folder while it installs, so concurrent runs are safe. Related: #67, #157.

7. Agents mistype the run id

In absolute paths the model wrote …288HGSE4 for …288HGZHGSE4. That happened in 9 tool calls across 6 steps, and caused 9 of the 13 permission rejections. The rejection message doesn't name the path, so agents blamed a long timeout or "a temporary glitch", and the judge shortened its timeouts because of it.

The id is a ULID: sortable and unique without coordination, but it puts 16 random characters into every path an agent sees (worktree, HOME, TMPDIR, screenshots).

  • A permission rejection names the path it refused, and the run's own folder.
  • Briefs avoid absolute run paths wherever a relative path or $TMPDIR works.
  • Give agents shorter names to copy. Each run gets a short number when it is created (the ledger number the console shows, stored rather than computed), and its folder and work branch are named by it, e.g. runs/62/ and osf/62-…. The database and API keep the ULID, and existing runs keep their folders.

8. Report length limits aren't in the briefs

The code review's first attempt made 59.5 minutes of real fixes, then failed only because findings[].what must be at most 800 characters and tests.command at most 400. Its brief says neither. The second attempt spent 6.7 min in a new session shortening the text.

  • Every brief that asks for a report states its schema's length limits, generated from the schemas in osf/pipeline/nodes.py so they cannot drift. That includes the code review and the judge (criterion at most 600, evidence at most 4,000).

9. Two quirks of this environment that briefs don't mention

  • Briefs that serve or test a web app say to bind to 127.0.0.1. Here localhost resolved to ::1, and Playwright's webServer waited out its 240 s.
  • They say WebGL here runs in software (SwiftShader, about 5–6 fps at 1080p), so frame rates measured here aren't real: don't tune the product for them or gate a test on them. apply[8] spent about 15 min building automatic resolution scaling into the renderer to pass an fps ≥ 20 check.

10. Apply groups in parallel, safely

Groups run one at a time on purpose (_earlier_siblings_allow in osf/engine/scheduler.py). When they once started together in one worktree, group 2 found group 1's work missing and redid it in the same files. Run 62 then spent 376 min on a single stream, though a stream barely slows when others run beside it: 47 tok/s alone, 47 with one more, 41–43 with two or three. Related: #128.

  • tasks.md groups declare which groups they depend on. The tasks step writes this, the review checks it, and script.budget records it. In run 62, physics (3) and the renderer (5) needed only groups 1 and 2.
  • A group starts once its dependencies are done, in a worktree and on a branch of its own, starting from the checkpoint its dependencies left.
  • When it finishes, Braid merges its branch into the run's worktree. A conflict goes to a resolve step, as a conflict with the base branch does, and is never silently overwritten.
  • Concurrency stays bounded by the box's slots.

11. Reasoning effort: Auto

Effort is a single setting each for implementation and planning; run 62 had both at high. Thoughts of two minutes or more (59 of them) held 67% of its 408 minutes of reasoning.

  • Settings offer Auto, which picks the effort per step from a map in the pipeline, shown in Settings. For example: high for planning, apply and fix; medium for the reviews, code review, write-spec-tests and the judge; low for summaries.
  • One effort per session: a step keeps its effort for all its turns, and a forked session (agent.fix forking an apply step) uses the effort of the session it forks. A change re-renders the start of the prompt (task 1).

Done when

Every task above is done or split off. Then a thorough run on a new project, compared with run 62 by model turns and generated tokens (wall clocks only roughly, since trogdor's speed changes), shows:

  • no stall after a last call;
  • Braid's scenario count agreeing with the tags;
  • no re-reads of pasted specs;
  • a judge that writes criteria as it goes and finishes within its limit.
Run 62 built a new project (a browser golf game on plasma-golf) with the thorough pipeline overnight, on qwen3.8-27b with reasoning effort `high`. Planning took 40 min. The eight apply groups took 376 min, one after another. write-spec-tests took 60 min (its limit), the code review 66, and the lanes 2 (unit 138/138, e2e 5/5). The judge then ran its full 2 hours without writing a single criterion. Of the 719 agent minutes, the model was generating for 74%, 77% of what it generated was reasoning, and in 86% of its turns no other request was running on trogdor. The tasks below are what Braid itself could have done better: briefs, tools and infrastructure. The numbers come from the run's step transcripts. **Out of scope: the context settings** (`agent_context`, compaction). Long sessions that slow down a little are preferred over short ones that can never finish. ## 1. The last call makes the model re-read the whole session Every step that reached its last call in run 62 then sat silent for minutes before its next turn: | Step | Context | Wait for the next turn's first token | |---|---|---| | code review | 247k | 4.5 min | | write-spec-tests | 272k | about 4 min, until it timed out before the turn began | | judge | 329k | 6+ min, while vLLM's KV usage climbed and no tokens came out | A turn at that size normally starts in 3 to 25 s. In run 40 (effort `medium`), the turn after a last call started in 2.6 s. **Most likely cause.** The last call names the `nothink` variant, which sends `chat_template_kwargs: {enable_thinking: false}`. At effort `high`, Braid sends no effort, and Qwen3.8's template opens the system message with its "Reasoning effort is set to xhigh" line (see `osf/agent/server.py`). If that line, or anything else near the start, renders differently with thinking off, the prompt differs from its first tokens and vLLM's prefix cache can't be used. So all 250–330k tokens are read again. **Ruled out** with opencode 1.18.34 and fakeinfer: opencode keeps an injected prompt at the end and leaves the earlier messages byte-identical, on an idle session and mid-turn, whatever the prompt's message id. Nothing was dropped from the history either: input tokens barely moved across the stall (247,552 → 248,522). - [ ] Confirm on trogdor through vLLM's `/tokenize`. Render one request with no kwargs, with `{enable_thinking: false}`, and with `{reasoning_effort: low}` (the `brief` variant), at effort `high` and at `medium`, and find where the token streams first differ. - [ ] Fix: a prompt sent into a running session must not change the start of the prompt. That covers the last call, the cut-off re-prompt, the handoff, and the console's steer, queue and redirect. Keep the session's own template settings, and ask for brevity in the text. Name a variant only where `/tokenize` shows it leaves the prompt's start unchanged. - [ ] A test, or a sim scenario, showing that the request after a last call carries the same `chat_template_kwargs` as the request before it. - [ ] Raise the judge's limit from 2 h to 3 h: `agent.judge`, `agent.judge.r2` and `agent.judge.r3`, in every pipeline that has them. ## 2. Nothing limits what the judge checks The judge is told to judge "each scenario a person could see in a browser". For a game, that's nearly all 136 scenarios, and it isn't told which ones the lanes' passing tests already cover. It never used `browser`. Instead it wrote about a dozen scripts of its own under its `TMPDIR`: Playwright drivers, and a rolldown bundle of the app's physics to check trajectories offline. After 104 minutes its `judge.json` still had no criteria, although the brief says to write them as it goes. - [ ] The judge's brief carries the lanes' results: which lanes passed, and which scenarios their passing tests cover (by tag, see task 3). - [ ] It judges only what no passing test covers, plus what needs a person's eyes: the look, the layout, readable text, errors on the page. - [ ] It looks through `browser`; scripts of its own only bring the page to a state. - [ ] Consider one judge per capability when there are many scenarios, the way `propose.spec` fans out. - [ ] A retried judge is told what the earlier attempt left in its `TMPDIR`. ## 3. Braid never matches the test tags to scenarios `scenario_coverage` (`osf/pipeline/apply.py`) counts a `// spec:` tag only when it is the exact scenario name. The apply brief requires a tag but never says what it should look like, so every apply group wrote slugs such as `ball-physics/cup-capture`. Braid counted 0 of 136 scenarios covered; matching slugs gives 88. As a result, write-spec-tests was told all 136 lacked tests (its agent worked out the 48 on its own), and the judge had nothing to lean on. Related: #138. - [ ] The apply brief states the tag format, as write-spec-tests' brief already does: `// spec: <capability>/<scenario>`, with the scenario's name exactly as its `#### Scenario:` heading has it. - [ ] `scenario_coverage` also accepts a normalised slug: lower case, with runs of other characters turned into `-`. ## 4. Specs are cut off at 8,000 characters in briefs `MAX_ARTIFACT_CHARS` (`osf/engine/prompts.py`) truncated 4 of the 6 delta specs in every apply, review, write-spec-tests and judge brief. Agents re-read pasted files 31 times as a result. Each apply brief was 84k characters, 43k of them all six specs, even for the renderer group, which needed two. - [ ] An apply group's brief pastes, in full, the delta specs of the capabilities its tasks touch, and lists the others by requirement and scenario name. - [ ] No brief cuts a spec off silently: it either pastes the whole spec, or gives the names and the path to read. ## 5. write-spec-tests planned everything before writing anything It spent 44 of its 60 minutes in six long thoughts (4.5 to 12.4 min each) before its first write. Its only file came at 55.8 min, after the last call, and was never run. That left 4 failing tests for the code review to diagnose; one of them did catch a real bug, a bounce the spec forbids. - [ ] The brief says to work one test file at a time: read what that file needs, write it, run it, then move to the next. - [ ] Fan it out per capability, the way `propose.spec` does: tests only, each in files of its own. ## 6. The project's Playwright can't get the browser it needs The image's Chromium (build 1243) matches Braid's Playwright MCP (#157) and serves the `browser` subagent well. The project's own `@playwright/test` is a different matter: 1.64 here, which needs build 1248. `PLAYWRIGHT_BROWSERS_PATH` points at the image's `/opt/ms-playwright`, which only root can write to, so `npx playwright install` fails with EACCES. apply[8] lost about 14 minutes to this: the install timed out at 300 s, then failed after 473 s, then succeeded in 31 s into a cache inside the run. The judge later pinned a browser executable by hand. On a laptop this never comes up, because `~/.cache/ms-playwright` is writable and the project installs its own build once. - [ ] `script.dependencies` (and `script.dependencies.tests`) installs the browsers the project's Playwright wants, right after the packages, whenever its manifest depends on `@playwright/test` or `playwright`. - [ ] They go into one writable cache on the state volume, shared by all runs, so each version is downloaded once. The cache also holds links to the image's builds, so the MCP server still finds its own. Agents' shells, the lanes and the MCP server all get this path. Playwright locks the folder while it installs, so concurrent runs are safe. Related: #67, #157. ## 7. Agents mistype the run id In absolute paths the model wrote `…288HGSE4` for `…288HGZHGSE4`. That happened in 9 tool calls across 6 steps, and caused 9 of the 13 permission rejections. The rejection message doesn't name the path, so agents blamed a long timeout or "a temporary glitch", and the judge shortened its timeouts because of it. The id is a ULID: sortable and unique without coordination, but it puts 16 random characters into every path an agent sees (worktree, `HOME`, `TMPDIR`, screenshots). - [ ] A permission rejection names the path it refused, and the run's own folder. - [ ] Briefs avoid absolute run paths wherever a relative path or `$TMPDIR` works. - [ ] Give agents shorter names to copy. Each run gets a short number when it is created (the ledger number the console shows, stored rather than computed), and its folder and work branch are named by it, e.g. `runs/62/` and `osf/62-…`. The database and API keep the ULID, and existing runs keep their folders. ## 8. Report length limits aren't in the briefs The code review's first attempt made 59.5 minutes of real fixes, then failed only because `findings[].what` must be at most 800 characters and `tests.command` at most 400. Its brief says neither. The second attempt spent 6.7 min in a new session shortening the text. - [ ] Every brief that asks for a report states its schema's length limits, generated from the schemas in `osf/pipeline/nodes.py` so they cannot drift. That includes the code review and the judge (`criterion` at most 600, `evidence` at most 4,000). ## 9. Two quirks of this environment that briefs don't mention - [ ] Briefs that serve or test a web app say to bind to `127.0.0.1`. Here `localhost` resolved to `::1`, and Playwright's webServer waited out its 240 s. - [ ] They say WebGL here runs in software (SwiftShader, about 5–6 fps at 1080p), so frame rates measured here aren't real: don't tune the product for them or gate a test on them. apply[8] spent about 15 min building automatic resolution scaling into the renderer to pass an fps ≥ 20 check. ## 10. Apply groups in parallel, safely Groups run one at a time on purpose (`_earlier_siblings_allow` in `osf/engine/scheduler.py`). When they once started together in one worktree, group 2 found group 1's work missing and redid it in the same files. Run 62 then spent 376 min on a single stream, though a stream barely slows when others run beside it: 47 tok/s alone, 47 with one more, 41–43 with two or three. Related: #128. - [ ] `tasks.md` groups declare which groups they depend on. The tasks step writes this, the review checks it, and `script.budget` records it. In run 62, physics (3) and the renderer (5) needed only groups 1 and 2. - [ ] A group starts once its dependencies are done, in a worktree and on a branch of its own, starting from the checkpoint its dependencies left. - [ ] When it finishes, Braid merges its branch into the run's worktree. A conflict goes to a resolve step, as a conflict with the base branch does, and is never silently overwritten. - [ ] Concurrency stays bounded by the box's slots. ## 11. Reasoning effort: Auto Effort is a single setting each for implementation and planning; run 62 had both at `high`. Thoughts of two minutes or more (59 of them) held 67% of its 408 minutes of reasoning. - [ ] Settings offer **Auto**, which picks the effort per step from a map in the pipeline, shown in Settings. For example: `high` for planning, apply and fix; `medium` for the reviews, code review, write-spec-tests and the judge; `low` for summaries. - [ ] One effort per session: a step keeps its effort for all its turns, and a forked session (`agent.fix` forking an apply step) uses the effort of the session it forks. A change re-renders the start of the prompt (task 1). ## Done when Every task above is done or split off. Then a thorough run on a new project, compared with run 62 by model turns and generated tokens (wall clocks only roughly, since trogdor's speed changes), shows: - no stall after a last call; - Braid's scenario count agreeing with the tags; - no re-reads of pasted specs; - a judge that writes criteria as it goes and finishes within its limit.
Author
Owner

Shipped on main through ten OpenSpec changes, archived. What each section got, with the numbers measured on the way:

1. The last call

  • Confirmed on trogdor with /tokenize (tools/template_probe.py): only a request with the session's own settings keeps the start of the prompt. {enable_thinking: false} changes it at token 3 at effort high (token 359 at medium), and dropping the tools changes it at token 41.
  • Fixed: the last call, the cut-off re-prompt, the handoff turn and the console's queued messages are sent with the step's own tools and settings and no variant (prompts-keep-the-prefix). The handoff note is now written to a file with the session's own tools. Live on trogdor at about 85k tokens of context, the turn after a last call started in 1.1 s, against 64.0 s with the old nothink variant.
  • tests/unit/test_prompt_prefix.py (pinned opencode against fakeinfer) holds that every later request carries the first one's chat_template_kwargs, tools and system message; a simulated run covers the handoff note.
  • The judge's limit is 3 h (agent.judge, .r2, .r3).

2. What the judge checks

  • Its brief carries the lanes that passed, with their commands, and the scenarios their passing tests are tagged with, from the round's record (verify.json now lists them).
  • It judges only the scenarios no passing test covers, plus B criteria for each page's look, layout, readable text and errors on the page or in the console. Up to 60 criteria, ids up to 200 characters.
  • It looks through the browser it has, and a script of its own may only bring the page to a state.
  • One judge per capability: not taken. The loop reads one judge.json per round, and with the covered scenarios gone run 62's judge would have had at most 48 criteria (the change's design).
  • A retried judge is told what the earlier attempt left in its TMPDIR.

3. Tags

  • The apply, unit-test and spec-test briefs give the tag with an example and the exact #### Scenario: heading rule.
  • scenario_coverage also matches a normalised slug. On run 62's 136 scenarios: 0 matched exactly, 88 by slug.

4. Specs in briefs

  • An apply group's brief pastes its own delta specs whole (its tasks' Specs: line) and the whole task list, and names the other specs by requirement and scenario. No brief cuts a spec silently: a cut names the path to read the rest.

5. write-spec-tests

  • One step per capability with an untested scenario, from a scripted count (.osf/test/scenarios.json), one at a time; its brief says to write and run one test file before the next. It may change only tests.

6. The project's Playwright browsers

  • The dependency steps install the browsers the project's own Playwright wants into one writable cache on the state volume, shared by every run, with links to the image's builds: installing the image's version takes 0.6 s and downloads nothing. osfd doctor checks the cache is writable.

7. Run ids

  • A rejected path is named in the rejection, with the run's own folder.
  • Briefs use relative paths and $TMPDIR.
  • A new run's folder is runs/<n>/ and its branch osf/<n>-<slug>, from its ledger number at admission; older runs keep theirs.

8. Report limits

  • Every brief that asks for a report states its schema's limits, generated from the schemas.

9. This machine's quirks

  • Web briefs say to bind to 127.0.0.1, and that WebGL here runs in software, so frame rates measured here are not real.

10. Apply groups in parallel

  • Done: tasks.md groups declare Depends on:, the review checks it, .osf/budget.json records it, and a group waits only for its dependencies: a failed group now stops only its dependents.
  • Split off as #172, to follow #171: a group in its own worktree, merged back. Measured on opencode 1.18.35, a group's session lives in its own directory for the busy map and the events, and every session of a run shares one PREVIEW_PORT, which only opencode 2's per-session environment can change.

11. Effort Auto

  • Settings → Reasoning effort has Auto: each step thinks at the effort its pipeline's efforts: map names (built-ins: high for exploring, proposing, applying, reconciling, fixing and revising, low for the summary, medium otherwise), and the page lists the map. A session keeps one effort, its subagents' included, and a fork keeps its parent's.
  • Live on trogdor, an apply group, a review lens and the summary step sent nothing (the template's own default), medium and low, and thought 289, 143 and 94 characters on the same question. That check found that opencode fills an entry's missing options from the base entry, which sent the @high steps out at medium; the base now sends no effort, and tests/unit/test_effort_entries.py holds it.

Verification

Fast lane, full lane and all 16 simulated runs pass; the few browser specs and the one real-opencode test that timed out under the machine's load pass alone. The full lane also turned up two flaws in the simulated runs' own stack, fixed in fe45309: a port range taken from the next ephemeral port, and a settings save on the readiness poll's 2 s timeout.

Production serves 3fe507f since 08:06:35Z. osfd doctor passes, the Playwright cache holds the image's links, and the eight earlier runs kept their numbers, folders and branches. Nine changes are archived (ea9142a); short-run-names follows once the next run shows its runs/<n>/ folder and osf/<n>-… branch.

Still to do for this issue: the thorough run on a new project, compared with run 62. It stays open until then.

Shipped on main through ten OpenSpec changes, archived. What each section got, with the numbers measured on the way: ## 1. The last call - Confirmed on trogdor with `/tokenize` (`tools/template_probe.py`): only a request with the session's own settings keeps the start of the prompt. `{enable_thinking: false}` changes it at token 3 at effort high (token 359 at medium), and dropping the tools changes it at token 41. - Fixed: the last call, the cut-off re-prompt, the handoff turn and the console's queued messages are sent with the step's own tools and settings and no variant (`prompts-keep-the-prefix`). The handoff note is now written to a file with the session's own tools. Live on trogdor at about 85k tokens of context, the turn after a last call started in 1.1 s, against 64.0 s with the old `nothink` variant. - `tests/unit/test_prompt_prefix.py` (pinned opencode against fakeinfer) holds that every later request carries the first one's `chat_template_kwargs`, tools and system message; a simulated run covers the handoff note. - The judge's limit is 3 h (`agent.judge`, `.r2`, `.r3`). ## 2. What the judge checks - Its brief carries the lanes that passed, with their commands, and the scenarios their passing tests are tagged with, from the round's record (`verify.json` now lists them). - It judges only the scenarios no passing test covers, plus `B` criteria for each page's look, layout, readable text and errors on the page or in the console. Up to 60 criteria, ids up to 200 characters. - It looks through the browser it has, and a script of its own may only bring the page to a state. - One judge per capability: not taken. The loop reads one `judge.json` per round, and with the covered scenarios gone run 62's judge would have had at most 48 criteria (the change's design). - A retried judge is told what the earlier attempt left in its `TMPDIR`. ## 3. Tags - The apply, unit-test and spec-test briefs give the tag with an example and the exact `#### Scenario:` heading rule. - `scenario_coverage` also matches a normalised slug. On run 62's 136 scenarios: 0 matched exactly, 88 by slug. ## 4. Specs in briefs - An apply group's brief pastes its own delta specs whole (its tasks' `Specs:` line) and the whole task list, and names the other specs by requirement and scenario. No brief cuts a spec silently: a cut names the path to read the rest. ## 5. write-spec-tests - One step per capability with an untested scenario, from a scripted count (`.osf/test/scenarios.json`), one at a time; its brief says to write and run one test file before the next. It may change only tests. ## 6. The project's Playwright browsers - The dependency steps install the browsers the project's own Playwright wants into one writable cache on the state volume, shared by every run, with links to the image's builds: installing the image's version takes 0.6 s and downloads nothing. `osfd doctor` checks the cache is writable. ## 7. Run ids - A rejected path is named in the rejection, with the run's own folder. - Briefs use relative paths and `$TMPDIR`. - A new run's folder is `runs/<n>/` and its branch `osf/<n>-<slug>`, from its ledger number at admission; older runs keep theirs. ## 8. Report limits - Every brief that asks for a report states its schema's limits, generated from the schemas. ## 9. This machine's quirks - Web briefs say to bind to `127.0.0.1`, and that WebGL here runs in software, so frame rates measured here are not real. ## 10. Apply groups in parallel - Done: `tasks.md` groups declare `Depends on:`, the review checks it, `.osf/budget.json` records it, and a group waits only for its dependencies: a failed group now stops only its dependents. - Split off as #172, to follow #171: a group in its own worktree, merged back. Measured on opencode 1.18.35, a group's session lives in its own directory for the busy map and the events, and every session of a run shares one `PREVIEW_PORT`, which only opencode 2's per-session environment can change. ## 11. Effort Auto - Settings → Reasoning effort has Auto: each step thinks at the effort its pipeline's `efforts:` map names (built-ins: `high` for exploring, proposing, applying, reconciling, fixing and revising, `low` for the summary, `medium` otherwise), and the page lists the map. A session keeps one effort, its subagents' included, and a fork keeps its parent's. - Live on trogdor, an apply group, a review lens and the summary step sent nothing (the template's own default), `medium` and `low`, and thought 289, 143 and 94 characters on the same question. That check found that opencode fills an entry's missing options from the base entry, which sent the `@high` steps out at `medium`; the base now sends no effort, and `tests/unit/test_effort_entries.py` holds it. ## Verification Fast lane, full lane and all 16 simulated runs pass; the few browser specs and the one real-opencode test that timed out under the machine's load pass alone. The full lane also turned up two flaws in the simulated runs' own stack, fixed in fe45309: a port range taken from the next ephemeral port, and a settings save on the readiness poll's 2 s timeout. Production serves 3fe507f since 08:06:35Z. `osfd doctor` passes, the Playwright cache holds the image's links, and the eight earlier runs kept their numbers, folders and branches. Nine changes are archived (ea9142a); `short-run-names` follows once the next run shows its `runs/<n>/` folder and `osf/<n>-…` branch. Still to do for this issue: the thorough run on a new project, compared with run 62. It stays open until then.
Author
Owner

Comparison run: run 65, stopped after the reviews

Run 65 took run 62's brief on thorough at effort Auto, against a fresh empty repository, with every change of this issue deployed. It is stopped at the user's request after the planning and review phase (16 agent steps, all succeeded), before the apply groups, the last call and the judge. It is kept and can be resumed.

On the 14 agent steps both runs have (explore, proposal, four of the specs, design, tasks, the four reviews):

run 62 run 65
turns 96 77
generated tokens (output and reasoning) 202k 205k
reads of the change's spec files 19 2
permission requests 1 1
wall time 101 min 138 min

What this shows:

  • Effort Auto works. Every turn named its step's own effort: the proposal, specs, design, tasks and revise steps at high, the four reviews and the second review at medium. Run 62 sent everything at the run's one effort.
  • The specs are no longer re-read. In run 62 the tasks step and three of the reviews re-read the change's specs 19 times, because the pasted copy stopped at 8,000 characters. In run 65 the reviews read none, and both reads were in one spec step, hole-content.
  • Fewer turns for the same output. The shared steps took 77 turns against 96, for about the same generated tokens.
  • No mistyped run paths so far. Run 65's one permission request was a read of the openspec CLI's own parser source outside the repository. Run 62's nine typed run ids all came in later steps, so this is not yet a test of that change.
  • Wall time can't be compared. The git-flow baseline run, run 64, shared trogdor with run 65 the whole time. Run 65's proposal also has six capabilities to run 62's four.

Not yet exercised:

  • the last call: whether the turn after it starts without a stall
  • the judge: writing its criteria, and finishing within its limit
  • the spec-test step: one capability at a time, with the scenario count agreeing with the tags
  • the shared Playwright browser cache
  • the refusal message naming the path in an apply step

These are left for the single verification run that will check the remaining changes together. Parallel apply groups moved to #172. This issue stays open until that run has checked them.

## Comparison run: run 65, stopped after the reviews Run 65 took run 62's brief on `thorough` at effort Auto, against a fresh empty repository, with every change of this issue deployed. It is stopped at the user's request after the planning and review phase (16 agent steps, all succeeded), before the apply groups, the last call and the judge. It is kept and can be resumed. On the 14 agent steps both runs have (explore, proposal, four of the specs, design, tasks, the four reviews): | | run 62 | run 65 | |---|---:|---:| | turns | 96 | 77 | | generated tokens (output and reasoning) | 202k | 205k | | reads of the change's spec files | 19 | 2 | | permission requests | 1 | 1 | | wall time | 101 min | 138 min | What this shows: - **Effort Auto works.** Every turn named its step's own effort: the proposal, specs, design, tasks and revise steps at `high`, the four reviews and the second review at `medium`. Run 62 sent everything at the run's one effort. - **The specs are no longer re-read.** In run 62 the tasks step and three of the reviews re-read the change's specs 19 times, because the pasted copy stopped at 8,000 characters. In run 65 the reviews read none, and both reads were in one spec step, `hole-content`. - **Fewer turns for the same output.** The shared steps took 77 turns against 96, for about the same generated tokens. - **No mistyped run paths so far.** Run 65's one permission request was a read of the openspec CLI's own parser source outside the repository. Run 62's nine typed run ids all came in later steps, so this is not yet a test of that change. - **Wall time can't be compared.** The git-flow baseline run, run 64, shared trogdor with run 65 the whole time. Run 65's proposal also has six capabilities to run 62's four. **Not yet exercised:** - the last call: whether the turn after it starts without a stall - the judge: writing its criteria, and finishing within its limit - the spec-test step: one capability at a time, with the scenario count agreeing with the tags - the shared Playwright browser cache - the refusal message naming the path in an apply step These are left for the single verification run that will check the remaining changes together. Parallel apply groups moved to #172. This issue stays open until that run has checked them.
Author
Owner

Run 66: the checks run 65 didn't reach

Run 66 (run_01M52JT93Q9X1VH3GD05D2HMZY) is a thorough run on a new project: a link shortener on Postgres, in a fresh public repository, cmoriarty/braid-combined-test. It ran on production (57f9a21) on trogdor at effort Auto. It succeeded, through the pull request, merge and archive. The #171 session's local quick-fix run shared trogdor from 15:50 to 15:59Z; the timing checks below fall outside that window.

Check from comment 3325 Run 66
write-spec-tests one capability at a time, with the count agreeing with the tags Yes. script.scenarios read "29 of 31 scenarios have a tagged test; 2 have none, in 2 capabilities". The step fanned out over just those 2, one after the other (2.7 and about 21 min). The pull request then reads "All 31 scenarios … have a tagged test".
the judge writes its criteria and finishes within its limit Yes. Round 3's judge (17:16 to 17:26Z) took 9.9 min of its 3 h, with a 48.5k peak context and 2.6k tokens out. It failed one criterion, B4: the Links page isn't readable at 1280×720, a real layout problem.
the shared Playwright browser cache Yes. The project chose Playwright 1.63.0, and the run's own home has no ms-playwright. Even so, npx playwright install chromium downloaded nothing in every round, and the e2e lane ran in 14 s.
the turn after the last call starts without a stall Not exercised. No step came near its time limit, so no osf.step.last_call was sent.
a refused path in an apply step is named in the refusal Not exercised. No agent asked for anything outside the run, so no permission request was refused.

Two of the five, the last call and the refusal, still have only their unit and simulated tests. A real run reaches them only by accident: a step near its limit, an agent reaching outside its folder. This issue can close on the rest, or stay open until a run happens to hit them.

## Run 66: the checks run 65 didn't reach Run 66 (`run_01M52JT93Q9X1VH3GD05D2HMZY`) is a `thorough` run on a new project: a link shortener on Postgres, in a fresh public repository, cmoriarty/braid-combined-test. It ran on production (57f9a21) on trogdor at effort Auto. It succeeded, through the pull request, merge and archive. The #171 session's local quick-fix run shared trogdor from 15:50 to 15:59Z; the timing checks below fall outside that window. | Check from comment 3325 | Run 66 | |---|---| | write-spec-tests one capability at a time, with the count agreeing with the tags | **Yes.** `script.scenarios` read "29 of 31 scenarios have a tagged test; 2 have none, in 2 capabilities". The step fanned out over just those 2, one after the other (2.7 and about 21 min). The pull request then reads "All 31 scenarios … have a tagged test". | | the judge writes its criteria and finishes within its limit | **Yes.** Round 3's judge (17:16 to 17:26Z) took 9.9 min of its 3 h, with a 48.5k peak context and 2.6k tokens out. It failed one criterion, B4: the Links page isn't readable at 1280×720, a real layout problem. | | the shared Playwright browser cache | **Yes.** The project chose Playwright 1.63.0, and the run's own home has no `ms-playwright`. Even so, `npx playwright install chromium` downloaded nothing in every round, and the e2e lane ran in 14 s. | | the turn after the last call starts without a stall | **Not exercised.** No step came near its time limit, so no `osf.step.last_call` was sent. | | a refused path in an apply step is named in the refusal | **Not exercised.** No agent asked for anything outside the run, so no permission request was refused. | Two of the five, the last call and the refusal, still have only their unit and simulated tests. A real run reaches them only by accident: a step near its limit, an agent reaching outside its folder. This issue can close on the rest, or stay open until a run happens to hit them.
Author
Owner

Closing on the evidence of runs 65, 66 and the verification run (run_01M53R7621CJW6PCZG7S6R94HS, thorough on cmoriarty/braid-combined-test, vLLM, succeeded and merged at 03:49Z).

  • Already on this issue: run 65's comparison (comment 3325) and run 66's evidence (comment 3348).
  • In the verification run: effort Auto ran explore, propose, apply and fix at high, and review, the judge and the summary at lower efforts. The judge drove its own browser in round 2 and passed. Every step finished first time, with no stall, time-out or mediation.
  • Not seen in any of these runs: a step reaching its last call, and a path refused outside the worktree. Unit tests cover both. The budget handoff has its own simulated run, handoff-note.

Please reopen if either misbehaves on a real run.

Closing on the evidence of runs 65, 66 and the verification run (`run_01M53R7621CJW6PCZG7S6R94HS`, thorough on cmoriarty/braid-combined-test, vLLM, succeeded and merged at 03:49Z). - Already on this issue: run 65's comparison (comment 3325) and run 66's evidence (comment 3348). - In the verification run: effort Auto ran explore, propose, apply and fix at high, and review, the judge and the summary at lower efforts. The judge drove its own browser in round 2 and passed. Every step finished first time, with no stall, time-out or mediation. - Not seen in any of these runs: a step reaching its last call, and a path refused outside the worktree. Unit tests cover both. The budget handoff has its own simulated run, `handoff-note`. Please reopen if either misbehaves on a real run.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#170
No description provided.