Apply groups run one at a time while trogdor could serve three: run the independent ones side by side, each in a worktree of its own #172

Closed
opened 2026-10-16 03:03:48 -04:00 by cmoriarty · 1 comment
Owner

Split off from #170 (task 10, its second part), and part of #128. To follow #171: it needs opencode 2's per-session environment, below.

Why

Run 62 spent 376 minutes applying eight groups on one stream, while trogdor serves a second and third stream at nearly the same speed (47 tok/s alone, 47 with one more, 41–43 with two or three). With #170, tasks.md says which group depends on which (Depends on: lines, recorded in .osf/budget.json): in run 62, physics (3) and the renderer (5) needed only groups 1 and 2, and could have run side by side.

They cannot just start together in the run's worktree. That was tried: group 2 found group 1's work missing and redid it in the same files, and a rollback of one takes the other's files with it.

The plan

The OpenSpec change is written and validates: openspec/changes/parallel-apply-groups/ (proposal, design, specs, tasks).

  • A switch, off by default: Settings → Runs → "Apply groups in parallel", kept per run.
  • On, a group starts once its dependencies are merged, in a worktree and on a branch of its own (runs/<n>/groups/<key>/), cut from the run's worktree as it is then. Its session, checks and checkpoints use that worktree, so a redo rolls back only it.
  • When it finishes, Braid merges it into the run's worktree: a three-way merge from the state it started from, with tasks.md ticks merged line by line and Braid's .osf/ records kept from the run unless only the group wrote one.
  • A conflict goes back to the group's own session to resolve, as a branch is brought up to date with its base; after three tries, or when it cannot, the step fails naming the files, for a person. The run's worktree never sees a marker.
  • A redo of a merged group takes its merge back out first. Concurrency stays bounded by the box's slots.

Measured on opencode 1.18.35 (2026-10-16)

One server started in the run's worktree, a session created in a linked worktree:

  • the session lives in the linked worktree: its shell ran there, and GET /session/{id} names it;
  • its busy entry is only in GET /session/status?directory=<its worktree>. Asked with the run's worktree, as step_complete, the watchdog and the question poll ask today, the map is empty while it works: a group step would be taken as finished at its first quiet moment;
  • its events are only on GET /event?directory=<its worktree>, so each group needs a firehose of its own;
  • a fork asked for in the run's worktree stays in the group's, so a fix round forking a group's session after that worktree is removed has to take the fresh-session fallback.

And git merge-tree on the production image (Debian bookworm, git 2.39.5) has --write-tree but not --merge-base (git 2.40): the merge gives both snapshot commits the group's base as their parent instead.

Why after #171

PREVIEW_PORT is in the run's opencode server environment, so every session of a run, every shell command in it and the app tool share one port. Two groups at once that each serve the app or run browser tests would take the same port, and most runs are web apps. opencode 1.x has no per-session environment. opencode 2 has PUT /api/session/{id}/environment, which sets the environment of a session's shell commands, so each group can have a port of its own there. The directory-scoped reads above are also exactly the API #171 rewrites.

Done when

  • The change's tasks are done on opencode 2, with the port per group.
  • Two simulated runs pass: two independent groups that overlap in time and merge cleanly, and two that conflict on one line and resolve.
  • One live run on trogdor with the switch on, on a small project with two independent groups, records the overlap and the merge here.
  • The switch stays off until the operator turns it on.
Split off from #170 (task 10, its second part), and part of #128. To follow #171: it needs opencode 2's per-session environment, below. ## Why Run 62 spent 376 minutes applying eight groups on one stream, while trogdor serves a second and third stream at nearly the same speed (47 tok/s alone, 47 with one more, 41–43 with two or three). With #170, `tasks.md` says which group depends on which (`Depends on:` lines, recorded in `.osf/budget.json`): in run 62, physics (3) and the renderer (5) needed only groups 1 and 2, and could have run side by side. They cannot just start together in the run's worktree. That was tried: group 2 found group 1's work missing and redid it in the same files, and a rollback of one takes the other's files with it. ## The plan The OpenSpec change is written and validates: `openspec/changes/parallel-apply-groups/` (proposal, design, specs, tasks). - A switch, off by default: Settings → Runs → "Apply groups in parallel", kept per run. - On, a group starts once its dependencies are merged, in a worktree and on a branch of its own (`runs/<n>/groups/<key>/`), cut from the run's worktree as it is then. Its session, checks and checkpoints use that worktree, so a redo rolls back only it. - When it finishes, Braid merges it into the run's worktree: a three-way merge from the state it started from, with `tasks.md` ticks merged line by line and Braid's `.osf/` records kept from the run unless only the group wrote one. - A conflict goes back to the group's own session to resolve, as a branch is brought up to date with its base; after three tries, or when it cannot, the step fails naming the files, for a person. The run's worktree never sees a marker. - A redo of a merged group takes its merge back out first. Concurrency stays bounded by the box's slots. ## Measured on opencode 1.18.35 (2026-10-16) One server started in the run's worktree, a session created in a linked worktree: - the session lives in the linked worktree: its shell ran there, and `GET /session/{id}` names it; - its busy entry is only in `GET /session/status?directory=<its worktree>`. Asked with the run's worktree, as `step_complete`, the watchdog and the question poll ask today, the map is empty while it works: a group step would be taken as finished at its first quiet moment; - its events are only on `GET /event?directory=<its worktree>`, so each group needs a firehose of its own; - a fork asked for in the run's worktree stays in the group's, so a fix round forking a group's session after that worktree is removed has to take the fresh-session fallback. And `git merge-tree` on the production image (Debian bookworm, git 2.39.5) has `--write-tree` but not `--merge-base` (git 2.40): the merge gives both snapshot commits the group's base as their parent instead. ## Why after #171 `PREVIEW_PORT` is in the run's opencode server environment, so every session of a run, every shell command in it and the `app` tool share one port. Two groups at once that each serve the app or run browser tests would take the same port, and most runs are web apps. opencode 1.x has no per-session environment. opencode 2 has `PUT /api/session/{id}/environment`, which sets the environment of a session's shell commands, so each group can have a port of its own there. The directory-scoped reads above are also exactly the API #171 rewrites. ## Done when - The change's tasks are done on opencode 2, with the port per group. - Two simulated runs pass: two independent groups that overlap in time and merge cleanly, and two that conflict on one line and resolve. - One live run on trogdor with the switch on, on a small project with two independent groups, records the overlap and the merge here. - The switch stays off until the operator turns it on.
Author
Owner

Shipped in three deploys on 2026-10-17: part A in 2cc4a20 (08:56Z, migration 15, backed up first), part B in 6dda289 (11:27Z), and Auto in fdd6c70 (23:28Z). Archived in OpenSpec as parallel-apply-groups in 874eb27.

What shipped

  1. A run from an approved change (fbbd046): POST /api/runs takes change. Admission checks that the change is on the base branch, and cuts the explore, propose and review steps out of the run's graph. The run's change is a column (migration 15).
  2. One step (a163182, 50c7be5; review fixes 55d09ca, acc1426, b2971aa, bc803b0): openspec.apply[all] does every task in one session and compacts in place at its alarm. It has one deadline (the node's limit times the groups) and a progress bound: two compactions in a row with no task ticked fail it.
  3. Groups side by side (part B, merge 0aafc75): each group works in a worktree and branch of its own, at most three at once, each with its own port, compose project and app. A group is merged back with git merge-tree once its checks hold, with tasks.md ticks merged three-way. A conflict goes to the group's own session, three tries, then to a person. It is labelled experimental.
  4. Auto, the default (5d9ede4, 89c7503, merge d0ab8a6): one step when a session's context is at least 200,000, group by group below that. One step and group by group can be forced; side by side is used only when chosen, never by Auto. The run bar says which was chosen.
  5. A simulated run for each: from-change, from-change-missing, apply-single, apply-single-no-progress, parallel-groups, parallel-conflict, apply-auto-small-context and apply-auto-plentiful.

The comparison. The subject was scale-recipes-and-shopping-list on the combined test repository: five groups, with 2 to 4 depending on 1 and 5 on 2 to 4. Each arm was a thorough run on a base branch of its own, with its own auto-approve off; the lead's watcher answered its merge and archive gates. The context per agent, as the runs' records give it: 400,000 for the groups arm and both parallel arms (their apply steps' budget allowance of 394,000), and 90,000 for run 73 (its alarms at 67,500). The single arm's records do not carry its context per agent.

arm run apply apply turns apply prompt tokens fix rounds judge
groups (run 69) 73.3 min 47.8 min 98 5.53M 0 6 min
single (run 70) 73.2 min 37.6 min 50 5.37M 0 18.5 min
parallel, first (run 71) 78.2 min 38.75 min 74 4.62M 1 9.7 + 11.8 min
parallel, second (run 72) 66.7 min 40.2 min 88 4.78M 0 11.1 min
single at 90,000 (run 73) 109.9 min 53.9 min 86 3.81M 1 30.1 min
  • The single arm's context peaked at 144,983, and it never compacted.
  • In the first parallel arm, group 1's session did all five groups' tasks, against its brief. Groups 2 to 4 had nothing left to do, so this arm measured no parallelism. In the second, groups 2 to 4 ran side by side for 12.0 min. In the groups arm the same three took 20.7 min one after another.
  • Run 73 ran one step at 90,000 (alarm 67,500) and compacted three times in one session: 69,383 → 9,142 with 14 tasks left, then 71,779 → 10,437 and 73,413 → 10,082 with 4 each. It used the fewest apply tokens of any arm, and its apply took the longest.
  • Every apply step of every arm ran in one attempt, with no mediated retry, and in every arm every task was ticked and every scenario had a tagged test before review.

A blind review scored the four diffs, labelled P to S, and the labels were revealed after scoring. Groups 4.38, single 4.00, first parallel 3.38, second parallel 3.25 (out of 5). All 47 scenarios pass in all four, and none has a blocking code-review finding. The second parallel run lost points on four things: 40 ties rounded wrong, the servings read with Number(), the shopping list computed in the view, and a Servings box 224 px tall that its judge missed.

The decision rule counted the second parallel run as the parallel arm:

  • No arm was worse in quality than another.
  • Parallel was only 8.9% faster than the next arm, and the rule needs 25%.
  • Single was recommended: 9.7% slower than the fastest arm, inside the rule's 15%.

The result was close, though: 10% more wall clock for single would flip the recommendation to groups. So the rule called for one more run of each of those two arms. That repeat was not run. On 2026-10-17 the operator turned the choice into a rule: Auto is the default, and applies in one step at 200k of context or more, else group by group.

Checked on production

  • Run 75 (fdd6c70) recorded apply_choice as {mode: single, setting: auto, context: 400000, one_step_at: 200000}, and its run bar read "thorough · auto: one step (400k context)".
  • Run 74 (86dc37f) used run 73's settings with #185 and #186 in: 79.3 min against 109.9, 5.98M prompt tokens against 10.75M, and no respawn. #186 gets the details once its change is archived.

Not done

  • Timing compactions with POST /compact before the arms (task 10.3). Runs 73 and 74 measured the apply step's compactions instead: from 2 min 1 s to 2 min 50 s each, from the alarm to the compacted record.
  • The readings each arm was to take at the time: nothing in flight on /api/slots, and /healthz and /api/status at its start and end. The steps' claims show each arm stayed on one osfd boot.
  • Some of the measures the design listed: generated tokens, the fresh and cached split, requests in flight, and the code review's findings by severity.
  • Parts of the blind review's method are not on record: the reviewer's model, the diffs' exact bounds, how the labels were assigned, and a count of undone ticks per diff. One of its four lenses was the browser, not one of the four criteria the task named.
  • The second run of the two leading arms, which the operator's decision replaced.
  • Not tested: in the first parallel arm, the judge used the question tool to say it was waiting for its browser subagent, and the arm's watcher answered it. Nobody checked whether auto-approve answers an agent's question when no one is watching.

Part B's merge had also dropped three blank lines in src/osf/engine/scheduler.py; 4ad5b90 restores them.

Auto is now the default. The operator can change it in Settings → Apply the change. Closing.

Shipped in three deploys on 2026-10-17: part A in 2cc4a20 (08:56Z, migration 15, backed up first), part B in 6dda289 (11:27Z), and Auto in fdd6c70 (23:28Z). Archived in OpenSpec as `parallel-apply-groups` in 874eb27. **What shipped** 1. **A run from an approved change** (fbbd046): `POST /api/runs` takes `change`. Admission checks that the change is on the base branch, and cuts the explore, propose and review steps out of the run's graph. The run's `change` is a column (migration 15). 2. **One step** (a163182, 50c7be5; review fixes 55d09ca, acc1426, b2971aa, bc803b0): `openspec.apply[all]` does every task in one session and compacts in place at its alarm. It has one deadline (the node's limit times the groups) and a progress bound: two compactions in a row with no task ticked fail it. 3. **Groups side by side** (part B, merge 0aafc75): each group works in a worktree and branch of its own, at most three at once, each with its own port, compose project and app. A group is merged back with `git merge-tree` once its checks hold, with `tasks.md` ticks merged three-way. A conflict goes to the group's own session, three tries, then to a person. It is labelled experimental. 4. **Auto, the default** (5d9ede4, 89c7503, merge d0ab8a6): one step when a session's context is at least 200,000, group by group below that. One step and group by group can be forced; side by side is used only when chosen, never by Auto. The run bar says which was chosen. 5. A simulated run for each: from-change, from-change-missing, apply-single, apply-single-no-progress, parallel-groups, parallel-conflict, apply-auto-small-context and apply-auto-plentiful. **The comparison.** The subject was `scale-recipes-and-shopping-list` on the combined test repository: five groups, with 2 to 4 depending on 1 and 5 on 2 to 4. Each arm was a thorough run on a base branch of its own, with its own auto-approve off; the lead's watcher answered its merge and archive gates. The context per agent, as the runs' records give it: 400,000 for the groups arm and both parallel arms (their apply steps' budget allowance of 394,000), and 90,000 for run 73 (its alarms at 67,500). The single arm's records do not carry its context per agent. | arm | run | apply | apply turns | apply prompt tokens | fix rounds | judge | |---|---|---|---|---|---|---| | groups (run 69) | 73.3 min | 47.8 min | 98 | 5.53M | 0 | 6 min | | single (run 70) | 73.2 min | 37.6 min | 50 | 5.37M | 0 | 18.5 min | | parallel, first (run 71) | 78.2 min | 38.75 min | 74 | 4.62M | 1 | 9.7 + 11.8 min | | parallel, second (run 72) | 66.7 min | 40.2 min | 88 | 4.78M | 0 | 11.1 min | | single at 90,000 (run 73) | 109.9 min | 53.9 min | 86 | 3.81M | 1 | 30.1 min | - The single arm's context peaked at 144,983, and it never compacted. - In the first parallel arm, group 1's session did all five groups' tasks, against its brief. Groups 2 to 4 had nothing left to do, so this arm measured no parallelism. In the second, groups 2 to 4 ran side by side for 12.0 min. In the groups arm the same three took 20.7 min one after another. - Run 73 ran one step at 90,000 (alarm 67,500) and compacted three times in one session: 69,383 → 9,142 with 14 tasks left, then 71,779 → 10,437 and 73,413 → 10,082 with 4 each. It used the fewest apply tokens of any arm, and its apply took the longest. - Every apply step of every arm ran in one attempt, with no mediated retry, and in every arm every task was ticked and every scenario had a tagged test before review. A blind review scored the four diffs, labelled P to S, and the labels were revealed after scoring. Groups 4.38, single 4.00, first parallel 3.38, second parallel 3.25 (out of 5). All 47 scenarios pass in all four, and none has a blocking code-review finding. The second parallel run lost points on four things: 40 ties rounded wrong, the servings read with `Number()`, the shopping list computed in the view, and a Servings box 224 px tall that its judge missed. The decision rule counted the second parallel run as the parallel arm: - No arm was worse in quality than another. - Parallel was only 8.9% faster than the next arm, and the rule needs 25%. - **Single was recommended**: 9.7% slower than the fastest arm, inside the rule's 15%. The result was close, though: 10% more wall clock for single would flip the recommendation to groups. So the rule called for one more run of each of those two arms. That repeat was not run. On 2026-10-17 the operator turned the choice into a rule: Auto is the default, and applies in one step at 200k of context or more, else group by group. **Checked on production** - Run 75 (fdd6c70) recorded `apply_choice` as `{mode: single, setting: auto, context: 400000, one_step_at: 200000}`, and its run bar read "thorough · auto: one step (400k context)". - Run 74 (86dc37f) used run 73's settings with #185 and #186 in: 79.3 min against 109.9, 5.98M prompt tokens against 10.75M, and no respawn. #186 gets the details once its change is archived. **Not done** - Timing compactions with `POST /compact` before the arms (task 10.3). Runs 73 and 74 measured the apply step's compactions instead: from 2 min 1 s to 2 min 50 s each, from the alarm to the compacted record. - The readings each arm was to take at the time: nothing in flight on `/api/slots`, and `/healthz` and `/api/status` at its start and end. The steps' claims show each arm stayed on one osfd boot. - Some of the measures the design listed: generated tokens, the fresh and cached split, requests in flight, and the code review's findings by severity. - Parts of the blind review's method are not on record: the reviewer's model, the diffs' exact bounds, how the labels were assigned, and a count of undone ticks per diff. One of its four lenses was the browser, not one of the four criteria the task named. - The second run of the two leading arms, which the operator's decision replaced. - Not tested: in the first parallel arm, the judge used the `question` tool to say it was waiting for its browser subagent, and the arm's watcher answered it. Nobody checked whether auto-approve answers an agent's question when no one is watching. Part B's merge had also dropped three blank lines in `src/osf/engine/scheduler.py`; 4ad5b90 restores them. Auto is now the default. The operator can change it in Settings → Apply the change. Closing.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#172
No description provided.