Proposal review: one review step on every backend; remove the four-lens fan-out #353

Closed
opened 2026-11-01 02:15:50 -05:00 by cmoriarty · 2 comments
Owner

What happens

  • Round one of the proposal review has two shapes:
    • four lenses side by side: agent.review fanned out over spec, scope, test and risk;
    • one combined reviewer: agent.review.combined (#275), which answers all four questions in one reading.
  • The run's parallel sessions picks between them, through the one_session and many_sessions conditions. Only a one-session backend gets the single reviewer; every other backend gets four.
  • The four lenses are Braid's own design, not OpenSpec's. The guide gives independent agreement as their point ("Review is four readings, not a vote"). It almost never happened: in the five four-lens runs of braid#350 (161, 163, 164, 165 and 166), the digest merged 2 of 80 findings as found by more than one lens.

What we want
The owner's call (2026-11-01): there should be only one review step.

  • One reviewer on every backend. Round one runs one step that answers the four questions, as agent.review.combined does now, whatever the parallel sessions.
  • The fan-out goes: remove the agent.review fan-out over the lenses and the one_session and many_sessions conditions.
  • Parallel sessions stops shaping the review. It still runs scenario tests side by side (#306).
  • One reading isn't treated as several reviewers. The digest's cross-lens machinery assumed independent readers. With one reader:
    • the revision brief must not say "N reviewers agreed" when one reading wrote a finding into two lens files;
    • findings are not ordered by n_lenses;
    • one lens file doesn't "refute" or "overrule" another.
    • Keeping the four lens files from one reading is fine, as now; folding them into one file is fine too.
  • Unattended approval keeps its safety. Nothing blocking, and the thin-review rule (#198): the reading must show it looked at the change for at least three of the four questions.
  • Round two is unchanged. It is already one recheck step.
  • Old runs still display. That means their four lens steps and their combined steps.

Done when

  • Every new thorough run has one round-one review step, on vLLM (6 sessions) and on a one-slot llama.cpp backend alike.
  • The digest, gate brief and unattended approval work from it, and no finding reads as agreed by several reviewers.
  • The specs, the guide and the sims describe one review step. Old runs open and show their review as before.

Where to look:

  • src/osf/pipeline/nodes.py:
    • _LENS_KEYS and _COMBINED_REVIEW_NODE (~3191–3206);
    • _THOROUGH (~3211–3226);
    • "agent.review": {"when": "many_sessions"} in _THOROUGH_FIELDS (~3316);
    • the when: names (~228–230);
    • the digest node's comment (~1498);
  • src/osf/engine/conditions.py: _sessions, _one_session and _many_sessions (~466–497), and the registry (~997). Only these read OSF_PARALLEL_SESSIONS, which src/osf/engine/scheduler.py sets in _condition_env (~2283) and in the step environment (~2750);
  • src/osf/engine/prompts.py:
    • the review-lens prompt's single-lens and combined paths (~1175–1260, _combined_lens_part ~1332);
    • the per-lens "found by" split (~1545);
    • "N reviewers agreed" in the revision brief (~1897);
  • src/osf/pipeline/review.py:
    • _merge_findings, _refutations and the n_lenses ordering (~1040–1080, ~1285–1305, ~1350);
    • MIN_LENSES_FOR_AUTO and the thin-review rule (~187, ~746–790);
  • src/osf/engine/executors.py: _judge_combined_review (~3176, ~3279), and combined_records in src/osf/engine/looked.py (~408);
  • the lens labels in ui/src/lib/steps.ts (~142);
  • the specs:
    • openspec/specs/proposal-review/spec.md: line 10, "A combined reviewer covers the four lenses on a one-session backend" (~511), and approval (~86);
    • openspec/specs/pipeline-steps/spec.md (~29);
  • docs/guide.md, "Review is four readings, not a vote" (~1833);
  • the sims and tests: sim/plays/default.yaml (~117–128), sim/solo.yaml, and tests/pipeline/test_loader.py (~154).

Expect: on stack 126 the four lenses took 9.7 minutes in run 166 and wrote 64,127 tokens between them. One reviewer at 45 t/s takes as long if it writes about 26,000. The first comment has run 166's numbers.

Related:

  • #275, one pipeline and the combined reviewer;
  • #274, parallel sessions kept per profile;
  • #306, scenario tests side by side;
  • #341, runs that got the combined reviewer by accident;
  • #198, the thin-review rule;
  • #287, merging findings by quote.

Rescoped by the owner on 2026-11-01. The first version asked for a choice between the two shapes and an A/B test. Its table of runs 160 to 166 and the outside evidence are in this issue's edit history.

**What happens** - Round one of the proposal review has two shapes: - **four lenses side by side:** `agent.review` fanned out over `spec`, `scope`, `test` and `risk`; - **one combined reviewer:** `agent.review.combined` (#275), which answers all four questions in one reading. - The run's parallel sessions picks between them, through the `one_session` and `many_sessions` conditions. Only a one-session backend gets the single reviewer; every other backend gets four. - The four lenses are Braid's own design, not OpenSpec's. The guide gives independent agreement as their point ("Review is four readings, not a vote"). It almost never happened: in the five four-lens runs of braid#350 (161, 163, 164, 165 and 166), the digest merged 2 of 80 findings as found by more than one lens. **What we want** The owner's call (2026-11-01): there should be only one review step. - **One reviewer on every backend.** Round one runs one step that answers the four questions, as `agent.review.combined` does now, whatever the parallel sessions. - **The fan-out goes:** remove the `agent.review` fan-out over the lenses and the `one_session` and `many_sessions` conditions. - **Parallel sessions stops shaping the review.** It still runs scenario tests side by side (#306). - **One reading isn't treated as several reviewers.** The digest's cross-lens machinery assumed independent readers. With one reader: - the revision brief must not say "N reviewers agreed" when one reading wrote a finding into two lens files; - findings are not ordered by `n_lenses`; - one lens file doesn't "refute" or "overrule" another. - Keeping the four lens files from one reading is fine, as now; folding them into one file is fine too. - **Unattended approval keeps its safety.** Nothing blocking, and the thin-review rule (#198): the reading must show it looked at the change for at least three of the four questions. - **Round two is unchanged.** It is already one recheck step. - **Old runs still display.** That means their four lens steps and their combined steps. **Done when** - Every new `thorough` run has one round-one review step, on vLLM (6 sessions) and on a one-slot llama.cpp backend alike. - The digest, gate brief and unattended approval work from it, and no finding reads as agreed by several reviewers. - The specs, the guide and the sims describe one review step. Old runs open and show their review as before. **Where to look:** - `src/osf/pipeline/nodes.py`: - `_LENS_KEYS` and `_COMBINED_REVIEW_NODE` (~3191–3206); - `_THOROUGH` (~3211–3226); - `"agent.review": {"when": "many_sessions"}` in `_THOROUGH_FIELDS` (~3316); - the `when:` names (~228–230); - the digest node's comment (~1498); - `src/osf/engine/conditions.py`: `_sessions`, `_one_session` and `_many_sessions` (~466–497), and the registry (~997). Only these read `OSF_PARALLEL_SESSIONS`, which `src/osf/engine/scheduler.py` sets in `_condition_env` (~2283) and in the step environment (~2750); - `src/osf/engine/prompts.py`: - the `review-lens` prompt's single-lens and combined paths (~1175–1260, `_combined_lens_part` ~1332); - the per-lens "found by" split (~1545); - "N reviewers agreed" in the revision brief (~1897); - `src/osf/pipeline/review.py`: - `_merge_findings`, `_refutations` and the `n_lenses` ordering (~1040–1080, ~1285–1305, ~1350); - `MIN_LENSES_FOR_AUTO` and the thin-review rule (~187, ~746–790); - `src/osf/engine/executors.py`: `_judge_combined_review` (~3176, ~3279), and `combined_records` in `src/osf/engine/looked.py` (~408); - the lens labels in `ui/src/lib/steps.ts` (~142); - the specs: - `openspec/specs/proposal-review/spec.md`: line 10, "A combined reviewer covers the four lenses on a one-session backend" (~511), and approval (~86); - `openspec/specs/pipeline-steps/spec.md` (~29); - `docs/guide.md`, "Review is four readings, not a vote" (~1833); - the sims and tests: `sim/plays/default.yaml` (~117–128), `sim/solo.yaml`, and `tests/pipeline/test_loader.py` (~154). **Expect:** on stack 126 the four lenses took 9.7 minutes in run 166 and wrote 64,127 tokens between them. One reviewer at 45 t/s takes as long if it writes about 26,000. The first comment has run 166's numbers. Related: - #275, one pipeline and the combined reviewer; - #274, parallel sessions kept per profile; - #306, scenario tests side by side; - #341, runs that got the combined reviewer by accident; - #198, the thin-review rule; - #287, merging findings by quote. *Rescoped by the owner on 2026-11-01. The first version asked for a choice between the two shapes and an A/B test. Its table of runs 160 to 166 and the outside evidence are in this issue's edit history.*
Author
Owner

Four-lens baseline: run 166, round one (stack 126, arm R, the four lenses side by side; measured from the run's transcripts)

  • Time: 9.7 min wall. Each lens took 6.5 to 9.7 min: spec 8.6, scope 6.5, test 8.5, risk 9.7.
  • Tokens: 64,127 generated in all, 55,290 of them reasoning, so about 16,000 per lens. Each lens's prompt peaked at 28k to 34k tokens.
  • Speed: with the four batched, each lens decoded at about 32 t/s, about 128 t/s together. One stream alone runs at 45 t/s.
  • What the combined reviewer must beat: running alone at 45 t/s, it matches 9.7 min only if it writes about 26,000 tokens or fewer (ignoring prefill). That is about 1.6 times one lens's average. If it writes more, the four lenses are faster on this backend.
  • Findings: 6 to fix and 9 notes, none merged as shared:
    • spec: 4 notes;
    • scope: 3 to fix, 2 notes;
    • test: 3 to fix, 1 note;
    • risk: 2 notes.
  • After the review: the revision took 5.1 min, and the gate approved unattended.
  • For comparison: in run 163 (the 4-bit model, also four lenses), round one took 11.4 min wall.
**Four-lens baseline: run 166, round one** (stack 126, arm R, the four lenses side by side; measured from the run's transcripts) - **Time:** 9.7 min wall. Each lens took 6.5 to 9.7 min: spec 8.6, scope 6.5, test 8.5, risk 9.7. - **Tokens:** 64,127 generated in all, 55,290 of them reasoning, so about 16,000 per lens. Each lens's prompt peaked at 28k to 34k tokens. - **Speed:** with the four batched, each lens decoded at about 32 t/s, about 128 t/s together. One stream alone runs at 45 t/s. - **What the combined reviewer must beat:** running alone at 45 t/s, it matches 9.7 min only if it writes about 26,000 tokens or fewer (ignoring prefill). That is about 1.6 times one lens's average. If it writes more, the four lenses are faster on this backend. - **Findings:** 6 to fix and 9 notes, none merged as shared: - spec: 4 notes; - scope: 3 to fix, 2 notes; - test: 3 to fix, 1 note; - risk: 2 notes. - **After the review:** the revision took 5.1 min, and the gate approved unattended. - **For comparison:** in run 163 (the 4-bit model, also four lenses), round one took 11.4 min wall.
cmoriarty changed title from Proposal review: let a run choose four lenses or one combined reviewer, so the two can be compared to Proposal review: one review step on every backend; remove the four-lens fan-out 2026-11-01 02:31:29 -05:00
Author
Owner

Done in batch A, merged as #356 (c00c62f1), and live on production: Braid reports commit c00c62f, which includes it. It was checked with the repo's tests and sims before the merge. A live check on vLLM is still to come: trogdor is off the network. Closed at the owner's request.

Done in batch A, merged as #356 (`c00c62f1`), and live on production: Braid reports commit `c00c62f`, which includes it. It was checked with the repo's tests and sims before the merge. A live check on vLLM is still to come: trogdor is off the network. Closed at the owner's request.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#353
No description provided.