Proposal review: act on the findings, or run fewer reviewers #256

Closed
opened 2026-10-27 15:30:58 -04:00 by cmoriarty · 1 comment
Owner

What happens
Four reviewers read every proposal. In 9 runs they raised 167 findings: 1 blocking, 124 should-fix and 42 notes. Only the blocking one changed anything (a real bug, in run 115). The should-fix findings were dropped.

What we want (one of)

  • One revise pass that applies the should-fix findings, or
  • One or two reviewers instead of four.

Done when
Review findings either change the proposal, or stop being made.

From the Braid vs Claude Code pinball test, 2026-10-27 (runs 110–120).

**What happens** Four reviewers read every proposal. In 9 runs they raised 167 findings: 1 blocking, 124 should-fix and 42 notes. Only the blocking one changed anything (a real bug, in run 115). The should-fix findings were dropped. **What we want** (one of) - One revise pass that applies the should-fix findings, or - One or two reviewers instead of four. **Done when** Review findings either change the proposal, or stop being made. _From the Braid vs Claude Code pinball test, 2026-10-27 (runs 110–120)._
Author
Owner

Correction first
Should-fix findings were not dropped: up to eight already reach the apply and test steps as findings that do not block. But those steps may not edit the change's documents, so findings about a spec (like "about 60 fps" with no tolerance) were fixed nowhere.

What shipped

  • When round 1 has nothing blocking but should-fix findings on the change's own documents, agent.revise revises those once, and the gate follows without another round of lenses.

Test
tests/pipeline/test_review.py.

Shipped in 5598ddf (OpenSpec change auto-approve-reviews-and-gate-words, archived).

**Correction first** Should-fix findings were not dropped: up to eight already reach the apply and test steps as findings that do not block. But those steps may not edit the change's documents, so findings about a spec (like "about 60 fps" with no tolerance) were fixed nowhere. **What shipped** - When round 1 has nothing blocking but should-fix findings on the change's own documents, `agent.revise` revises those once, and the gate follows without another round of lenses. **Test** `tests/pipeline/test_review.py`. _Shipped in 5598ddf (OpenSpec change `auto-approve-reviews-and-gate-words`, archived)._
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#256
No description provided.