Auto-approve holds every Sonnet proposal review for a person #244

Closed
opened 2026-10-27 11:38:02 -04:00 by cmoriarty · 1 comment
Owner

What happens
With auto-approve on, human.review-proposal still waits for a person. The log says the reviewers "answered in under a minute without reading the change".

Why that's wrong

  • The reviewers get the whole change in their prompt, so there's nothing left to read.
  • They did review it: 11 to 22 specific findings per run.
  • The 60-second limit was set for slow local models. On Sonnet, a careful review takes 15–30 s, so every review is held.

Cost
38.8 min of waiting in the pinball test.

Also
While it's held, the console labels it agent.review-proposal and shows "waiting on you". A gate held for a person should show as a human step, with the reason.

Fix

  • Don't flag quick_no_reads when the change was in the prompt. Keep the guard for a reviewer with no findings and no reasoning.
  • Make the label follow the hold.

Done when
With auto-approve on, a fast Sonnet review with real findings is approved on its own.

Where to look: auto_hold and lens_flags in src/osf/pipeline/review.py.

From the Braid vs Claude Code pinball test, 2026-10-27 (runs 110–120).

**What happens** With auto-approve on, `human.review-proposal` still waits for a person. The log says the reviewers "answered in under a minute without reading the change". **Why that's wrong** - The reviewers get the whole change in their prompt, so there's nothing left to read. - They did review it: 11 to 22 specific findings per run. - The 60-second limit was set for slow local models. On Sonnet, a careful review takes 15–30 s, so every review is held. **Cost** 38.8 min of waiting in the pinball test. **Also** While it's held, the console labels it `agent.review-proposal` and shows "waiting on you". A gate held for a person should show as a human step, with the reason. **Fix** - Don't flag `quick_no_reads` when the change was in the prompt. Keep the guard for a reviewer with no findings and no reasoning. - Make the label follow the hold. **Done when** With auto-approve on, a fast Sonnet review with real findings is approved on its own. Where to look: `auto_hold` and `lens_flags` in `src/osf/pipeline/review.py`. _From the Braid vs Claude Code pinball test, 2026-10-27 (runs 110–120)._
cmoriarty changed title from Auto-approve holds Sonnet's proposal reviews: lenses served the change are flagged as not reading it to Auto-approve holds every Sonnet proposal review for a person 2026-10-27 15:31:33 -04:00
Author
Owner

What shipped

  • A fast lens with no reads holds the review only when it also shows no findings and no reasoning. Sonnet's quick lenses with findings no longer hold auto-approve.
  • When auto-approve does leave a gate for a person, it records osf.gate.held with the reason (schema 18). The console then names the step human., and the gate card (or a review's decision panel) says why.

Tests
tests/pipeline/test_review.py, tests/engine/test_auto_approve.py, ui/src/lib/gateText.test.ts.

Shipped in 5598ddf (OpenSpec change auto-approve-reviews-and-gate-words, archived).

**What shipped** - A fast lens with no reads holds the review only when it also shows no findings and no reasoning. Sonnet's quick lenses with findings no longer hold auto-approve. - When auto-approve does leave a gate for a person, it records `osf.gate.held` with the reason (schema 18). The console then names the step `human.`, and the gate card (or a review's decision panel) says why. **Tests** `tests/pipeline/test_review.py`, `tests/engine/test_auto_approve.py`, `ui/src/lib/gateText.test.ts`. _Shipped in 5598ddf (OpenSpec change `auto-approve-reviews-and-gate-words`, archived)._
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#244
No description provided.