Brief hygiene from run 172: the 68k-char apply brief, late task ticks, the judge's unclear contradiction, each judge writing its own validator, reviewers re-reading pasted text #386

Closed
opened 2026-11-04 17:31:20 -05:00 by cmoriarty · 1 comment
Owner

What happens

  • The apply brief is 68k chars (~81k tokens), so the session started at 81k of 226k and alarmed after 27 min of real work.
  • Tasks 3.x were finished at 08:44 and ticked at 10:11:57 (87 min late), so the compaction summary said "functionally done but not ticked" and the handoff was wrong.
  • The judge brief says both "fail and unclear fail the round" and "the verdict is fail when any criterion failed ... pass otherwise"; the r3 judge spent ~7k reasoning tokens reconciling them. Each judge round wrote its own node -e schema validator (3 times).
  • The apply step placed all 69 // spec: tags inside the test bodies instead of above the it; rule B6 failed and 2 min went to moving them: the rule has no literal example.
  • agent.review.combined re-read spec files that were pasted into its brief ("they were pasted, but just...") and rewrote spec.json twice; agent.code-review re-read 22 test files the write-spec-tests lint had already passed (1.8 min) and simulated the README and Dockerfile by hand (1.2 min).
  • 37 of 53 apply reads had no range despite "read narrowly"; the brief forbids $TMPDIR scripts, which pushed the apply agent's harness into the repository.

Fix

  • Move the delta specs and the long rule sections to files the agent reads on demand (.osf/brief/specs.md, .osf/brief/rules.md), keeping the brief to the task and the pointers.
  • Tie ticking to the green test report: when the report names the tasks' tests, the step ticks them in the same turn.
  • Fix the unclear wording; ship the judge-report validator as a tool or a scripted check.
  • Give B6 one literal example: // spec: cap/scenario directly above it(.
  • Review brief: "the pasted change text is authoritative; write each lens once." Code-review brief: skip test files the lint passed.

Done when
The apply session starts under 30k tokens of brief, ticks follow green runs, and a judge round has one verdict rule and no validator of its own. ~25 min on run 172.

From the run 172 transcript review (Scratch on vLLM + MTP k=4, 2026-11-04; report in the owner's ~/repos/vLLM/reports/). Related: braid#378, braid#379.

**What happens** - The apply brief is 68k chars (~81k tokens), so the session started at 81k of 226k and alarmed after 27 min of real work. - Tasks 3.x were finished at 08:44 and ticked at 10:11:57 (87 min late), so the compaction summary said "functionally done but not ticked" and the handoff was wrong. - The judge brief says both "`fail` and `unclear` fail the round" and "the verdict is `fail` when any criterion failed ... pass otherwise"; the r3 judge spent ~7k reasoning tokens reconciling them. Each judge round wrote its own `node -e` schema validator (3 times). - The apply step placed all 69 `// spec:` tags inside the test bodies instead of above the `it`; rule B6 failed and 2 min went to moving them: the rule has no literal example. - `agent.review.combined` re-read spec files that were pasted into its brief ("they were pasted, but just...") and rewrote `spec.json` twice; `agent.code-review` re-read 22 test files the write-spec-tests lint had already passed (1.8 min) and simulated the README and Dockerfile by hand (1.2 min). - 37 of 53 apply reads had no range despite "read narrowly"; the brief forbids `$TMPDIR` scripts, which pushed the apply agent's harness into the repository. **Fix** - Move the delta specs and the long rule sections to files the agent reads on demand (`.osf/brief/specs.md`, `.osf/brief/rules.md`), keeping the brief to the task and the pointers. - Tie ticking to the green `test` report: when the report names the tasks' tests, the step ticks them in the same turn. - Fix the `unclear` wording; ship the judge-report validator as a tool or a scripted check. - Give B6 one literal example: `// spec: cap/scenario` directly above `it(`. - Review brief: "the pasted change text is authoritative; write each lens once." Code-review brief: skip test files the lint passed. **Done when** The apply session starts under 30k tokens of brief, ticks follow green runs, and a judge round has one verdict rule and no validator of its own. ~25 min on run 172. _From the run 172 transcript review (Scratch on vLLM + MTP k=4, 2026-11-04; report in the owner's `~/repos/vLLM/reports/`). Related: braid#378, braid#379._
Author
Owner

Shipped in PR #390, production ddd4b6b.

  • One verdict rule.
  • fixloop check-report as a numbered step of ending, so no judge writes its own validator.
  • carried_from shown as a number, a literal example for B6, ticks on a green test report, and fewer second reads.

Dropped after measuring: moving the delta specs out of the brief. The 81k that apply started at was mostly inherited fork history: its fresh session started at 25.9k, brief included. The move would undo #170. The fork rule in #383 deals with the 81k.

Evidence: tests/pipeline/test_check_report.py. In a Free replay of round 3, the check caught a false carry before the report was final. agent.code-review took 32.7 minutes and agent.review.combined 1.8.

Did not clearly hold: agent.review.combined still made a batch of reads on that model, so the line on second reads is unproved.

**Shipped** in PR #390, production ddd4b6b. - One verdict rule. - `fixloop check-report` as a numbered step of ending, so no judge writes its own validator. - `carried_from` shown as a number, a literal example for B6, ticks on a green test report, and fewer second reads. **Dropped after measuring: moving the delta specs out of the brief.** The 81k that apply started at was mostly inherited fork history: its fresh session started at 25.9k, brief included. The move would undo #170. The fork rule in #383 deals with the 81k. **Evidence:** `tests/pipeline/test_check_report.py`. In a Free replay of round 3, the check caught a false carry before the report was final. `agent.code-review` took 32.7 minutes and `agent.review.combined` 1.8. **Did not clearly hold:** `agent.review.combined` still made a batch of reads on that model, so the line on second reads is unproved.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#386
No description provided.