Briefs held to the rules, and the rules a guard holds (#443 #430) #446

Merged
cmoriarty merged 2 commits from brief-audit-443 into main 2026-11-08 22:47:10 -05:00
Owner

Closes the work of braid#443 (the audit of every brief, with its research) and the reopened braid#430.

What the audit found

Every brief of runs 174 to 183, read as the agent received it: five opened on a delegation block most of them never used, four ended on this machine's frame rate, the whole change was pasted into five steps, and run 183's round 3 Visual check was 65,929 characters with its "look at 6, carry 53" 21,500 characters into one section. The audit and the research are in docs/design/measurements/.

Briefs (briefs-held-to-the-rules)

On a worktree shaped like run 183's, in characters:

brief before after budget
propose 10,033 9,511 10,000
combined review 45,955 46,031 50,000
revise 42,712 5,479 12,000
the operator's revise 68,755 1,787 2,000
apply 53,329 46,455 50,000
Visual check, first look 53,351 34,816 35,000
Visual check, a later round 59,682 12,234 15,000
summary 6,167 2,392 8,000
  • Every brief opens with its task and ends on how the step ends.
  • The delegation section goes whole to the steps that delegate, and is one line elsewhere. "This machine" sits with the environment facts.
  • What a step reads whole is pasted; the rest is a path.
  • A later round's Visual check has a brief of its own.
  • The planner the thorough pipeline uses is sent the tasks rules with their example group (braid#430).
  • Whole-file examples of visual.json and untested.json.
  • A budget per brief, tested as sent, for all 25 builders. The ceiling on source text is gone.
  • python -m osf.sim --briefs <dir> writes every brief a simulated run sent.

Guards and checks (rules-a-guard-holds)

  • A command that runs $OSF_PYTHON and names Braid's own package is refused, the briefs' commands excepted.
  • A Visual check's script may not import the app's modules or link them out of the worktree.
  • A look is refused once 12 have passed with no criterion added to the report.
  • unclear with cannot_stage fails the round and becomes a test the fix round writes; a passing test takes it off the next look list.
  • The message after a compaction tells a Visual check what it has left.
  • B9, recorded: a test that itself calls an unseeded random source, shown to a fix round on a red lane.

Checked

  • ./tools/test.sh passes.
  • A scripted replay of run 183's first-look Visual check against these guards: 10 writes and links of the solver refused, 7 calls into Braid's package refused, 23 looks refused after the twelfth with an empty report. The replay found two faults in the first version of the guards, fixed here.
  • Sims: the new judge-cannot-stage passes. The default play's report said play where the schema asks for session, which failed seven sims on main; fixed. Two more fail on main for a reason of their own (braid#445).

Not done: replays on real models before and after for each brief. The next Scratch run is the comparison with run 183.

🤖 Generated with Claude Code

Closes the work of braid#443 (the audit of every brief, with its research) and the reopened braid#430. ## What the audit found Every brief of runs 174 to 183, read as the agent received it: five opened on a delegation block most of them never used, four ended on this machine's frame rate, the whole change was pasted into five steps, and run 183's round 3 Visual check was 65,929 characters with its "look at 6, carry 53" 21,500 characters into one section. The audit and the research are in `docs/design/measurements/`. ## Briefs (`briefs-held-to-the-rules`) On a worktree shaped like run 183's, in characters: | brief | before | after | budget | |---|---:|---:|---:| | propose | 10,033 | 9,511 | 10,000 | | combined review | 45,955 | 46,031 | 50,000 | | revise | 42,712 | 5,479 | 12,000 | | the operator's revise | 68,755 | 1,787 | 2,000 | | apply | 53,329 | 46,455 | 50,000 | | Visual check, first look | 53,351 | 34,816 | 35,000 | | Visual check, a later round | 59,682 | 12,234 | 15,000 | | summary | 6,167 | 2,392 | 8,000 | - Every brief opens with its task and ends on how the step ends. - The delegation section goes whole to the steps that delegate, and is one line elsewhere. "This machine" sits with the environment facts. - What a step reads whole is pasted; the rest is a path. - A later round's Visual check has a brief of its own. - The planner the thorough pipeline uses is sent the tasks rules with their example group (braid#430). - Whole-file examples of `visual.json` and `untested.json`. - A budget per brief, tested as sent, for all 25 builders. The ceiling on source text is gone. - `python -m osf.sim --briefs <dir>` writes every brief a simulated run sent. ## Guards and checks (`rules-a-guard-holds`) - A command that runs `$OSF_PYTHON` and names Braid's own package is refused, the briefs' commands excepted. - A Visual check's script may not import the app's modules or link them out of the worktree. - A look is refused once 12 have passed with no criterion added to the report. - `unclear` with `cannot_stage` fails the round and becomes a test the fix round writes; a passing test takes it off the next look list. - The message after a compaction tells a Visual check what it has left. - B9, recorded: a test that itself calls an unseeded random source, shown to a fix round on a red lane. ## Checked - `./tools/test.sh` passes. - A scripted replay of run 183's first-look Visual check against these guards: 10 writes and links of the solver refused, 7 calls into Braid's package refused, 23 looks refused after the twelfth with an empty report. The replay found two faults in the first version of the guards, fixed here. - Sims: the new `judge-cannot-stage` passes. The default play's report said `play` where the schema asks for `session`, which failed seven sims on main; fixed. Two more fail on main for a reason of their own (braid#445). Not done: replays on real models before and after for each brief. The next Scratch run is the comparison with run 183. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat: briefs held to the rules, and the rules a guard holds (#443 #430)
Some checks failed
test / suite (pull_request) Failing after 43s
41b14e4e3c
An audit of every brief runs 174 to 183 sent found five that opened on a
block most never used, four that ended on this machine's frame rate, the
whole change pasted into five steps, and a later round's brief that was the
first brief with sections added (65,929 characters in run 183's round 3).

Briefs:
- every brief opens with its task and ends on how the step ends
- the delegation section goes whole to the steps that delegate, one line
  elsewhere; the machine block sits with the environment facts
- what a step reads whole is pasted, the rest is a path: revise 42,712 to
  5,479 characters on a run-183-shaped fixture, the operator's revise 68,755
  to 1,787, a first-look Visual check 53,351 to 34,816, a later round 59,682
  to 12,234
- the planner the thorough pipeline uses is sent the tasks rules and their
  example group (#430), and the rule for a scenario about a random outcome
- whole-file examples of visual.json and untested.json
- a budget per brief, tested as sent; the ceiling on source text is gone
- python -m osf.sim --briefs <dir> writes every brief a simulated run sent

Guards and checks:
- a command that runs $OSF_PYTHON and names Braid's own package is refused,
  the briefs' commands excepted
- a Visual check's script may not import the app's modules or link them out
- a look is refused once 12 have passed with no criterion added
- unclear with cannot_stage fails the round and becomes a test the fix round
  writes; a passing test takes it off the next look list
- the message after a compaction says what a Visual check has left
- B9, recorded: a test that itself calls an unseeded random source

The default play's report still said `play` where the schema asks for
`session`; seven sims failed on main for it. Two more fail for a reason of
their own (#445).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
test: the per-artifact brief's budget does not depend on the OpenSpec CLI being on the PATH
All checks were successful
test / suite (pull_request) Successful in 46s
dbfbb2365a
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
cmoriarty deleted branch brief-audit-443 2026-11-08 22:47:10 -05:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid!446
No description provided.