Audit every brief Braid sends: size, focus and noise, with research on how to brief an agent #443

Closed
opened 2026-11-08 18:54:46 -05:00 by cmoriarty · 3 comments
Owner

What happens. Braid's briefs have grown by accretion: every run's finding added a paragraph, and nothing takes one out. The efficiency passes so far measured minutes, turns and the first-look brief; the only size guard is a ceiling on the brief text in the source (BRIEF_TEXT_CEILING, 118,500 characters), which says nothing about what one agent is sent. A brief full of text the step does not use hides the directions it needs and brings the compaction sooner. Run 183 (braid#439), as sent:

step brief (chars) sections note
openspec.explore 7,287 9
openspec.propose 12,295 10 never carried the tasks rules (#430)
agent.review.combined 58,893 39
agent.revise 54,645 34 a one-minute step
openspec.apply[all] 67,168 39 compacted twice
agent.code-review 253,196 36 pastes by design (#427)
agent.fix 5,955 4
agent.visual-check (first look) 59,375 33 compacted twice
agent.fix.r2 11,758 5
agent.visual-check.r3 65,929 26 larger than the first look

The round 3 Visual check is the worked example: its instruction is "look at 6, carry 53 by id", and it sits 21,500 characters into a 24k section that pastes all 53 passed criteria with their evidence (about 400 characters each; the section was 8k in run 174 and 16k in run 181), beside 12k of specs and about 20k of first-look rules that one line says do not apply. The same 3.6k delegation block opens five of the briefs.

Research first. A written review, with sources, of how to brief an agent: what the model makers publish on prompt and context engineering for long-running agents (order and placement of instructions, what goes first and last, how much context degrades attention, examples against rules, when to reference a file instead of pasting it, how repeated or contradictory rules are read), what the agent harnesses do (Claude Code, opencode, Codex, Aider and their system prompts and compaction), and what is specific to our model (Qwen 3.8 on vLLM: prefix caching, context window, how it treats long briefs). End with a short set of rules Braid's briefs will be held to.

Audit. Every brief Braid builds, in every form it takes: each step of each pipeline, first attempt and retry, each round of the review, the fix loop and the Visual check, fresh session and continued, with and without browser, Free and vLLM, plus the subagents' prompts and Repair and Ask. For each, from real transcripts: its size as sent, section by section; which sections the step used (did the agent act on it, or did a run ever show it needed); what is duplicated across sections or already in the session; what is pasted that could be a path; what is said twice or contradicts another line; and where the instruction that matters most sits.

Fix. From the audit, per brief: lead with what the step must do and its end condition; keep a rule only where a run showed it needed and say it once; reference what is on disk instead of pasting it; a later round or a retry gets a brief written for that round, not the first brief plus sections; shared blocks (delegation, machine facts) are sent where they are used. A size budget per assembled brief, held by tests that build the brief for a representative run (for example a round 2 Visual check after 50 passed criteria), replaces the source ceiling as the guard. Briefs stay application-agnostic.

Done when. The research note and the audit table are under docs/design/measurements/; each brief has a budget and a test that builds it; the later-round Visual check brief for run 183's case is under a third of its size with the look list in its first screen; a Scratch run on the new briefs is compared with run 183 on compactions, turns and what each step missed.

From run 183 (braid#439, finding 17).

Also in scope: six findings from run 183 (braid#439), added on the owner's word. Each is a place where a brief said something and nothing held it, or said nothing where it should have; the audit decides for each whether the fix is words, a check, or both.

  1. The planner writes MAY in probabilistic scenarios (finding 4; run 182 had the same). The review flags it and revise fixes it, every run. The proposal rules should say how a random outcome is stated so it can be checked (a seed, or a property over many).
  2. A test on an unseeded random source passes the weak-test lint (finding 9). Round 1 went red on a test that drove bots over a random deal and asserted something chance decides. The test rules and ban-check say nothing of it.
  3. The judge runs scripts that import the worktree's modules, and reads Braid's own source (findings 10 to 12). The brief forbids the first in a sentence the judge did not act on (a solver, then a driver, an hour of the first look); nothing mentions the second (inspect.getsource(osf.pipeline.fixloop) through $OSF_PYTHON). Where a rule matters this much, the guard should hold it and the brief should say it once, early.
  4. "Write the report as you go" is satisfied by an empty file (finding 13). The first look is refused until the file exists; the judge wrote it empty and added 59 criteria at the end, after two compactions.
  5. An unclear that could not be staged goes back on the look list with no bound on one criterion (finding 19). Round 3 spent half its 83 minutes on game-end/shared-win again. The later-round brief says "one targeted look each" and nothing holds it; an unstageable unclear should reach the fix round as "write the test".
  6. The apply compacts twice, the Visual check twice (finding 3). The briefs are part of what fills those windows; the audit measures how much of each window at the first compaction was brief, pasted material and the agent's own looks.

Fix round 2 took 52 minutes for two small fixes (79 turns) and is unexamined: read its brief and transcript as part of the audit.

**What happens.** Braid's briefs have grown by accretion: every run's finding added a paragraph, and nothing takes one out. The efficiency passes so far measured minutes, turns and the first-look brief; the only size guard is a ceiling on the brief text in the source (`BRIEF_TEXT_CEILING`, 118,500 characters), which says nothing about what one agent is sent. A brief full of text the step does not use hides the directions it needs and brings the compaction sooner. Run 183 (braid#439), as sent: | step | brief (chars) | sections | note | |---|---:|---:|---| | `openspec.explore` | 7,287 | 9 | | | `openspec.propose` | 12,295 | 10 | never carried the tasks rules (#430) | | `agent.review.combined` | 58,893 | 39 | | | `agent.revise` | 54,645 | 34 | a one-minute step | | `openspec.apply[all]` | 67,168 | 39 | compacted twice | | `agent.code-review` | 253,196 | 36 | pastes by design (#427) | | `agent.fix` | 5,955 | 4 | | | `agent.visual-check` (first look) | 59,375 | 33 | compacted twice | | `agent.fix.r2` | 11,758 | 5 | | | `agent.visual-check.r3` | 65,929 | 26 | larger than the first look | The round 3 Visual check is the worked example: its instruction is "look at 6, carry 53 by id", and it sits 21,500 characters into a 24k section that pastes all 53 passed criteria with their evidence (about 400 characters each; the section was 8k in run 174 and 16k in run 181), beside 12k of specs and about 20k of first-look rules that one line says do not apply. The same 3.6k delegation block opens five of the briefs. **Research first.** A written review, with sources, of how to brief an agent: what the model makers publish on prompt and context engineering for long-running agents (order and placement of instructions, what goes first and last, how much context degrades attention, examples against rules, when to reference a file instead of pasting it, how repeated or contradictory rules are read), what the agent harnesses do (Claude Code, opencode, Codex, Aider and their system prompts and compaction), and what is specific to our model (Qwen 3.8 on vLLM: prefix caching, context window, how it treats long briefs). End with a short set of rules Braid's briefs will be held to. **Audit.** Every brief Braid builds, in every form it takes: each step of each pipeline, first attempt and retry, each round of the review, the fix loop and the Visual check, fresh session and continued, with and without `browser`, Free and vLLM, plus the subagents' prompts and Repair and Ask. For each, from real transcripts: its size as sent, section by section; which sections the step used (did the agent act on it, or did a run ever show it needed); what is duplicated across sections or already in the session; what is pasted that could be a path; what is said twice or contradicts another line; and where the instruction that matters most sits. **Fix.** From the audit, per brief: lead with what the step must do and its end condition; keep a rule only where a run showed it needed and say it once; reference what is on disk instead of pasting it; a later round or a retry gets a brief written for that round, not the first brief plus sections; shared blocks (delegation, machine facts) are sent where they are used. A size budget per assembled brief, held by tests that build the brief for a representative run (for example a round 2 Visual check after 50 passed criteria), replaces the source ceiling as the guard. Briefs stay application-agnostic. **Done when.** The research note and the audit table are under `docs/design/measurements/`; each brief has a budget and a test that builds it; the later-round Visual check brief for run 183's case is under a third of its size with the look list in its first screen; a Scratch run on the new briefs is compared with run 183 on compactions, turns and what each step missed. From run 183 (braid#439, finding 17). **Also in scope: six findings from run 183 (braid#439), added on the owner's word.** Each is a place where a brief said something and nothing held it, or said nothing where it should have; the audit decides for each whether the fix is words, a check, or both. 1. **The planner writes MAY in probabilistic scenarios** (finding 4; run 182 had the same). The review flags it and revise fixes it, every run. The proposal rules should say how a random outcome is stated so it can be checked (a seed, or a property over many). 2. **A test on an unseeded random source passes the weak-test lint** (finding 9). Round 1 went red on a test that drove bots over a random deal and asserted something chance decides. The test rules and `ban-check` say nothing of it. 3. **The judge runs scripts that import the worktree's modules, and reads Braid's own source** (findings 10 to 12). The brief forbids the first in a sentence the judge did not act on (a solver, then a driver, an hour of the first look); nothing mentions the second (`inspect.getsource(osf.pipeline.fixloop)` through `$OSF_PYTHON`). Where a rule matters this much, the guard should hold it and the brief should say it once, early. 4. **\"Write the report as you go\" is satisfied by an empty file** (finding 13). The first look is refused until the file exists; the judge wrote it empty and added 59 criteria at the end, after two compactions. 5. **An `unclear` that could not be staged goes back on the look list with no bound on one criterion** (finding 19). Round 3 spent half its 83 minutes on `game-end/shared-win` again. The later-round brief says \"one targeted look each\" and nothing holds it; an unstageable `unclear` should reach the fix round as \"write the test\". 6. **The apply compacts twice, the Visual check twice** (finding 3). The briefs are part of what fills those windows; the audit measures how much of each window at the first compaction was brief, pasted material and the agent's own looks. Fix round 2 took 52 minutes for two small fixes (79 turns) and is unexamined: read its brief and transcript as part of the audit.
Author
Owner

Research and audit written (branch brief-audit-443, uncommitted): docs/design/measurements/2026-11-08-briefing-agents-research.md and docs/design/measurements/2026-11-08-brief-audit.md; the script and its tables are under docs/design/measurements/work/brief-audit/.

The research ends in 14 rules (open with the task; close with the task; nothing between two pastes; a section earns its place in this step; one round, one brief; say it once; say what to do, and guard a hard boundary; few rules; one example; reference what is on disk; stable text first; the goal comes back after a compaction; a size budget tested as sent; a change is measured).

What the audit found in briefs as sent, runs 174 to 183 (98 agent steps):

  1. Five briefs open with the 3.6k delegation block. The planner carried it in 10 runs and never delegated; the code review used a subagent in 2 of 9, the apply in 3 of 10, the Visual check in 12 of 15.
  2. Four briefs end on this machine's frame rate, the code review and the fix among them.
  3. The whole change (proposal, design, tasks, five delta specs, the operator's brief; 50 to 55k) is pasted into five steps. Revise gets 54,645 characters to resolve four findings in a minute. Of 424k characters of non-code brief in run 183, about 302k were also sent to another step.
  4. A later round is the first brief plus sections: round 3's Visual check is 65,929 characters, its "look at 6, carry 53" 21,500 characters into a section pasting 53 criteria's evidence.
  5. The Visual check's fixed text is 17,800 characters with 40-odd prohibitions; the ones that held in run 183 were the ones a guard holds.
  6. A brief can also lack the one rule its step needs: the planner never gets the tasks rules (#430); the hand-over files have no worked example (#440).
  7. Apply compacted in 4 of 10 runs, the Visual check in 4 of 15; nothing restates the goal afterwards.
  8. The probes are used (14 of 15): a block that earns its place.

Not measured: the briefs these runs never sent (the per-artifact chain, review lenses and recheck, apply groups, the plan steps, repair steps, retries, Free forms, Repair and Ask). The audit proposes a budget per brief (later-round Visual check 65,929 → 15,000; revise 54,645 → 12,000; first-look Visual check 59,375 → 35,000; apply 67,168 → 50,000; summarize 22,148 → 8,000) for the owner to accept or change before the OpenSpec change is written.

**Research and audit written (branch `brief-audit-443`, uncommitted): `docs/design/measurements/2026-11-08-briefing-agents-research.md` and `docs/design/measurements/2026-11-08-brief-audit.md`; the script and its tables are under `docs/design/measurements/work/brief-audit/`.** The research ends in 14 rules (open with the task; close with the task; nothing between two pastes; a section earns its place in this step; one round, one brief; say it once; say what to do, and guard a hard boundary; few rules; one example; reference what is on disk; stable text first; the goal comes back after a compaction; a size budget tested as sent; a change is measured). What the audit found in briefs as sent, runs 174 to 183 (98 agent steps): 1. Five briefs open with the 3.6k delegation block. The planner carried it in 10 runs and never delegated; the code review used a subagent in 2 of 9, the apply in 3 of 10, the Visual check in 12 of 15. 2. Four briefs end on this machine's frame rate, the code review and the fix among them. 3. The whole change (proposal, design, tasks, five delta specs, the operator's brief; 50 to 55k) is pasted into five steps. Revise gets 54,645 characters to resolve four findings in a minute. Of 424k characters of non-code brief in run 183, about 302k were also sent to another step. 4. A later round is the first brief plus sections: round 3's Visual check is 65,929 characters, its "look at 6, carry 53" 21,500 characters into a section pasting 53 criteria's evidence. 5. The Visual check's fixed text is 17,800 characters with 40-odd prohibitions; the ones that held in run 183 were the ones a guard holds. 6. A brief can also lack the one rule its step needs: the planner never gets the tasks rules (#430); the hand-over files have no worked example (#440). 7. Apply compacted in 4 of 10 runs, the Visual check in 4 of 15; nothing restates the goal afterwards. 8. The probes are used (14 of 15): a block that earns its place. Not measured: the briefs these runs never sent (the per-artifact chain, review lenses and recheck, apply groups, the plan steps, repair steps, retries, Free forms, Repair and Ask). The audit proposes a budget per brief (later-round Visual check 65,929 → 15,000; revise 54,645 → 12,000; first-look Visual check 59,375 → 35,000; apply 67,168 → 50,000; summarize 22,148 → 8,000) for the owner to accept or change before the OpenSpec change is written.
Author
Owner

Proposed as two OpenSpec changes on branch brief-audit-443 (uncommitted, both openspec validate --strict), waiting on the owner's review before any apply.

briefs-held-to-the-rules (this issue and #430): every brief opens and closes with its task; the delegation block and the machine's facts go where they are used; a later round, a retry and a fresh session get a brief of their own (build_judge_later: look list and changed files first, carried ids on one line, no passed evidence); paste what is read whole and name the rest (revise gets the findings and what they quote); one worked example of each artifact (a task group, visual.json, untested.json, a spec tag, each report); every planner gets the tasks rules; a rule once, as what to do, with no run or issue number in any text sent; how a random outcome is specified and tested; a budget per assembled brief, tested on a fixture shaped like run 183 (Visual check first look 35,000, later round 15,000; apply 50,000; review 50,000; revise 12,000; planner 10,000; summary 8,000; fix 6,000), with BRIEF_TEXT_CEILING retired. The simulated runs gain --briefs <dir> so the briefs runs 174 to 183 never sent are measured and scored too. Six task groups, by brief; replays of run 183's revise, apply, code review and both Visual checks before and after.

rules-a-guard-holds (the six findings): $OSF_PYTHON runs Braid's commands only; a Visual check's script under the run's tmp may not import the worktree or link its modules; a look is refused when twelve have passed since the report last gained a criterion; unclear with "cannot_stage": true fails the round and reaches the fix round as a test to write, and leaves the look list once tagged; the message after a compaction restates what the step leaves and how it ends, with a later round's remaining looks; the lint records a test's own unseeded random call (B7, not blocking).

Not in either: the shape check on the hand-over files (#440) and the reason check on the untested hatch (#441), which keep state-change scenarios from reaching the Visual check at all. They are the same causal chain and belong in the same batch.

**Proposed as two OpenSpec changes on branch `brief-audit-443` (uncommitted, both `openspec validate --strict`), waiting on the owner's review before any apply.** `briefs-held-to-the-rules` (this issue and #430): every brief opens and closes with its task; the delegation block and the machine's facts go where they are used; a later round, a retry and a fresh session get a brief of their own (`build_judge_later`: look list and changed files first, carried ids on one line, no passed evidence); paste what is read whole and name the rest (revise gets the findings and what they quote); one worked example of each artifact (a task group, `visual.json`, `untested.json`, a spec tag, each report); every planner gets the tasks rules; a rule once, as what to do, with no run or issue number in any text sent; how a random outcome is specified and tested; a budget per assembled brief, tested on a fixture shaped like run 183 (Visual check first look 35,000, later round 15,000; apply 50,000; review 50,000; revise 12,000; planner 10,000; summary 8,000; fix 6,000), with `BRIEF_TEXT_CEILING` retired. The simulated runs gain `--briefs <dir>` so the briefs runs 174 to 183 never sent are measured and scored too. Six task groups, by brief; replays of run 183's revise, apply, code review and both Visual checks before and after. `rules-a-guard-holds` (the six findings): `$OSF_PYTHON` runs Braid's commands only; a Visual check's script under the run's tmp may not import the worktree or link its modules; a look is refused when twelve have passed since the report last gained a criterion; `unclear` with `"cannot_stage": true` fails the round and reaches the fix round as a test to write, and leaves the look list once tagged; the message after a compaction restates what the step leaves and how it ends, with a later round's remaining looks; the lint records a test's own unseeded random call (B7, not blocking). Not in either: the shape check on the hand-over files (#440) and the reason check on the untested hatch (#441), which keep state-change scenarios from reaching the Visual check at all. They are the same causal chain and belong in the same batch.
Author
Owner

Shipped in PR 446, as two changes: briefs-held-to-the-rules and rules-a-guard-holds. The audit and the research are in the repository: docs/design/measurements/2026-11-08-brief-audit.md and 2026-11-08-briefing-agents-research.md.

The briefs, before and after

Built on a worktree shaped like run 183's (five capabilities, 104 scenarios, 59 criteria), in characters:

brief before after budget
propose 10,033 9,511 10,000
combined review 45,955 46,031 50,000
revise 42,712 5,479 12,000
the operator's revise 68,755 1,787 2,000
apply 53,329 46,455 50,000
code review, without its pastes 6,000
Visual check, first look 53,351 34,816 35,000
Visual check, a later round 59,682 12,234 15,000
summary 6,167 2,392 8,000

Every brief now opens with its task and ends on how the step ends. A test builds all 25 on the fixture and fails one that goes over its budget. python -m osf.sim --briefs <dir> writes every brief a simulated run sent, to read one as the agent does.

The six findings from run 183

finding what holds it now
the planner writes MAY in a scenario about a random outcome the planner's rules say to state it for a seed or as a property, with an example
a test on an unseeded random source passes the lint B9, recorded, and shown to a fix round whose lane is red
the Visual check runs scripts over the app's modules and reads Braid's source two guards
"write the report as you go" satisfied by an empty file a look is refused once 12 have passed with no criterion added
a criterion that cannot be staged returns to the look list unclear with cannot_stage fails the round and becomes a test the fix round writes
the long steps compact and lose their goal the message after a compaction says what the step leaves and which criteria are left

Evidence from the laptop

A scripted replay of run 183's first-look Visual check (./tools/replay.sh scripted, the step's own 519 tool calls against today's guards):

  • 10 refused: the ln -sfn <work>/node_modules … and every write of the solver that imported @scratch/engine. "Refused: a Visual check's script may not run the app's own modules: it would judge the code, not what a person sees. Reach the state through the running app (its page, its state hook, its network interface), or mark the criterion unclear with "cannot_stage": true and say what could not be reached."
  • 7 refused: the calls that printed osf.pipeline.fixloop and JUDGE_SCHEMA. "Refused: Braid's own package is not yours to run or read. To see what is wrong with a report, run the check command in your brief (…): it prints each fault. The report's form is in your brief."
  • 23 refused: every look after the twelfth while the report held no criterion. "Add what you have judged to .osf/fix/r2/judge.json first: 12 looks have passed since it last gained a criterion."

The replay found two faults in the first version of the guards, fixed before the merge: a workspace named as a folder (packages/engine, as run 183's project names them) was not read, so the solver's writes passed; and harmless "$OSF_PYTHON" -c "import json…" one-liners were refused.

New tests fail without the change and pass with it: the budgets, the frame, the guards, cannot_stage, the compaction message, B9. The new sim judge-cannot-stage passes.

Not done

  • No real model has read the new briefs. A Free replay of run 183's revise step was tried twice; both times every request on the strong tier was refused and nothing was served. The plan of five steps, three times each, before and after, was not run.
  • The next Scratch run is the measurement against run 183: turns, compactions, and what each step missed.
  • braid#440 and braid#441 (the shape and the reasons of the hand-over files) are separate changes. The briefs now show a whole-file example of each; the checks are still to write.
  • Two sims fail on main for a reason of their own: braid#445.
Shipped in PR 446, as two changes: `briefs-held-to-the-rules` and `rules-a-guard-holds`. The audit and the research are in the repository: `docs/design/measurements/2026-11-08-brief-audit.md` and `2026-11-08-briefing-agents-research.md`. ## The briefs, before and after Built on a worktree shaped like run 183's (five capabilities, 104 scenarios, 59 criteria), in characters: | brief | before | after | budget | |---|---:|---:|---:| | propose | 10,033 | 9,511 | 10,000 | | combined review | 45,955 | 46,031 | 50,000 | | revise | 42,712 | 5,479 | 12,000 | | the operator's revise | 68,755 | 1,787 | 2,000 | | apply | 53,329 | 46,455 | 50,000 | | code review, without its pastes | | | 6,000 | | Visual check, first look | 53,351 | 34,816 | 35,000 | | Visual check, a later round | 59,682 | 12,234 | 15,000 | | summary | 6,167 | 2,392 | 8,000 | Every brief now opens with its task and ends on how the step ends. A test builds all 25 on the fixture and fails one that goes over its budget. `python -m osf.sim --briefs <dir>` writes every brief a simulated run sent, to read one as the agent does. ## The six findings from run 183 | finding | what holds it now | |---|---| | the planner writes MAY in a scenario about a random outcome | the planner's rules say to state it for a seed or as a property, with an example | | a test on an unseeded random source passes the lint | B9, recorded, and shown to a fix round whose lane is red | | the Visual check runs scripts over the app's modules and reads Braid's source | two guards | | "write the report as you go" satisfied by an empty file | a look is refused once 12 have passed with no criterion added | | a criterion that cannot be staged returns to the look list | `unclear` with `cannot_stage` fails the round and becomes a test the fix round writes | | the long steps compact and lose their goal | the message after a compaction says what the step leaves and which criteria are left | ## Evidence from the laptop A scripted replay of run 183's first-look Visual check (`./tools/replay.sh scripted`, the step's own 519 tool calls against today's guards): - 10 refused: the `ln -sfn <work>/node_modules …` and every write of the solver that imported `@scratch/engine`. "Refused: a Visual check's script may not run the app's own modules: it would judge the code, not what a person sees. Reach the state through the running app (its page, its state hook, its network interface), or mark the criterion `unclear` with `"cannot_stage": true` and say what could not be reached." - 7 refused: the calls that printed `osf.pipeline.fixloop` and `JUDGE_SCHEMA`. "Refused: Braid's own package is not yours to run or read. To see what is wrong with a report, run the check command in your brief (…): it prints each fault. The report's form is in your brief." - 23 refused: every look after the twelfth while the report held no criterion. "Add what you have judged to .osf/fix/r2/judge.json first: 12 looks have passed since it last gained a criterion." The replay found two faults in the first version of the guards, fixed before the merge: a workspace named as a folder (`packages/engine`, as run 183's project names them) was not read, so the solver's writes passed; and harmless `"$OSF_PYTHON" -c "import json…"` one-liners were refused. New tests fail without the change and pass with it: the budgets, the frame, the guards, `cannot_stage`, the compaction message, B9. The new sim `judge-cannot-stage` passes. ## Not done - **No real model has read the new briefs.** A Free replay of run 183's revise step was tried twice; both times every request on the strong tier was refused and nothing was served. The plan of five steps, three times each, before and after, was not run. - The next Scratch run is the measurement against run 183: turns, compactions, and what each step missed. - braid#440 and braid#441 (the shape and the reasons of the hand-over files) are separate changes. The briefs now show a whole-file example of each; the checks are still to write. - Two sims fail on main for a reason of their own: braid#445.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#443
No description provided.