Audit every brief Braid sends: size, focus and noise, with research on how to brief an agent #443
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happens. Braid's briefs have grown by accretion: every run's finding added a paragraph, and nothing takes one out. The efficiency passes so far measured minutes, turns and the first-look brief; the only size guard is a ceiling on the brief text in the source (
BRIEF_TEXT_CEILING, 118,500 characters), which says nothing about what one agent is sent. A brief full of text the step does not use hides the directions it needs and brings the compaction sooner. Run 183 (braid#439), as sent:openspec.exploreopenspec.proposeagent.review.combinedagent.reviseopenspec.apply[all]agent.code-reviewagent.fixagent.visual-check(first look)agent.fix.r2agent.visual-check.r3The round 3 Visual check is the worked example: its instruction is "look at 6, carry 53 by id", and it sits 21,500 characters into a 24k section that pastes all 53 passed criteria with their evidence (about 400 characters each; the section was 8k in run 174 and 16k in run 181), beside 12k of specs and about 20k of first-look rules that one line says do not apply. The same 3.6k delegation block opens five of the briefs.
Research first. A written review, with sources, of how to brief an agent: what the model makers publish on prompt and context engineering for long-running agents (order and placement of instructions, what goes first and last, how much context degrades attention, examples against rules, when to reference a file instead of pasting it, how repeated or contradictory rules are read), what the agent harnesses do (Claude Code, opencode, Codex, Aider and their system prompts and compaction), and what is specific to our model (Qwen 3.8 on vLLM: prefix caching, context window, how it treats long briefs). End with a short set of rules Braid's briefs will be held to.
Audit. Every brief Braid builds, in every form it takes: each step of each pipeline, first attempt and retry, each round of the review, the fix loop and the Visual check, fresh session and continued, with and without
browser, Free and vLLM, plus the subagents' prompts and Repair and Ask. For each, from real transcripts: its size as sent, section by section; which sections the step used (did the agent act on it, or did a run ever show it needed); what is duplicated across sections or already in the session; what is pasted that could be a path; what is said twice or contradicts another line; and where the instruction that matters most sits.Fix. From the audit, per brief: lead with what the step must do and its end condition; keep a rule only where a run showed it needed and say it once; reference what is on disk instead of pasting it; a later round or a retry gets a brief written for that round, not the first brief plus sections; shared blocks (delegation, machine facts) are sent where they are used. A size budget per assembled brief, held by tests that build the brief for a representative run (for example a round 2 Visual check after 50 passed criteria), replaces the source ceiling as the guard. Briefs stay application-agnostic.
Done when. The research note and the audit table are under
docs/design/measurements/; each brief has a budget and a test that builds it; the later-round Visual check brief for run 183's case is under a third of its size with the look list in its first screen; a Scratch run on the new briefs is compared with run 183 on compactions, turns and what each step missed.From run 183 (braid#439, finding 17).
Also in scope: six findings from run 183 (braid#439), added on the owner's word. Each is a place where a brief said something and nothing held it, or said nothing where it should have; the audit decides for each whether the fix is words, a check, or both.
ban-checksay nothing of it.inspect.getsource(osf.pipeline.fixloop)through$OSF_PYTHON). Where a rule matters this much, the guard should hold it and the brief should say it once, early.unclearthat could not be staged goes back on the look list with no bound on one criterion (finding 19). Round 3 spent half its 83 minutes ongame-end/shared-winagain. The later-round brief says "one targeted look each" and nothing holds it; an unstageableunclearshould reach the fix round as "write the test".Fix round 2 took 52 minutes for two small fixes (79 turns) and is unexamined: read its brief and transcript as part of the audit.
Research and audit written (branch
brief-audit-443, uncommitted):docs/design/measurements/2026-11-08-briefing-agents-research.mdanddocs/design/measurements/2026-11-08-brief-audit.md; the script and its tables are underdocs/design/measurements/work/brief-audit/.The research ends in 14 rules (open with the task; close with the task; nothing between two pastes; a section earns its place in this step; one round, one brief; say it once; say what to do, and guard a hard boundary; few rules; one example; reference what is on disk; stable text first; the goal comes back after a compaction; a size budget tested as sent; a change is measured).
What the audit found in briefs as sent, runs 174 to 183 (98 agent steps):
Not measured: the briefs these runs never sent (the per-artifact chain, review lenses and recheck, apply groups, the plan steps, repair steps, retries, Free forms, Repair and Ask). The audit proposes a budget per brief (later-round Visual check 65,929 → 15,000; revise 54,645 → 12,000; first-look Visual check 59,375 → 35,000; apply 67,168 → 50,000; summarize 22,148 → 8,000) for the owner to accept or change before the OpenSpec change is written.
Proposed as two OpenSpec changes on branch
brief-audit-443(uncommitted, bothopenspec validate --strict), waiting on the owner's review before any apply.briefs-held-to-the-rules(this issue and #430): every brief opens and closes with its task; the delegation block and the machine's facts go where they are used; a later round, a retry and a fresh session get a brief of their own (build_judge_later: look list and changed files first, carried ids on one line, no passed evidence); paste what is read whole and name the rest (revise gets the findings and what they quote); one worked example of each artifact (a task group,visual.json,untested.json, a spec tag, each report); every planner gets the tasks rules; a rule once, as what to do, with no run or issue number in any text sent; how a random outcome is specified and tested; a budget per assembled brief, tested on a fixture shaped like run 183 (Visual check first look 35,000, later round 15,000; apply 50,000; review 50,000; revise 12,000; planner 10,000; summary 8,000; fix 6,000), withBRIEF_TEXT_CEILINGretired. The simulated runs gain--briefs <dir>so the briefs runs 174 to 183 never sent are measured and scored too. Six task groups, by brief; replays of run 183's revise, apply, code review and both Visual checks before and after.rules-a-guard-holds(the six findings):$OSF_PYTHONruns Braid's commands only; a Visual check's script under the run's tmp may not import the worktree or link its modules; a look is refused when twelve have passed since the report last gained a criterion;unclearwith"cannot_stage": truefails the round and reaches the fix round as a test to write, and leaves the look list once tagged; the message after a compaction restates what the step leaves and how it ends, with a later round's remaining looks; the lint records a test's own unseeded random call (B7, not blocking).Not in either: the shape check on the hand-over files (#440) and the reason check on the untested hatch (#441), which keep state-change scenarios from reaching the Visual check at all. They are the same causal chain and belong in the same batch.
Shipped in PR 446, as two changes:
briefs-held-to-the-rulesandrules-a-guard-holds. The audit and the research are in the repository:docs/design/measurements/2026-11-08-brief-audit.mdand2026-11-08-briefing-agents-research.md.The briefs, before and after
Built on a worktree shaped like run 183's (five capabilities, 104 scenarios, 59 criteria), in characters:
Every brief now opens with its task and ends on how the step ends. A test builds all 25 on the fixture and fails one that goes over its budget.
python -m osf.sim --briefs <dir>writes every brief a simulated run sent, to read one as the agent does.The six findings from run 183
unclearwithcannot_stagefails the round and becomes a test the fix round writesEvidence from the laptop
A scripted replay of run 183's first-look Visual check (
./tools/replay.sh scripted, the step's own 519 tool calls against today's guards):ln -sfn <work>/node_modules …and every write of the solver that imported@scratch/engine. "Refused: a Visual check's script may not run the app's own modules: it would judge the code, not what a person sees. Reach the state through the running app (its page, its state hook, its network interface), or mark the criterionunclearwith"cannot_stage": trueand say what could not be reached."osf.pipeline.fixloopandJUDGE_SCHEMA. "Refused: Braid's own package is not yours to run or read. To see what is wrong with a report, run the check command in your brief (…): it prints each fault. The report's form is in your brief."The replay found two faults in the first version of the guards, fixed before the merge: a workspace named as a folder (
packages/engine, as run 183's project names them) was not read, so the solver's writes passed; and harmless"$OSF_PYTHON" -c "import json…"one-liners were refused.New tests fail without the change and pass with it: the budgets, the frame, the guards,
cannot_stage, the compaction message, B9. The new simjudge-cannot-stagepasses.Not done