cmoriarty/braid
1
0
Fork 0
Code Issues 15 Pull requests Projects Releases Packages 1 Wiki Activity Actions

An agent's empty question holds its run until a person answers it: Braid should answer it itself #147

New issue
Closed
opened 2026-10-05 18:46:44 -04:00 by cmoriarty · 2 comments
cmoriarty commented 2026-10-05 18:46:44 -04:00
Owner
Copy link

What happened

In run 48 (git-flow, scratch-139), agent.write-spec-tests had done its work: its background test task had just reported PASS. At 22:17:22Z the model then called opencode's question tool with {"questions": []}.

  • Braid opened a question gate for it ("The agent asked a question with no content") and parked the step in awaiting_question.
  • Nothing ran for 27½ minutes, until I answered the gate with empty answers at 22:44:50Z.
  • The step resumed and succeeded within seconds.

Why it matters

  • An empty question gives a person nothing to answer and nothing to add, so the gate only waits.
  • Questions are not auto-approved. A run with auto-approve on waits until someone happens to look.
  • The step's time limit keeps running while it waits. This one had 12 minutes left when it was answered. At its limit the attempt would have been stopped and retried from its entry checkpoint, redoing about 20 minutes of work.

How often: production's log has four agent questions. The other three had one question each and were answered within about 2 minutes. This is the first empty one.

Proposal

  • When question.asked carries no questions, Braid replies {"answers": []} itself and opens no gate. opencode accepts that reply for a request with no questions, the tool call returns, and the model sees an empty answer.
  • The reply is recorded on the log, so the transcript shows the question and why nobody was asked.
  • The gate's own comment says there is "no default that would be honest to send on the human's behalf". For an empty question, the empty reply is that honest default: it is the only answer there is.
  • Separately, worth deciding: should a question's wait pause the step's time limit?
**What happened** In run 48 (git-flow, scratch-139), `agent.write-spec-tests` had done its work: its background test task had just reported PASS. At 22:17:22Z the model then called opencode's `question` tool with `{"questions": []}`. - Braid opened a question gate for it ("The agent asked a question with no content") and parked the step in `awaiting_question`. - Nothing ran for 27½ minutes, until I answered the gate with empty answers at 22:44:50Z. - The step resumed and succeeded within seconds. **Why it matters** - An empty question gives a person nothing to answer and nothing to add, so the gate only waits. - Questions are not auto-approved. A run with auto-approve on waits until someone happens to look. - The step's time limit keeps running while it waits. This one had 12 minutes left when it was answered. At its limit the attempt would have been stopped and retried from its entry checkpoint, redoing about 20 minutes of work. **How often:** production's log has four agent questions. The other three had one question each and were answered within about 2 minutes. This is the first empty one. **Proposal** - When `question.asked` carries no questions, Braid replies `{"answers": []}` itself and opens no gate. opencode accepts that reply for a request with no questions, the tool call returns, and the model sees an empty answer. - The reply is recorded on the log, so the transcript shows the question and why nobody was asked. - The gate's own comment says there is "no default that would be honest to send on the human's behalf". For an empty question, the empty reply is that honest default: it is the only answer there is. - Separately, worth deciding: should a question's wait pause the step's time limit?
cmoriarty commented 2026-10-05 22:15:48 -04:00
Author
Owner
Copy link

A simulated run now shows this bug (#149): tests/sim/empty-question.yaml, a known failure of #147.

  • Its model asks a question with no content in agent.implement.
  • Today the run holds on the question gate, and the scenario reports the open gate after 20 s of quiet.
  • The fix should make it pass, and remove known_failure: "#147" from the scenario in the same change: the sim lane fails until the marker goes.

Run it alone with python -m osf.sim empty-question.

A simulated run now shows this bug (#149): `tests/sim/empty-question.yaml`, a known failure of #147. - Its model asks a question with no content in `agent.implement`. - Today the run holds on the question gate, and the scenario reports the open gate after 20 s of quiet. - The fix should make it pass, and remove `known_failure: "#147"` from the scenario in the same change: the sim lane fails until the marker goes. Run it alone with `python -m osf.sim empty-question`.
cmoriarty commented 2026-10-05 23:45:42 -04:00
Author
Owner
Copy link

Shipped in a68aeb5 (live since f0213ef's deploy), archived as answer-empty-agent-question.

What changed

  • When an agent's question has no questions in it, Braid replies {"answers": []} itself, at once. It opens no gate, the step is never parked waiting on a question, and no notification goes out.
  • The reply is on the log as a reply_question operation like a forwarded answer. Its intent says the question had no content, so nobody was asked. The transcript still shows the question and its answer.
  • If the reply fails, Braid opens the question gate as before, so the agent is never blocked on a question nobody can see.
  • The same path covers a pending question found after a dropped connection.
  • New spec: agent-questions.

Verified

  • tests/sim/empty-question.yaml passes and its known_failure marker is gone. With the fix reverted, it holds on the gate as run 48 did.
  • The deploy's self-check ran it in the new image: pass.

Not in this change: whether waiting on a question should pause the step's time limit. It affects every question, so it is left for its own issue if wanted.

Deploy note: a68aeb5's own deploy failed the self-check on a /tmp/tmp.* left behind, with every simulated run passing. The likeliest cause is the browser check's Chromium writing into its scratch directory after it was removed. f0213ef kills that process group first, and lists what a leftover holds if one recurs.

Shipped in a68aeb5 (live since f0213ef's deploy), archived as `answer-empty-agent-question`. **What changed** - When an agent's question has no questions in it, Braid replies `{"answers": []}` itself, at once. It opens no gate, the step is never parked waiting on a question, and no notification goes out. - The reply is on the log as a `reply_question` operation like a forwarded answer. Its intent says the question had no content, so nobody was asked. The transcript still shows the question and its answer. - If the reply fails, Braid opens the question gate as before, so the agent is never blocked on a question nobody can see. - The same path covers a pending question found after a dropped connection. - New spec: `agent-questions`. **Verified** - `tests/sim/empty-question.yaml` passes and its `known_failure` marker is gone. With the fix reverted, it holds on the gate as run 48 did. - The deploy's self-check ran it in the new image: pass. **Not in this change:** whether waiting on a question should pause the step's time limit. It affects every question, so it is left for its own issue if wanted. **Deploy note:** a68aeb5's own deploy failed the self-check on a `/tmp/tmp.*` left behind, with every simulated run passing. The likeliest cause is the browser check's Chromium writing into its scratch directory after it was removed. f0213ef kills that process group first, and lists what a leftover holds if one recurs.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
archive/prefect-kestra-experiments
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cmoriarty/braid#147
No description provided.