The spine names a dead attempt's tool call as running

cmoriarty commented

2026-09-27 01:11:02 -04:00

Owner

Problem

The spine and the digest name a tool call as running when its session is dead. Seen on production minutes after #71 deployed:

The deploy of bd3902f restarted osfd at 01:07 while openspec.apply[4] of run_01M3GB3KN38YT9TZJ7KA7RQ5M0 was in attempt 1. Attempt 1's session had just started a bash call: part …PWI9u4, whose one and only record is pending at 01:06:59 with input {}.
Attempt 2 opened at 01:07:22 in a new session, and read, grepped and ran its own bash calls.
For the whole 90 s I captured /api/stream, 270 digest frames for the step said running_tool: bash, running_target: "", with running_for_s climbing to 133 s. That's 01:06:59 plus 133 s, so the frames were counting attempt 1's dead call, and they showed it whenever attempt 2 had nothing in flight of its own.

Cause

The digest keeps one fold per step, across attempts: _backfill reads the step's recent rows, and on_commit appends every event. derive_activity hands the whole fold to running_tool_call, which takes the newest part still pending/running. A part whose session died never gets another record, so it stays in flight for good.

The watchdog's _tool_in_flight reads WHERE step_id = ? AND attempt = ?, so the stall redo is not affected.

Proposal

running_tool_call (or derive_activity before it) considers only the tool parts of the newest attempt in the events it's given. A dead session's call inside the same attempt (a budget respawn) is worth covering in the same change if it's cheap: the respawn names the session that took over.

Done when

After a restart, or any new attempt, the spine never names a tool call from an earlier attempt as running.
A test drives an attempt-1 pending part followed by attempt-2 events and gets no running tool (or attempt 2's own).

## Problem The spine and the digest name a tool call as running when its session is dead. Seen on production minutes after #71 deployed: - The deploy of `bd3902f` restarted osfd at 01:07 while `openspec.apply[4]` of `run_01M3GB3KN38YT9TZJ7KA7RQ5M0` was in attempt 1. Attempt 1's session had just started a `bash` call: part `…PWI9u4`, whose one and only record is `pending` at 01:06:59 with input `{}`. - Attempt 2 opened at 01:07:22 in a new session, and read, grepped and ran its own `bash` calls. - For the whole 90 s I captured `/api/stream`, 270 digest frames for the step said `running_tool: bash`, `running_target: ""`, with `running_for_s` climbing to 133 s. That's 01:06:59 plus 133 s, so the frames were counting attempt 1's dead call, and they showed it whenever attempt 2 had nothing in flight of its own. ## Cause The digest keeps one fold per step, across attempts: `_backfill` reads the step's recent rows, and `on_commit` appends every event. `derive_activity` hands the whole fold to `running_tool_call`, which takes the newest part still `pending`/`running`. A part whose session died never gets another record, so it stays in flight for good. The watchdog's `_tool_in_flight` reads `WHERE step_id = ? AND attempt = ?`, so the stall redo is not affected. ## Proposal `running_tool_call` (or `derive_activity` before it) considers only the tool parts of the newest attempt in the events it's given. A dead session's call inside the same attempt (a budget respawn) is worth covering in the same change if it's cheap: the respawn names the session that took over. ## Done when - After a restart, or any new attempt, the spine never names a tool call from an earlier attempt as running. - A test drives an attempt-1 `pending` part followed by attempt-2 events and gets no running tool (or attempt 2's own).

cmoriarty referenced this issue

2026-09-27 01:11:05 -04:00

A running command reads as silence: show what is running, and don't redo a step waiting on its own tool call #69

cmoriarty referenced this issue

2026-09-27 09:27:08 -04:00

A restart of osfd throws away running steps' work: recovery should continue in place, as resume does #70

cmoriarty referenced this issue

2026-10-03 09:55:19 -04:00

A failing test suite passes agent.test.unit: its completion only checks that the report exists #137

cmoriarty referenced this issue from a commit

2026-10-03 11:01:36 -04:00

Propose running-tool-of-newest-attempt (#73) [skip ci]

cmoriarty referenced this issue from a commit

2026-10-03 11:01:36 -04:00

fix(digest): the running tool call is the newest attempt's own, in the session now working (#73)

cmoriarty commented

2026-10-03 11:06:27 -04:00

Author

Owner

Fixed and live on production (9ae119c, deployed 2026-10-03).

What changed

running_tool_call takes only the newest attempt's own tool calls (the attempt the caller names, else the newest in the rows it is given), and none from before the newest budget respawn of that attempt.
The digest's step fold remembers the attempt the step is now in and passes it on, so a restart (or any new attempt) can no longer leave a dead session's pending call named as running.

How it was checked

Unit tests: an attempt-1 pending call followed by attempt-2 events names nothing; attempt 2's own call is named; a call logged before attempt 2's first row is not named; a call from before a respawn is not named; rows that carry no attempt behave as before; and the digest frame over two attempts.
A replay of this issue's own step (openspec.apply[4] of run_01M3GB3KN38YT9TZJ7KA7RQ5M0, 6,320 log rows): the old code named bash with running_for_s climbing for 133 s into attempt 2; the new code names nothing ("Thinking") at 10 s, 60 s and 133 s. The same replay through the deployed code in production's container gives the same answer.
The fast and full lanes pass.

Change record: openspec/changes/archive/2026-10-03-running-tool-of-newest-attempt.

Fixed and live on production (`9ae119c`, deployed 2026-10-03). **What changed** - `running_tool_call` takes only the newest attempt's own tool calls (the attempt the caller names, else the newest in the rows it is given), and none from before the newest budget respawn of that attempt. - The digest's step fold remembers the attempt the step is now in and passes it on, so a restart (or any new attempt) can no longer leave a dead session's pending call named as running. **How it was checked** - Unit tests: an attempt-1 pending call followed by attempt-2 events names nothing; attempt 2's own call is named; a call logged before attempt 2's first row is not named; a call from before a respawn is not named; rows that carry no attempt behave as before; and the digest frame over two attempts. - A replay of this issue's own step (`openspec.apply[4]` of `run_01M3GB3KN38YT9TZJ7KA7RQ5M0`, 6,320 log rows): the old code named `bash` with `running_for_s` climbing for 133 s into attempt 2; the new code names nothing ("Thinking") at 10 s, 60 s and 133 s. The same replay through the deployed code in production's container gives the same answer. - The fast and full lanes pass. Change record: `openspec/changes/archive/2026-10-03-running-tool-of-newest-attempt`.

cmoriarty closed this issue

2026-10-03 11:06:27 -04:00

cmoriarty referenced this issue from a commit

2026-10-03 11:10:21 -04:00

Archive six changes (#73, #113, #114, #125, #126, #127) [skip ci]

Rows
Columns

The spine names a dead attempt's tool call as running #73

Problem

Cause

Proposal

Done when