cmoriarty/braid
1
0
Fork 0
Code Issues 15 Pull requests Projects Releases Packages 1 Wiki Activity Actions

The agent-view spec's "carries its reveal across settling" test leaves 2 to 18 characters to carry, and fails under load #135

New issue
Closed
opened 2026-10-03 00:43:41 -04:00 by cmoriarty · 1 comment
cmoriarty commented 2026-10-03 00:43:41 -04:00
Owner
Copy link

ui/e2e/agent-view.spec.ts, "a streaming answer › carries its reveal across settling when the part settles mid-reveal" (@slow, about line 202), depends on timing it does not control.

Seen (2026-10-03)

  • It failed once in a full lane (./tools/test.sh full, Playwright at 4 workers) with "the reveal had already finished; nothing was carried": 1,167 of 1,167 characters were visible at the hand-over from the live answer to the settled one.
  • On a quiet machine it passed 8 of 8 runs (npx playwright test e2e/agent-view.spec.ts:202 --repeat-each=8 --workers=1), and the #130 session never saw it fail.
  • A probe of the same scenario (POST /_fake/scenario?settle=0.02, the spec's sampling) measured the characters still unrevealed at the hand-over. Over 16 runs it was between 2 and 18, and the longest frame was 17 to 19 ms. Builds with and without the #133 list fix had the same margins.

Why the margin is so small

The fixture answer (ANSWER_TEXT in ui/dev/fakeosfd.py) is 1,299 characters, streamed in bursts of 80 every 250 ms, so its last burst carries only 19 (1,299 mod 80). With settle=0.02, the settled event goes out on the other stream 20 ms after that burst. A frame of about 30 ms or more, or a settled event held up by CPU load, lets the reveal finish first, and the test fails.

It is also weaker than it reads. With at most 18 characters left, the second assertion (what was left still arrived at most 24 characters a frame) cannot fail, because the whole remainder fits in one frame's allowance.

Wanted

The hand-over lands mid-reveal by construction, not by a few characters, and the test keeps its assertions: the reveal is carried across settling, at most 24 characters a frame. Directions, not decided: settle the part before its last burst arrives (ui/src/lib/reveal.ts already handles a part that settles before its last burst), or make the scenario's last burst large. Changing ANSWER_TEXT is risky, because other specs depend on it.

Verify under load the way #130 did: seven busy loops and --repeat-each=16 --workers=1, before and after.

`ui/e2e/agent-view.spec.ts`, "a streaming answer › carries its reveal across settling when the part settles mid-reveal" (`@slow`, about line 202), depends on timing it does not control. **Seen (2026-10-03)** - It failed once in a full lane (`./tools/test.sh full`, Playwright at 4 workers) with "the reveal had already finished; nothing was carried": 1,167 of 1,167 characters were visible at the hand-over from the live answer to the settled one. - On a quiet machine it passed 8 of 8 runs (`npx playwright test e2e/agent-view.spec.ts:202 --repeat-each=8 --workers=1`), and the #130 session never saw it fail. - A probe of the same scenario (`POST /_fake/scenario?settle=0.02`, the spec's sampling) measured the characters still unrevealed at the hand-over. Over 16 runs it was between 2 and 18, and the longest frame was 17 to 19 ms. Builds with and without the #133 list fix had the same margins. **Why the margin is so small** The fixture answer (`ANSWER_TEXT` in `ui/dev/fakeosfd.py`) is 1,299 characters, streamed in bursts of 80 every 250 ms, so its last burst carries only 19 (1,299 mod 80). With `settle=0.02`, the settled event goes out on the other stream 20 ms after that burst. A frame of about 30 ms or more, or a settled event held up by CPU load, lets the reveal finish first, and the test fails. It is also weaker than it reads. With at most 18 characters left, the second assertion (what was left still arrived at most 24 characters a frame) cannot fail, because the whole remainder fits in one frame's allowance. **Wanted** The hand-over lands mid-reveal by construction, not by a few characters, and the test keeps its assertions: the reveal is carried across settling, at most 24 characters a frame. Directions, not decided: settle the part before its last burst arrives (`ui/src/lib/reveal.ts` already handles a part that settles before its last burst), or make the scenario's last burst large. Changing `ANSWER_TEXT` is risky, because other specs depend on it. Verify under load the way #130 did: seven busy loops and `--repeat-each=16 --workers=1`, before and after.
cmoriarty commented 2026-10-03 01:49:24 -04:00
Author
Owner
Copy link

Shipped in 0059708 and archived in ba2d22a. Both commits are [skip ci]: the change is test-only, so there was nothing to deploy.

What changed

  • The test resets the scenario with settle=-0.375. The answer settles 375 ms before its last burst, so its last two bursts (99 characters) are never streamed. The settled answer always holds text the live one never had, so the hand-over is mid-reveal by construction. The fixture already behaved this way for a negative settle; only its comment in ui/dev/fakeosfd.py changed.
  • Its first check now asks for more than 24 characters left at the hand-over. A remainder of 24 or fewer passes the 24-a-frame pace check even when it is drawn at once. The failure message gives the numbers, for example "1167 of 1167 characters were visible at the hand-over: too few were left to tell a carried reveal from one shown at once". The pace check is unchanged.
  • The test-lanes spec has a new requirement: "The agent-view spec's hand-over is mid-reveal by construction".

Measured

Characters still to reveal at the hand-over, from a probe of the spec's own sampling:

settle Machine Runs Left Runs that fail
0.02 (before) quiet 4 0–12 1
0.02 (before) seven busy loops 16 0–18 1
−0.375 (after) quiet 4 119–130 0
−0.375 (after) seven busy loops 32 112–137 0
  • On the wire, the answer's last delta now carries 1,200 characters. Its settled event, 120 to 160 ms later, carries all 1,299.
  • Under seven busy loops the test passed 16 of 16, both before and after. With the old setting, the new first check fails 4 of 4.
  • After the hand-over, the rest took 24 to 28 frames, at most 10 characters a frame.
  • The fast lane passed in 243 s. The full lane passed on a quiet machine in 157 s. A first full lane that overlapped another session's full lane (load 54 to 66) failed 4 backend timing tests against a real opencode serve; all 4 pass alone.
Shipped in 0059708 and archived in ba2d22a. Both commits are `[skip ci]`: the change is test-only, so there was nothing to deploy. **What changed** - The test resets the scenario with `settle=-0.375`. The answer settles 375 ms before its last burst, so its last two bursts (99 characters) are never streamed. The settled answer always holds text the live one never had, so the hand-over is mid-reveal by construction. The fixture already behaved this way for a negative `settle`; only its comment in `ui/dev/fakeosfd.py` changed. - Its first check now asks for more than 24 characters left at the hand-over. A remainder of 24 or fewer passes the 24-a-frame pace check even when it is drawn at once. The failure message gives the numbers, for example "1167 of 1167 characters were visible at the hand-over: too few were left to tell a carried reveal from one shown at once". The pace check is unchanged. - The `test-lanes` spec has a new requirement: "The agent-view spec's hand-over is mid-reveal by construction". **Measured** Characters still to reveal at the hand-over, from a probe of the spec's own sampling: | `settle` | Machine | Runs | Left | Runs that fail | |---|---|---|---|---| | 0.02 (before) | quiet | 4 | 0–12 | 1 | | 0.02 (before) | seven busy loops | 16 | 0–18 | 1 | | −0.375 (after) | quiet | 4 | 119–130 | 0 | | −0.375 (after) | seven busy loops | 32 | 112–137 | 0 | - On the wire, the answer's last delta now carries 1,200 characters. Its settled event, 120 to 160 ms later, carries all 1,299. - Under seven busy loops the test passed 16 of 16, both before and after. With the old setting, the new first check fails 4 of 4. - After the hand-over, the rest took 24 to 28 frames, at most 10 characters a frame. - The fast lane passed in 243 s. The full lane passed on a quiet machine in 157 s. A first full lane that overlapped another session's full lane (load 54 to 66) failed 4 backend timing tests against a real `opencode serve`; all 4 pass alone.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
archive/prefect-kestra-experiments
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cmoriarty/braid#135
No description provided.