Audit test suites for performance #39

New issue

Closed

opened 2026-09-15 11:18:01 -04:00 by cmoriarty · 1 comment

cmoriarty commented

2026-09-15 11:18:01 -04:00

Owner

I want to ensure we have a lightning fast test suite that agents can run often. This will likely be easier to accomplish after #30 is completed to allow for keyboard control of every aspect of Braid.

Ultimately, we should have a lightning fast test suite that tests unit tests, and UI tests. Any tests marks slow (over 5s) should be audited and improved if possible, but if they remain slow then they become part of the full regression test that is only done prior to merging/pushing a change (not done during development).

In addition, the final deployment tends to take a while, so if there any ways to speed that up as well, we should do it as part of this work.

I want to ensure we have a lightning fast test suite that agents can run often. This will likely be easier to accomplish after #30 is completed to allow for keyboard control of every aspect of Braid. Ultimately, we should have a lightning fast test suite that tests unit tests, and UI tests. Any tests marks slow (over 5s) should be audited and improved if possible, but if they remain slow then they become part of the full regression test that is only done prior to merging/pushing a change (not done during development). In addition, the final deployment tends to take a while, so if there any ways to speed that up as well, we should do it as part of this work.

cmoriarty referenced this issue from a commit

2026-09-26 18:32:35 -04:00

feat(ui): every console action has a key and a name, and the tests split into a fast lane and a full one (#30, #39)

cmoriarty referenced this issue from a pull request that will close it,

2026-09-26 18:33:32 -04:00

feat(ui): every console action has a key and a name, and the tests split into a fast lane and a full one (#30, #39) #58

cmoriarty referenced this issue

2026-09-26 18:39:03 -04:00

follow-the-tail browser specs time out: the run they open shows no transcript #59

cmoriarty closed this issue

2026-09-26 18:40:10 -04:00

cmoriarty commented

2026-09-26 18:41:42 -04:00

Author

Owner

Shipped in #58 (commit c88c254) and archived as openspec/changes/archive/2026-09-26-keyboard-commands-and-fast-tests. The behaviour is specified in openspec/specs/test-lanes and openspec/specs/deployment.

Test lanes

./tools/test.sh is the fast lane, for use while developing. It runs the backend unit tests on every core (pytest-xdist), the UI type check, vitest and build, and the Playwright specs against the fixture backend, leaving out the slow tests. ./tools/test.sh full runs everything before a merge or push, including the @live specs when OSF_URL is set.
Times: backend 344 s → 30 s. Browser specs 61 s → 17 s in the fast lane. The whole fast lane now takes 57 s (about 7 min before); the full lane takes 110 s.
Made faster: five liveness tests waited out a real 30 s redo timeout against a fake executor that never stops, 190 s in all. Five executor tests waited on real 5 s and 3 s windows.
Marked slow: 14 Playwright specs whose time is the thing they measure (streaming and layout observation windows), tagged @slow. The two real-osfd specs are tagged @live.
The 5 s rule: tests/fast_lane_budget.py names every unmarked backend test over 5 s at the end of each run.
Also fixed: a settings spec that failed on the second run against a fixture server left running.

Deploy

In run #25, every runtime layer rebuilt (apt, the Node tools, Chromium and uv sync, ~150 s), and the push re-sent ~1 GB (2 m 14 s). The cause: ARG BRAID_COMMIT was declared before the RUN steps, and Docker passes a build arg to each later RUN, so every commit was a cache miss for all of them. It now comes after the last RUN, and the Python dependencies install before COPY src.
Tested with two local builds: a new commit alone rebuilt in 0.5 s with every layer cached; a source change rebuilt in 1.5 s.
An idle production is now confirmed in about 5 s instead of a fixed 30 s, and the new commit is polled every 2 s instead of 10.

Follow-up: #59, the follow-the-tail live specs, which were already failing before this change.

Shipped in #58 (commit c88c254) and archived as `openspec/changes/archive/2026-09-26-keyboard-commands-and-fast-tests`. The behaviour is specified in `openspec/specs/test-lanes` and `openspec/specs/deployment`. **Test lanes** - `./tools/test.sh` is the fast lane, for use while developing. It runs the backend unit tests on every core (pytest-xdist), the UI type check, vitest and build, and the Playwright specs against the fixture backend, leaving out the slow tests. `./tools/test.sh full` runs everything before a merge or push, including the `@live` specs when `OSF_URL` is set. - Times: backend 344 s → 30 s. Browser specs 61 s → 17 s in the fast lane. The whole fast lane now takes **57 s** (about 7 min before); the full lane takes 110 s. - **Made faster:** five liveness tests waited out a real 30 s redo timeout against a fake executor that never stops, 190 s in all. Five executor tests waited on real 5 s and 3 s windows. - **Marked slow:** 14 Playwright specs whose time is the thing they measure (streaming and layout observation windows), tagged `@slow`. The two real-osfd specs are tagged `@live`. - **The 5 s rule:** `tests/fast_lane_budget.py` names every unmarked backend test over 5 s at the end of each run. - **Also fixed:** a settings spec that failed on the second run against a fixture server left running. **Deploy** - In run #25, every runtime layer rebuilt (apt, the Node tools, Chromium and `uv sync`, ~150 s), and the push re-sent ~1 GB (2 m 14 s). The cause: `ARG BRAID_COMMIT` was declared before the `RUN` steps, and Docker passes a build arg to each later `RUN`, so every commit was a cache miss for all of them. It now comes after the last `RUN`, and the Python dependencies install before `COPY src`. - Tested with two local builds: a new commit alone rebuilt in 0.5 s with every layer cached; a source change rebuilt in 1.5 s. - An idle production is now confirmed in about 5 s instead of a fixed 30 s, and the new commit is polled every 2 s instead of 10. **Follow-up:** #59, the `follow-the-tail` live specs, which were already failing before this change.

cmoriarty referenced this issue

2026-09-26 22:06:53 -04:00

Agents run the slow test suite: they are never told the project's fast test command #65

cmoriarty referenced this issue

2026-09-26 22:11:50 -04:00

fix(agents): agents run the project's own fast test command, and Braid's AGENTS.md block stays current (#65) #66