CI runs the fast lane on every push, beside the deploy #231

Closed
opened 2026-10-20 19:23:17 -04:00 by cmoriarty · 2 comments
Owner

Tier 3 of the test-suite speed-up (#227). deploy.yml builds the image and runs braid-selfcheck only; nothing runs the test suite on a push, so main is red at times (#227 found test_spec_limits failing on main) and a developer finds out on the next full lane.

Proposal: a test.yml workflow on push and pull request that runs ./tools/test.sh (the fast lane: backend unit tests, UI types, vitest, build, the browser specs against the fixture, the smoke simulated runs). It needs a runner with uv, Node, the Playwright browsers and the pinned opencode, or a job container that has them; not the production host, which the deploy's runner is. The soundcheck runner (strongbad) is throttled to about half clock, so the job's time is measured before it is made a required check. The deploy keeps running as it does; this is a parallel signal first.

Tier 3 of the test-suite speed-up (#227). deploy.yml builds the image and runs braid-selfcheck only; nothing runs the test suite on a push, so main is red at times (#227 found test_spec_limits failing on main) and a developer finds out on the next full lane. Proposal: a `test.yml` workflow on push and pull request that runs `./tools/test.sh` (the fast lane: backend unit tests, UI types, vitest, build, the browser specs against the fixture, the smoke simulated runs). It needs a runner with uv, Node, the Playwright browsers and the pinned opencode, or a job container that has them; not the production host, which the deploy's runner is. The soundcheck runner (strongbad) is throttled to about half clock, so the job's time is measured before it is made a required check. The deploy keeps running as it does; this is a parallel signal first.
Author
Owner

Shipped: .forgejo/workflows/test.yml runs the fast lane on every push and pull request on the instance runner (ubuntu-latest, the Playwright v1.64.1-noble image plus uv, git from the git-core PPA and tini), as a commit status beside the deploy; the deploy does not wait for it. It is a signal, not a required check, until its time on the runner is known over a few runs.

The first runs were worth more than the workflow. Run 173 on the runner, 4,662 passed and 19 failed, by cause:

  • Load: -n auto took the runner's 24 vCPUs and the timing tests starved; now 6 pytest workers, 4 browser workers (OSF_E2E_WORKERS, new), 2 simulated runs at a time.
  • root: three tests that need an unwritable path skip as root, as two already did.
  • git 2.43: merge-tree --write-tree with trees for sides answers rc 1 and an empty tree (2.45+ accepts trees); osf.git now raises on the empty tree instead of failing later in diff-tree, and the job installs a current git.
  • SQLite 3.45: DROP COLUMN in the migration-17 tests' set-up trips on the comments in schema.sql; those four skip below 3.46.
  • dash, a real bug: /bin/sh -c "sleep 30" forks sleep on dash where macOS's sh execs it, so a Stop's SIGTERM to the shell left the command running with the step's pipes open and the step could not end until the command did. Production's image is Debian with the same dash. Each scripted step now runs in its own session and a stop or timeout kills the group (TERM, then KILL after 5 s); test_interrupted.py has the test that would have caught it. A test also wrote $RANDOM, which dash has not.
  • No reaper: the job container's PID 1 never reaps, so a process the lane killed stayed a zombie and five tests saw it "still running". options: --init is not applied by the runner and dumb-init below PID 1 does not become a subreaper; tini -s does, so the lane runs under it.

Where it stands (runs 178 and 179): backend 4,675 passed in about 200 s with six workers; UI green; browser specs 352 passed and the same two failed on both runs, follow-transition.spec.ts:758 (a click while the highlight travels) and keyboard.spec.ts:367 (braid.run("tab.close"), a 10 s predicate), with four workers in 7.8 min and two in 12.2 min. Both pass on the laptops, so they are the runner's speed, not a regression; the lane stops at a red step, so the simulated runs have not yet run there. The issue stays open for that: make those two specs hold on a half-clock machine (or keep the window-measuring specs out of CI), then watch one green run and record the job's time. The workflow is live on every push meanwhile, as a signal.

Shipped: `.forgejo/workflows/test.yml` runs the fast lane on every push and pull request on the instance runner (`ubuntu-latest`, the Playwright `v1.64.1-noble` image plus uv, git from the git-core PPA and `tini`), as a commit status beside the deploy; the deploy does not wait for it. It is a signal, not a required check, until its time on the runner is known over a few runs. The first runs were worth more than the workflow. Run 173 on the runner, 4,662 passed and 19 failed, by cause: - **Load**: `-n auto` took the runner's 24 vCPUs and the timing tests starved; now 6 pytest workers, 4 browser workers (`OSF_E2E_WORKERS`, new), 2 simulated runs at a time. - **root**: three tests that need an unwritable path skip as root, as two already did. - **git 2.43**: `merge-tree --write-tree` with trees for sides answers rc 1 and an empty tree (2.45+ accepts trees); `osf.git` now raises on the empty tree instead of failing later in `diff-tree`, and the job installs a current git. - **SQLite 3.45**: `DROP COLUMN` in the migration-17 tests' set-up trips on the comments in `schema.sql`; those four skip below 3.46. - **dash, a real bug**: `/bin/sh -c "sleep 30"` forks `sleep` on dash where macOS's sh execs it, so a Stop's SIGTERM to the shell left the command running with the step's pipes open and the step could not end until the command did. Production's image is Debian with the same dash. Each scripted step now runs in its own session and a stop or timeout kills the group (TERM, then KILL after 5 s); `test_interrupted.py` has the test that would have caught it. A test also wrote `$RANDOM`, which dash has not. - **No reaper**: the job container's PID 1 never reaps, so a process the lane killed stayed a zombie and five tests saw it "still running". `options: --init` is not applied by the runner and `dumb-init` below PID 1 does not become a subreaper; `tini -s` does, so the lane runs under it. **Where it stands (runs 178 and 179):** backend 4,675 passed in about 200 s with six workers; UI green; browser specs 352 passed and the same two failed on both runs, `follow-transition.spec.ts:758` (a click while the highlight travels) and `keyboard.spec.ts:367` (`braid.run("tab.close")`, a 10 s predicate), with four workers in 7.8 min and two in 12.2 min. Both pass on the laptops, so they are the runner's speed, not a regression; the lane stops at a red step, so the simulated runs have not yet run there. The issue stays open for that: make those two specs hold on a half-clock machine (or keep the window-measuring specs out of CI), then watch one green run and record the job's time. The workflow is live on every push meanwhile, as a signal.
Author
Owner

Green end to end on run 184 (23 minutes on the runner: backend 4,717 passed in 235 s, 352 browser specs in 733 s with two workers, the seven smoke simulated runs about 5 minutes two at a time) and merged to main (2d16333); the change is archived.

What the last runs added to the list above:

  • The two specs that failed on both earlier runs: follow-transition's click-while-moving test measures the row's 150 ms fade and is tagged @slow as a timing measurement (it stays in the full lane); keyboard's tab.close test sent go.runs while the run the close landed on was still loading, and now waits for the run bar first.
  • review-loop's round-two fade-in samples the gate's travel frame by frame and wants more than three positions; the runner drew three once in four runs, so it is @slow too.
  • The simulated runs were refused at admission without the OpenSpec CLI; the job installs @fission-ai/openspec@1.14.1, the Dockerfile's version.

The job is a status beside the deploy's on every push and pull request, not a required check; the workflow's header records the time, and making it a gate is a separate decision on that figure.

Green end to end on run 184 (23 minutes on the runner: backend 4,717 passed in 235 s, 352 browser specs in 733 s with two workers, the seven smoke simulated runs about 5 minutes two at a time) and merged to main (`2d16333`); the change is archived. What the last runs added to the list above: - The two specs that failed on both earlier runs: `follow-transition`'s click-while-moving test measures the row's 150 ms fade and is tagged `@slow` as a timing measurement (it stays in the full lane); `keyboard`'s tab.close test sent `go.runs` while the run the close landed on was still loading, and now waits for the run bar first. - `review-loop`'s round-two fade-in samples the gate's travel frame by frame and wants more than three positions; the runner drew three once in four runs, so it is `@slow` too. - The simulated runs were refused at admission without the OpenSpec CLI; the job installs `@fission-ai/openspec@1.14.1`, the Dockerfile's version. The job is a status beside the deploy's on every push and pull request, not a required check; the workflow's header records the time, and making it a gate is a separate decision on that figure.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#231
No description provided.