CI and the image publish: build once, cache the dependencies, gate latest on the e2e #49

Merged
cmoriarty merged 4 commits from ci/speed into develop 2026-10-29 20:40:11 -04:00
Owner

Issue #44: CI and the image publish, each under two minutes.

The fast lane is now under one minute. The service lane and publish are not under two: the in-image e2e alone takes 154–165 s on the runner. The four commits below remove everything else that could be removed from the workflow side. The rest is the e2e itself (test code, not touched here) and the runner's disk.

Before and after

Before: the five develop pushes 262/263 to 272/273 (2026-10-26 to 29). After: this PR's runs 276–282, each building as after a backend-only change (a temporary probe step, dropped from the branch). Step times come from the job view; the build, e2e and push times come from the step logs.

Job / step Before (median, range) After
Fast lane, wall 151 s (129–178) 45–48 s (cold cache: 87 s)
Set up Python (actions/setup-python) 53 s (36–71) removed: the job image's Python 3.12
Fresh venv, pip, Chromium 40 s (40–58) 3–5 s cache restore (venv + headless shell)
npm install + page build twice: in Pytest lane, then again once: "Type check and bundle" 8–10 s
Pytest lane 37 s (35–43), with npm + build 19–22 s
Service lane, wall (PR, main) 337 s (283–529) 182–215 s
Image build 163 s (121–251) 4 s alone; 17–51 s next to the fast lane (see Runner)
In-image e2e 160 s (154–217) 154–165 s
Publish, wall (develop) 230 s (186–328), not gated by the e2e estimated 175–225 s, gated by the e2e: not measured yet
Publish: set up job 29 s (22–55) —
Publish: build 128 s, shared with the service lane's build same as the service lane's: 4–51 s
Publish: push 59 s (58–64): ~1.4 GB of dependency layers per push estimated a few seconds: one ~2 MB code layer

Local docker build --progress=plain with a warm cache (Apple M3), before → after: backend-only change 44.6 s → 0.1 s, frontend-only 46.3 s → 3.0 s, pyproject.toml change 42.6 s → 21.4 s. On the runner, a cold build of the new Dockerfile (what the first push after a pyproject.toml change or a new ISO week costs) took 202 s (run 274).

Changes

  1. Dockerfile: dependencies before code. ffmpeg, then the Python dependencies from pyproject.toml alone, then Chromium, then the built page and the code. A code-only change rebuilds and pushes one ~2 MB layer instead of ~1.4 GB. pip and npm downloads sit in BuildKit cache mounts; npm ci; DEPS_REFRESH (the workflows pass the ISO week) rebuilds the dependency layers weekly, since they used to be refreshed on every build. The context leaves out openspec, docs, brand, .gitea, .claude.
  2. Fast lane: caches, one build. It uses the job image's Python 3.12 (Ubuntu 24.04's, 3.12.3) instead of actions/setup-python. The venv and Playwright's headless shell, and node_modules, come from actions/cache, keyed on the manifest hash, the Python or Node version and the ISO week. It installs only the headless shell, because every test launches headless=True. Restoring the full browser took longer than downloading it. make build runs once, and the pytest lane reuses it.
  3. One build per push, and the e2e gates latest. On a develop push, publish-image builds the image, runs the in-image e2e, then pushes :<sha> and latest. ci's service lane skips develop pushes and still runs on pull requests and main. Before this, a red e2e did not block latest: d58803d, 0e6bc84 and c9631ed were all published with a failed in-image e2e.
  4. --network none for the in-image e2e. The best-effort try of the real page now fails at once instead of waiting out Cloudflare: 30–33 s → 17–20 s.

Guarantees kept:

  • Publish still fails, and never skips, without a daemon or the registry secrets.
  • Pull requests never publish.
  • A develop push pushes both latest and :<sha>, and only after the e2e passes.
  • The fast lane still runs the full pytest lane, the frontend unit tests, and the type check plus bundle.

Not measured until this is merged

  • Publish on develop: its wall time, the push size of a code-only change, and that the service lane is skipped on a develop push (the if: was checked only on pull requests, where the job runs).
  • On a develop push, the e2e verdict now appears on publish-image's job ("Build the one container (linux/amd64), e2e inside it, push it to the registry"). ci shows only the fast lane there.

What still stands between this and two minutes

  • The e2e: 154–165 s on its own. Leg by leg on the runner:

    • container and app start: 6 s
    • real-page try: 17–20 s
    • first game: 17–19 s
    • games 1–7: 27–28 s
    • failure leg: 17 s
    • recording: 21–25 s
    • replay: 17–18 s
    • channel: 27 s

    Getting under two minutes needs changes in backend/tests/e2e_click_to_play.py. For example: skip the real-page try in CI, and run the channel leg (its own app and port) beside part (b).

  • Runner disk. BuildKit's layer commit waits for writeback. While the fast lane extracts its caches, about 750 MB of dirty pages drain at ~30 MB/s, so the 2 MB COPY backend/ takes 25–49 s. Without the fast lane it takes 3 s (run 281). Sequential writes on the same disk run at 485 MB/s; 4 KiB synced writes manage 4.5 MB/s.

  • Flake seen once (run 277): "'Hawks vs. Magic' is not shown on the page". The feed fetch has a 10 s timeout, and it was during one of those stalls. Now that the e2e gates latest, a flake like this holds a deploy until a re-run.

🤖 Generated with Claude Code

Issue #44: CI and the image publish, each under two minutes. **The fast lane is now under one minute. The service lane and publish are not under two: the in-image e2e alone takes 154–165 s on the runner.** The four commits below remove everything else that could be removed from the workflow side. The rest is the e2e itself (test code, not touched here) and the runner's disk. ## Before and after Before: the five develop pushes 262/263 to 272/273 (2026-10-26 to 29). After: this PR's runs 276–282, each building as after a backend-only change (a temporary probe step, dropped from the branch). Step times come from the job view; the build, e2e and push times come from the step logs. | Job / step | Before (median, range) | After | |---|---|---| | **Fast lane, wall** | **151 s** (129–178) | **45–48 s** (cold cache: 87 s) | | Set up Python (actions/setup-python) | 53 s (36–71) | removed: the job image's Python 3.12 | | Fresh venv, pip, Chromium | 40 s (40–58) | 3–5 s cache restore (venv + headless shell) | | npm install + page build | twice: in Pytest lane, then again | once: "Type check and bundle" 8–10 s | | Pytest lane | 37 s (35–43), with npm + build | 19–22 s | | **Service lane, wall** (PR, main) | **337 s** (283–529) | **182–215 s** | | Image build | 163 s (121–251) | 4 s alone; 17–51 s next to the fast lane (see Runner) | | In-image e2e | 160 s (154–217) | 154–165 s | | **Publish, wall** (develop) | **230 s** (186–328), not gated by the e2e | estimated 175–225 s, gated by the e2e: **not measured yet** | | Publish: set up job | 29 s (22–55) | — | | Publish: build | 128 s, shared with the service lane's build | same as the service lane's: 4–51 s | | Publish: push | 59 s (58–64): ~1.4 GB of dependency layers per push | estimated a few seconds: one ~2 MB code layer | Local `docker build --progress=plain` with a warm cache (Apple M3), before → after: backend-only change 44.6 s → 0.1 s, frontend-only 46.3 s → 3.0 s, `pyproject.toml` change 42.6 s → 21.4 s. On the runner, a cold build of the new Dockerfile (what the first push after a `pyproject.toml` change or a new ISO week costs) took 202 s (run 274). ## Changes 1. **Dockerfile: dependencies before code.** ffmpeg, then the Python dependencies from `pyproject.toml` alone, then Chromium, then the built page and the code. A code-only change rebuilds and pushes one ~2 MB layer instead of ~1.4 GB. pip and npm downloads sit in BuildKit cache mounts; `npm ci`; `DEPS_REFRESH` (the workflows pass the ISO week) rebuilds the dependency layers weekly, since they used to be refreshed on every build. The context leaves out `openspec`, `docs`, `brand`, `.gitea`, `.claude`. 2. **Fast lane: caches, one build.** It uses the job image's Python 3.12 (Ubuntu 24.04's, 3.12.3) instead of `actions/setup-python`. The venv and Playwright's headless shell, and `node_modules`, come from `actions/cache`, keyed on the manifest hash, the Python or Node version and the ISO week. It installs only the headless shell, because every test launches `headless=True`. Restoring the full browser took longer than downloading it. `make build` runs once, and the pytest lane reuses it. 3. **One build per push, and the e2e gates `latest`.** On a develop push, publish-image builds the image, runs the in-image e2e, then pushes `:<sha>` and `latest`. ci's service lane skips develop pushes and still runs on pull requests and main. Before this, a red e2e did **not** block `latest`: d58803d, 0e6bc84 and c9631ed were all published with a failed in-image e2e. 4. **`--network none` for the in-image e2e.** The best-effort try of the real page now fails at once instead of waiting out Cloudflare: 30–33 s → 17–20 s. Guarantees kept: - Publish still fails, and never skips, without a daemon or the registry secrets. - Pull requests never publish. - A develop push pushes both `latest` and `:<sha>`, and only after the e2e passes. - The fast lane still runs the full pytest lane, the frontend unit tests, and the type check plus bundle. ## Not measured until this is merged - Publish on develop: its wall time, the push size of a code-only change, and that the service lane is skipped on a develop push (the `if:` was checked only on pull requests, where the job runs). - On a develop push, the e2e verdict now appears on publish-image's job ("Build the one container (linux/amd64), e2e inside it, push it to the registry"). ci shows only the fast lane there. ## What still stands between this and two minutes - **The e2e: 154–165 s on its own.** Leg by leg on the runner: - container and app start: 6 s - real-page try: 17–20 s - first game: 17–19 s - games 1–7: 27–28 s - failure leg: 17 s - recording: 21–25 s - replay: 17–18 s - channel: 27 s Getting under two minutes needs changes in `backend/tests/e2e_click_to_play.py`. For example: skip the real-page try in CI, and run the channel leg (its own app and port) beside part (b). - **Runner disk.** BuildKit's layer commit waits for writeback. While the fast lane extracts its caches, about 750 MB of dirty pages drain at ~30 MB/s, so the 2 MB `COPY backend/` takes 25–49 s. Without the fast lane it takes 3 s (run 281). Sequential writes on the same disk run at 485 MB/s; 4 KiB synced writes manage 4.5 MB/s. - **Flake seen once (run 277):** "'Hawks vs. Magic' is not shown on the page". The feed fetch has a 10 s timeout, and it was during one of those stalls. Now that the e2e gates `latest`, a flake like this holds a deploy until a re-run. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The code was copied first, so every backend or frontend change reinstalled
the Python dependencies, Chromium with its system packages and ffmpeg, and
the publish job pushed all three again. On the runner that was 100 to 200 s
of build (pip 21-32 s, Chromium 40-59 s, ffmpeg 20-96 s, 6 s of export) and
about 60 s of push per change (runs 262-271, 2026-10-26 to 29).

The final stage now installs ffmpeg, then the dependencies from
pyproject.toml alone (an editable install of a placeholder package, which
only puts /app on the path), then Chromium, and copies the built page and
the code last. Measured locally with a warm cache (Apple M3, arm64):

  change            before   after
  backend only      44.6 s    0.1 s
  frontend only     46.3 s    3.0 s
  pyproject.toml    42.6 s   21.4 s

pip and npm keep their downloads in BuildKit cache mounts, npm ci installs
from the lockfile, and Playwright's apt lists are cleaned in its own layer.
DEPS_REFRESH (the workflows pass the ISO week) rebuilds the dependency
layers weekly, so the image does not keep a month-old ffmpeg or Chromium
just because no manifest changed. The build context leaves out the specs,
docs, workflows and .claude worktrees.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Of the fast lane's 2m42s-2m58s, a minute went to actions/setup-python
(36-71 s: a download, then copying a Python into an empty tool cache on
the runner's slow disk) and 40-58 s to a fresh venv, pip install and the
Chromium download. The page was built twice (make test's build target,
then "Type check and bundle") and npm installed twice.

The job now uses the job image's own Python 3.12 (Ubuntu 24.04's) and
restores backend/.venv, ~/.cache/ms-playwright and frontend/node_modules
with actions/cache, keyed on the manifest's hash, the Python or Node
version and the ISO week; a miss installs as before and saves. make build
runs the type check and the bundle first, and the pytest lane finds the
page up to date. The lane still runs the full pytest lane, the frontend
unit tests and the type check plus bundle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A red e2e never blocked latest: publish-image built and pushed on its own
while ci's service lane built the same image on the same daemon and ran
the e2e beside it. d58803d, 0e6bc84 and c9631ed all published latest with
a failed in-image e2e (2026-10-24 to 26).

On a push to develop, publish-image now builds the image, runs the e2e
inside it, and only then pushes the commit's tag and latest; a failure
stops the job before the push. ci's service lane skips develop pushes,
since publish-image runs the same build and check there, and still covers
pull requests and main. Publish still fails without a daemon or the
registry secrets, and pull requests still never publish.

The service lane's build and e2e are separate steps, so their times show
on their own, and it runs the image by the ID its own build wrote rather
than a shared tag another run could move. Both builds pass the same
platform and DEPS_REFRESH, so they share the runner's layer cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Probe: each ci run builds as after a backend-only change (temporary, for measuring)
All checks were successful
ci / Unit tests, type check, bundle (pull_request) Successful in 2m32s
ci / One container, e2e click-to-play inside the image (pull_request) Successful in 6m27s
bbd76254dd
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Probe: a second run, warm (temporary)
All checks were successful
ci / Unit tests, type check, bundle (pull_request) Successful in 1m5s
ci / One container, e2e click-to-play inside the image (pull_request) Successful in 3m44s
9316b96477
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cmoriarty force-pushed ci/speed from 9316b96477
All checks were successful
ci / Unit tests, type check, bundle (pull_request) Successful in 1m5s
ci / One container, e2e click-to-play inside the image (pull_request) Successful in 3m44s
to dd3b9d8bee
All checks were successful
ci / Unit tests, type check, bundle (pull_request) Successful in 1m26s
ci / One container, e2e click-to-play inside the image (pull_request) Successful in 3m2s
2026-10-29 20:08:36 -04:00
Compare
Probe: warm run 1 (temporary)
Some checks failed
ci / Unit tests, type check, bundle (pull_request) Successful in 48s
ci / One container, e2e click-to-play inside the image (pull_request) Failing after 1m2s
338fd40e01
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Probe: warm run 2 (temporary)
All checks were successful
ci / Unit tests, type check, bundle (pull_request) Successful in 48s
ci / One container, e2e click-to-play inside the image (pull_request) Successful in 3m35s
2f4cfa0551
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Probe: the runner's disk speed, warm run 3 (temporary)
All checks were successful
ci / Unit tests, type check, bundle (pull_request) Successful in 45s
ci / One container, e2e click-to-play inside the image (pull_request) Successful in 3m15s
784cbc2d53
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Probe: sample dirty memory during the build (temporary)
All checks were successful
ci / Unit tests, type check, bundle (pull_request) Successful in 45s
ci / One container, e2e click-to-play inside the image (pull_request) Successful in 3m21s
53f9f0ba59
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Probe: the image build alone, without the fast lane (temporary)
All checks were successful
ci / Unit tests, type check, bundle (pull_request) Has been skipped
ci / One container, e2e click-to-play inside the image (pull_request) Successful in 2m47s
13d5dfc77b
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cmoriarty force-pushed ci/speed from 13d5dfc77b
All checks were successful
ci / Unit tests, type check, bundle (pull_request) Has been skipped
ci / One container, e2e click-to-play inside the image (pull_request) Successful in 2m47s
to d621d66e02
All checks were successful
ci / Unit tests, type check, bundle (pull_request) Successful in 47s
ci / One container, e2e click-to-play inside the image (pull_request) Successful in 3m29s
2026-10-29 20:34:53 -04:00
Compare
cmoriarty merged commit 8e6834b810 into develop 2026-10-29 20:40:11 -04:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/sports-breaker!49
No description provided.