CI and the image publish each finish in under two minutes #44

Closed
opened 2026-10-29 18:31:43 -04:00 by cmoriarty · 3 comments
Owner

Why

A push to develop takes over five minutes to publish the image and over six for CI to report. Every deploy and every fix-forward waits on both. The goal is each workflow under two minutes, wall time.

What we measured (2026-10-29, Forgejo tasks API, successful runs only)

Job Workflow Median Worst Today (3e1e79d)
Build the one container (linux/amd64), push it to the registry publish-image.yml 3m25s 5m28s 5m28s
One container, e2e click-to-play inside the image ci.yml (service-lane) 4m31s 7m22s still running at filing
Unit tests, type check, bundle ci.yml (fast-lane) 2m42s (one reliable sample) 2m42s

The tasks API's updated_at moves after a run ends for older tasks, so durations over an hour were dropped as unreliable. Per-step times are not in the API and are the first thing to measure.

Suspects (from reading the files; to be confirmed by profiling)

  1. The Dockerfile's layer order defeats the cache. COPY backend/ /app/ comes before pip install -e /app, playwright install --with-deps chromium (apt packages plus a Chromium download) and the ffmpeg apt-get install. So every backend change rebuilds the three heaviest layers, and the publish job pushes them all again. Installing the dependencies, Chromium and ffmpeg before copying the code should leave a code-only change with a few seconds of build and a small push.
  2. The same image is built twice per push, at the same time. ci's service-lane and publish-image both build the Dockerfile on the runner host's one Docker daemon, and today they ran concurrently. Consider building once and running the in-image e2e against that image: either the publish job runs the e2e before pushing latest, or ci tests the pushed :<sha> image.
  3. The fast lane starts from nothing every run.
    • It makes a fresh venv, runs pip install -e "backend[test]" and playwright install chromium, with no cache.
    • The frontend is installed twice: make test's build target, then npm --prefix frontend install again.
    • It is built twice: make test's frontend/dist target, then "Type check and bundle".
    • Candidates: a pip, npm and Playwright cache (actions/cache, or a directory on the runner host), npm ci, and one frontend build.
  4. The push size. With suspect 1 fixed, a code-only change should push one small layer, not the Chromium and ffmpeg layers.
  5. The e2e itself. Container start plus a headless click-to-play; measure it separately from the build.

What changes

  • Profile first, and post the numbers here.
    • Per-step times for each job, from the step logs.
    • docker build --progress=plain per-layer times, for three kinds of change: a backend-only change, a frontend-only change and a dependency change (pyproject.toml or package-lock.json).
  • Then fix in order of measured cost until a backend-only push gets each workflow under two minutes.
  • Keep the guarantees the workflows give today:
    • a red e2e still blocks a broken image;
    • publish still fails, never skips, without a daemon or the secrets;
    • latest and :<sha> are both pushed.

Done when three consecutive backend-only pushes to develop show ci (both jobs, wall time) and publish-image each finishing in under two minutes, measured from the tasks API. A dependency change must still build and pass, and its times are recorded here too.

## Why A push to develop takes over five minutes to publish the image and over six for CI to report. Every deploy and every fix-forward waits on both. The goal is each workflow under **two minutes**, wall time. ## What we measured (2026-10-29, Forgejo tasks API, successful runs only) | Job | Workflow | Median | Worst | Today (3e1e79d) | |---|---|---|---|---| | Build the one container (linux/amd64), push it to the registry | publish-image.yml | 3m25s | 5m28s | 5m28s | | One container, e2e click-to-play inside the image | ci.yml (service-lane) | 4m31s | 7m22s | still running at filing | | Unit tests, type check, bundle | ci.yml (fast-lane) | 2m42s (one reliable sample) | | 2m42s | The tasks API's `updated_at` moves after a run ends for older tasks, so durations over an hour were dropped as unreliable. Per-step times are not in the API and are the first thing to measure. ## Suspects (from reading the files; to be confirmed by profiling) 1. **The Dockerfile's layer order defeats the cache.** `COPY backend/ /app/` comes before `pip install -e /app`, `playwright install --with-deps chromium` (apt packages plus a Chromium download) and the ffmpeg `apt-get install`. So every backend change rebuilds the three heaviest layers, and the publish job pushes them all again. Installing the dependencies, Chromium and ffmpeg before copying the code should leave a code-only change with a few seconds of build and a small push. 2. **The same image is built twice per push, at the same time.** ci's service-lane and publish-image both build the Dockerfile on the runner host's one Docker daemon, and today they ran concurrently. Consider building once and running the in-image e2e against that image: either the publish job runs the e2e before pushing `latest`, or ci tests the pushed `:<sha>` image. 3. **The fast lane starts from nothing every run.** - It makes a fresh venv, runs `pip install -e "backend[test]"` and `playwright install chromium`, with no cache. - The frontend is installed twice: `make test`'s build target, then `npm --prefix frontend install` again. - It is built twice: `make test`'s `frontend/dist` target, then "Type check and bundle". - Candidates: a pip, npm and Playwright cache (actions/cache, or a directory on the runner host), `npm ci`, and one frontend build. 4. **The push size.** With suspect 1 fixed, a code-only change should push one small layer, not the Chromium and ffmpeg layers. 5. **The e2e itself.** Container start plus a headless click-to-play; measure it separately from the build. ## What changes - **Profile first, and post the numbers here.** - Per-step times for each job, from the step logs. - `docker build --progress=plain` per-layer times, for three kinds of change: a backend-only change, a frontend-only change and a dependency change (`pyproject.toml` or `package-lock.json`). - **Then fix in order of measured cost** until a backend-only push gets each workflow under two minutes. - **Keep the guarantees the workflows give today:** - a red e2e still blocks a broken image; - publish still fails, never skips, without a daemon or the secrets; - `latest` and `:<sha>` are both pushed. **Done when** three consecutive backend-only pushes to develop show ci (both jobs, wall time) and publish-image each finishing in under two minutes, measured from the tasks API. A dependency change must still build and pass, and its times are recorded here too.
Author
Owner

Where this stands (2026-10-29, after PR #49 merged as 8e6834b)

Job Before After
CI fast lane (unit tests, type check, bundle) 151 s median 45–48 s on PR runs with a warm cache; 89 s on the first develop run, whose cache was empty
CI on a develop push, wall about 5.6 min (the service lane) the fast lane alone: its in-image e2e moved into publish
Publish 230 s median, not gated by the e2e 230 s on the first develop run (build, e2e, push), now gated
Image build after a code-only change 163 s 4–50 s
Image push after a code-only change about 1.4 GB one layer of about 2 MB expected (the first develop push uploaded the reordered dependency layers once)

Found on the way: publish never waited for the e2e. d58803d, 0e6bc84 and c9631ed published latest with a red in-image e2e. Since PR #49, a red e2e keeps latest on the last good image.

Left for under two minutes

  • The in-image e2e is about 155–165 s of fixed legs, all in backend/tests/e2e_click_to_play.py:

    • skip the real-page attempt in CI (17–20 s);
    • run the channel leg (27 s, its own app and port) alongside part (b);
    • look for other legs that wait on real clocks.

    Without these, publish can't get under two minutes.

  • Runner disk writeback: while the fast lane extracts its caches, about 750 MB of unwritten data drains at about 30 MB/s, and a concurrent image build's small code copy waits 25–49 s for it. Faster storage for /var/lib/docker, or another disk cache mode on the hypervisor, would save 25–45 s per build.

  • Runner housekeeping (corrected 2026-10-30): the runner's Docker holds about 100 GB, not 209. The earlier figure added build cache that shares its space with the image layers. 88 GB of it is 64 old :<sha> tags that publish never removed. PR #51 removes them and stops new ones accumulating. Pull python:3.12-slim now and then so the base image doesn't go stale.

  • Security: the runner's Docker daemon serves its API unencrypted on 0.0.0.0:2375. Anyone on the LAN can run root containers on that VM. Bind it to the socket only, or to TLS.

Done when, unchanged: three consecutive backend-only pushes with both workflows under two minutes.

## Where this stands (2026-10-29, after PR #49 merged as 8e6834b) | Job | Before | After | |---|---|---| | CI fast lane (unit tests, type check, bundle) | 151 s median | 45–48 s on PR runs with a warm cache; 89 s on the first develop run, whose cache was empty | | CI on a develop push, wall | about 5.6 min (the service lane) | the fast lane alone: its in-image e2e moved into publish | | Publish | 230 s median, **not gated** by the e2e | 230 s on the first develop run (build, e2e, push), now gated | | Image build after a code-only change | 163 s | 4–50 s | | Image push after a code-only change | about 1.4 GB | one layer of about 2 MB expected (the first develop push uploaded the reordered dependency layers once) | **Found on the way:** publish never waited for the e2e. d58803d, 0e6bc84 and c9631ed published `latest` with a red in-image e2e. Since PR #49, a red e2e keeps `latest` on the last good image. ## Left for under two minutes - **The in-image e2e is about 155–165 s of fixed legs**, all in `backend/tests/e2e_click_to_play.py`: - skip the real-page attempt in CI (17–20 s); - run the channel leg (27 s, its own app and port) alongside part (b); - look for other legs that wait on real clocks. Without these, publish can't get under two minutes. - **Runner disk writeback:** while the fast lane extracts its caches, about 750 MB of unwritten data drains at about 30 MB/s, and a concurrent image build's small code copy waits 25–49 s for it. Faster storage for `/var/lib/docker`, or another disk cache mode on the hypervisor, would save 25–45 s per build. - **Runner housekeeping** (corrected 2026-10-30): the runner's Docker holds about 100 GB, not 209. The earlier figure added build cache that shares its space with the image layers. 88 GB of it is 64 old `:<sha>` tags that publish never removed. PR #51 removes them and stops new ones accumulating. Pull `python:3.12-slim` now and then so the base image doesn't go stale. - **Security:** the runner's Docker daemon serves its API **unencrypted on 0.0.0.0:2375**. Anyone on the LAN can run root containers on that VM. Bind it to the socket only, or to TLS. **Done when**, unchanged: three consecutive backend-only pushes with both workflows under two minutes.
Author
Owner

PR #54 (2026-10-29): the e2e runs in about 30 s instead of about 2 min (measured in the image, --network none). The fake servers' listen backlog of 5 was most of it.

Job Alone Running alongside the other job
Container build + e2e, PR service lane 120 s
Publish on develop 125 s 260 s
Fast lane 48 s 124 s

The second column is both jobs on the same push at once, run 293 on d8a7264. Their combined writes hit the runner disk stalls in #52. That's the remaining gap to two minutes.

Fixed on the way: an order-dependent flake. Host reputation stayed in memory between tests, so a channel test's "wrong game" rejection re-ordered the prewarm tests' ladders. The lane now resets it with each test's data directory (d8a7264).

**PR #54 (2026-10-29): the e2e runs in about 30 s instead of about 2 min** (measured in the image, `--network none`). The fake servers' listen backlog of 5 was most of it. | Job | Alone | Running alongside the other job | |---|---|---| | Container build + e2e, PR service lane | 120 s | | | Publish on develop | 125 s | 260 s | | Fast lane | 48 s | 124 s | The second column is both jobs on the same push at once, run 293 on d8a7264. Their combined writes hit the runner disk stalls in #52. That's the remaining gap to two minutes. **Fixed on the way:** an order-dependent flake. Host reputation stayed in memory between tests, so a channel test's "wrong game" rejection re-ordered the prewarm tests' ladders. The lane now resets it with each test's data directory (d8a7264).
Author
Owner

Closing into #52. PR #49, #51 and #54 brought the fast lane to about 48 s and publish to about 125 s when each runs alone. What's left is the runner's disk stalling when both run at once, and #52 tracks that.

Closing into #52. PR #49, #51 and #54 brought the fast lane to about 48 s and publish to about 125 s when each runs alone. What's left is the runner's disk stalling when both run at once, and #52 tracks that.
Sign in to join this conversation.
No labels
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/sports-breaker#44
No description provided.