CI and the image publish each finish in under two minutes #44
Labels
No labels
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
cmoriarty/sports-breaker#44
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
A push to develop takes over five minutes to publish the image and over six for CI to report. Every deploy and every fix-forward waits on both. The goal is each workflow under two minutes, wall time.
What we measured (2026-10-29, Forgejo tasks API, successful runs only)
3e1e79d)The tasks API's
updated_atmoves after a run ends for older tasks, so durations over an hour were dropped as unreliable. Per-step times are not in the API and are the first thing to measure.Suspects (from reading the files; to be confirmed by profiling)
COPY backend/ /app/comes beforepip install -e /app,playwright install --with-deps chromium(apt packages plus a Chromium download) and the ffmpegapt-get install. So every backend change rebuilds the three heaviest layers, and the publish job pushes them all again. Installing the dependencies, Chromium and ffmpeg before copying the code should leave a code-only change with a few seconds of build and a small push.latest, or ci tests the pushed:<sha>image.pip install -e "backend[test]"andplaywright install chromium, with no cache.make test's build target, thennpm --prefix frontend installagain.make test'sfrontend/disttarget, then "Type check and bundle".npm ci, and one frontend build.What changes
docker build --progress=plainper-layer times, for three kinds of change: a backend-only change, a frontend-only change and a dependency change (pyproject.tomlorpackage-lock.json).latestand:<sha>are both pushed.Done when three consecutive backend-only pushes to develop show ci (both jobs, wall time) and publish-image each finishing in under two minutes, measured from the tasks API. A dependency change must still build and pass, and its times are recorded here too.
Where this stands (2026-10-29, after PR #49 merged as
8e6834b)Found on the way: publish never waited for the e2e.
d58803d,0e6bc84andc9631edpublishedlatestwith a red in-image e2e. Since PR #49, a red e2e keepslateston the last good image.Left for under two minutes
The in-image e2e is about 155–165 s of fixed legs, all in
backend/tests/e2e_click_to_play.py:Without these, publish can't get under two minutes.
Runner disk writeback: while the fast lane extracts its caches, about 750 MB of unwritten data drains at about 30 MB/s, and a concurrent image build's small code copy waits 25–49 s for it. Faster storage for
/var/lib/docker, or another disk cache mode on the hypervisor, would save 25–45 s per build.Runner housekeeping (corrected 2026-10-30): the runner's Docker holds about 100 GB, not 209. The earlier figure added build cache that shares its space with the image layers. 88 GB of it is 64 old
:<sha>tags that publish never removed. PR #51 removes them and stops new ones accumulating. Pullpython:3.12-slimnow and then so the base image doesn't go stale.Security: the runner's Docker daemon serves its API unencrypted on 0.0.0.0:2375. Anyone on the LAN can run root containers on that VM. Bind it to the socket only, or to TLS.
Done when, unchanged: three consecutive backend-only pushes with both workflows under two minutes.
PR #54 (2026-10-29): the e2e runs in about 30 s instead of about 2 min (measured in the image,
--network none). The fake servers' listen backlog of 5 was most of it.The second column is both jobs on the same push at once, run 293 on
d8a7264. Their combined writes hit the runner disk stalls in #52. That's the remaining gap to two minutes.Fixed on the way: an order-dependent flake. Host reputation stayed in memory between tests, so a channel test's "wrong game" rejection re-ordered the prewarm tests' ladders. The lane now resets it with each test's data directory (
d8a7264).Closing into #52. PR #49, #51 and #54 brought the fast lane to about 48 s and publish to about 125 s when each runs alone. What's left is the runner's disk stalling when both run at once, and #52 tracks that.