The runner keeps no copy of the images it builds, and trims week-old build cache #51

Merged
cmoriarty merged 2 commits from ci/runner-cleanup into develop 2026-10-29 21:38:49 -04:00
Owner

Part of #44.

The runner's Docker held about 100 GB on 2026-10-30. 88 GB of it was 64 local sports-breaker:<sha> tags: publish tagged, pushed and kept every image. Since #49 a push adds about 2 MB, but each week's dependency refresh (DEPS_REFRESH) builds a new set of dependency layers, about 1.5 GB, and the old tags kept every past week's set. Left alone, that comes to about 80 GB a year.

Changes

  • publish-image: once the push is done, the job removes:

    • this repository's local tags (the registry keeps every one, so rollbacks are unaffected);
    • untagged images more than a day old (a day, so a PR's untagged image isn't removed before its e2e runs);
    • build cache unused for a week.

    The step runs after a failure too.

  • ci service lane: removes its untagged image after the e2e. It doesn't use -f, so an image another build has since tagged stays.

What to expect

  • The first develop push after this merges removes the 64 old tags. Most of their layers are shared with build-cache records last used this week, so those drain over the following week as until=168h reaches them.
  • After that, the runner should hold one or two weeks of dependency layers plus small entries: a few GB.
  • The build-cache prune covers the whole daemon, so other repositories on this runner lose cache entries they haven't used for a week too.

The next build's cache hits come from the build cache, not from the removed tags, so build times shouldn't change. The first develop run will confirm it.

Also: timeouts for a hung lane (a1075ec). Run 285's pytest lane sat 25 minutes on idle xdist workers, with no browser or server under them.

  • make test now passes --timeout=120. pytest-timeout is already a test dependency, and the slowest test takes about 13 s. A hung test now fails with its stack, which shows which test it was.
  • The fast-lane job stops at 10 minutes, instead of the runner's three-hour default.

🤖 Generated with Claude Code

Part of #44. The runner's Docker held about 100 GB on 2026-10-30. 88 GB of it was 64 local `sports-breaker:<sha>` tags: publish tagged, pushed and kept every image. Since #49 a push adds about 2 MB, but each week's dependency refresh (`DEPS_REFRESH`) builds a new set of dependency layers, about 1.5 GB, and the old tags kept every past week's set. Left alone, that comes to about 80 GB a year. **Changes** - **publish-image:** once the push is done, the job removes: - this repository's local tags (the registry keeps every one, so rollbacks are unaffected); - untagged images more than a day old (a day, so a PR's untagged image isn't removed before its e2e runs); - build cache unused for a week. The step runs after a failure too. - **ci service lane:** removes its untagged image after the e2e. It doesn't use `-f`, so an image another build has since tagged stays. **What to expect** - The first develop push after this merges removes the 64 old tags. Most of their layers are shared with build-cache records last used this week, so those drain over the following week as `until=168h` reaches them. - After that, the runner should hold one or two weeks of dependency layers plus small entries: a few GB. - The build-cache prune covers the whole daemon, so other repositories on this runner lose cache entries they haven't used for a week too. The next build's cache hits come from the build cache, not from the removed tags, so build times shouldn't change. The first develop run will confirm it. **Also: timeouts for a hung lane** (a1075ec). Run 285's pytest lane sat 25 minutes on idle xdist workers, with no browser or server under them. - `make test` now passes `--timeout=120`. `pytest-timeout` is already a test dependency, and the slowest test takes about 13 s. A hung test now fails with its stack, which shows which test it was. - The fast-lane job stops at 10 minutes, instead of the runner's three-hour default. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The runner keeps no copy of the images it builds, and trims week-old build cache
Some checks failed
ci / One container, e2e click-to-play inside the image (pull_request) Successful in 3m49s
ci / Unit tests, type check, bundle (pull_request) Has been cancelled
8c0f064d36
publish-image removes this repository's local tags once the push is done
(the registry keeps every tag for rollbacks), untagged images more than a
day old, and build cache unused for a week. The ci service lane removes
its untagged image after its e2e. The 64 :<sha> tags left on the runner
since 2026-10-23 held 88 GB, and each week's dependency refresh added
another 1.5 GB set that no tag ever released.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A hung test fails at two minutes and a hung fast lane at ten
All checks were successful
ci / Unit tests, type check, bundle (pull_request) Successful in 52s
ci / One container, e2e click-to-play inside the image (pull_request) Successful in 3m48s
a1075ec494
Run 285's pytest lane sat 25 minutes on idle xdist workers. make test
now passes --timeout=120 (pytest-timeout is already a test dependency),
so a hung test fails with its stack; the fast-lane job stops at ten
minutes instead of the runner's three-hour default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cmoriarty merged commit 932a36610e into develop 2026-10-29 21:38:49 -04:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/sports-breaker!51
No description provided.