Shared caches with two runs at once, the store reused after a deploy, and slimming a finished run's environments #192

Closed
opened 2026-10-18 01:34:59 -04:00 by cmoriarty · 3 comments
Owner

Follow-up to #67. Lean run folders have been live since 4309dfe (2026-10-17), and were archived in OpenSpec as lean-run-folders in 874eb27.

Where it stands

  • Run 74's folder is 139 MB: work 135 MB, of which node_modules is 132 MB. home/.npm, cache/ and tmp/ are at 0, and there is no opencode plugin folder.
  • The shared cache is 840 MB.
  • osfd logs "runs get UV_LINK_MODE=clone".

A finished run is now almost entirely its dependency environments.

1. Two runs installing at once (#67's task 9.6)

With the operator's go, start two quick-fix runs on one throwaway repository at once. Check that:

  • both install and test green;
  • each run's home/.npm/_cacache is a link to the store, and each _npx is its own;
  • /home/braid gains no .npm and no .cache/uv (name here anything else that appears there);
  • once both have succeeded, each tmp/ holds nothing but handoff/;
  • each footprint and the archive dialog give the breakdown.

2. The store reused after a deploy (task 9.7)

After the next deploy that follows 1: a run on the same repository installs its packages from the store. The store does not grow, and its script.dependencies takes less time than the first install in 1.

3. Production checks after the deploy, not yet recorded (task 9.5)

  • The self-check's braid-check-shared-caches lines in the deploy log.
  • osfd doctor on production.
  • Whether run 73's tmp/ holds nothing but handoff/. The rest of the runs that finished before the deploy was read at archive with GET /api/runs/{id}/footprint?measure=1: runs 55 to 73 hold no home/.npm or cache/ contents (caches 0), and the succeeded ones' tmp/ is empty, except run 73's 12,750 bytes, which the breakdown does not split.

4. Slimming a finished run

#67's acceptance, carried here:

  • A finished run on the ledger is slimmed.
  • A resumed slimmed run re-installs and continues.

Why it was left (lean-run-folders design D9). A finished run's node_modules and .venv are now the largest part of it: 137 of 169 MB in run 67, 264 of 468 MB in the flipart runs, and 132 of 139 MB in run 74. But any finished run can be re-opened, by Resume, a redo or an accept, and a re-opened step expects the environments script.dependencies made:

  • script.dependencies.tests reinstalls only the directories whose manifests changed;
  • a resumed step may be any step, scripted ones first.

So a safe slim needs a re-install, keyed on .osf/deps.json, that holds a re-opened run's steps until it finishes. Until then, archiving with "delete its files" reclaims a run whole.

The breakdown to measure it by (design D8) is in place: GET /api/runs/{id}/footprint?measure=1 and the archive dialog split a run into dependencies, caches, temp, sessions and work.

Done when

  • 1 and 2 are recorded here with their numbers, and 3 has been read and recorded.
  • A finished run's environments are removed, and a re-opened run reinstalls them before any of its steps runs.
  • Both behaviours have unit tests and a simulated run, and both are seen on production.
Follow-up to #67. Lean run folders have been live since 4309dfe (2026-10-17), and were archived in OpenSpec as `lean-run-folders` in 874eb27. ## Where it stands - Run 74's folder is 139 MB: work 135 MB, of which `node_modules` is 132 MB. `home/.npm`, `cache/` and `tmp/` are at 0, and there is no opencode plugin folder. - The shared cache is 840 MB. - osfd logs "runs get UV_LINK_MODE=clone". A finished run is now almost entirely its dependency environments. ## 1. Two runs installing at once (#67's task 9.6) With the operator's go, start two quick-fix runs on one throwaway repository at once. Check that: - both install and test green; - each run's `home/.npm/_cacache` is a link to the store, and each `_npx` is its own; - `/home/braid` gains no `.npm` and no `.cache/uv` (name here anything else that appears there); - once both have succeeded, each `tmp/` holds nothing but `handoff/`; - each footprint and the archive dialog give the breakdown. ## 2. The store reused after a deploy (task 9.7) After the next deploy that follows 1: a run on the same repository installs its packages from the store. The store does not grow, and its `script.dependencies` takes less time than the first install in 1. ## 3. Production checks after the deploy, not yet recorded (task 9.5) - The self-check's `braid-check-shared-caches` lines in the deploy log. - `osfd doctor` on production. - Whether run 73's `tmp/` holds nothing but `handoff/`. The rest of the runs that finished before the deploy was read at archive with `GET /api/runs/{id}/footprint?measure=1`: runs 55 to 73 hold no `home/.npm` or `cache/` contents (caches 0), and the succeeded ones' `tmp/` is empty, except run 73's 12,750 bytes, which the breakdown does not split. ## 4. Slimming a finished run #67's acceptance, carried here: - A finished run on the ledger is slimmed. - A resumed slimmed run re-installs and continues. **Why it was left** (lean-run-folders design D9). A finished run's `node_modules` and `.venv` are now the largest part of it: 137 of 169 MB in run 67, 264 of 468 MB in the flipart runs, and 132 of 139 MB in run 74. But any finished run can be re-opened, by Resume, a redo or an accept, and a re-opened step expects the environments `script.dependencies` made: - `script.dependencies.tests` reinstalls only the directories whose manifests changed; - a resumed step may be any step, scripted ones first. So a safe slim needs a re-install, keyed on `.osf/deps.json`, that holds a re-opened run's steps until it finishes. Until then, archiving with "delete its files" reclaims a run whole. The breakdown to measure it by (design D8) is in place: `GET /api/runs/{id}/footprint?measure=1` and the archive dialog split a run into dependencies, caches, temp, sessions and work. ## Done when - 1 and 2 are recorded here with their numbers, and 3 has been read and recorded. - A finished run's environments are removed, and a re-opened run reinstalls them before any of its steps runs. - Both behaviours have unit tests and a simulated run, and both are seen on production.
Author
Owner

Part 4 shipped in PR 468 (main ddfd8a93), archived as batch-2026-11-09b (PR 469). Parts 1 to 3 are production recordings and stay open here.

What changed.

  • The finished-run tidy also deletes the environments the working copy's .osf/deps.json records as installed (node_modules, .venv), only when the working copy is inside the run's folder, each environment's parent inside the working copy, and past unsafe_to_delete; a link goes as a link. Nothing else of the working copy is touched. The footprint's dependencies part shows what is left.
  • A re-opened run (resume, redo, accept) whose recorded environments are missing is held by the scheduler while $OSF_PYTHON -m osf.pipeline.deps install --root . --changed runs in the working copy with the dependencies step's own environment. --changed now counts a recorded environment the disk lacks as changed, so an unchanged manifest is installed again. A failed install is logged and the steps run anyway. The hold shows only in the log (installing backend/.venv again before its steps run, then environments installed again in N s).

Evidence on the laptop.

  • Unit tests: tests/engine/test_slim_finished_run.py (what the tidy lists, and what it refuses), tests/pipeline/test_deps_missing_environments.py (the record read; the --changed pass reinstalls a missing environment and runs nothing when it is there), tests/engine/test_restore_environments.py (the hold, the release, the command and its environment).
  • Sim slimmed-run-resumes: a quick-fix run on hello-venv is stopped after its first turn, resumed with after: slimmed (a new operator option that waits until the footprint measures no dependencies), and merges with backend/.venv/bin/python back and script.dependencies run once. Its osfd log: tidied finished run …: 126635 bytes (dependencies 32955, caches 93680), installing backend/.venv again before its steps run, environments installed again in 2 s, and at the end tidied finished run …: (dependencies 11050892, caches 93680, temp 16974364). ./tools/sim.sh --smoke: 7 of 7.

Left for production (parts 1 to 3): two quick-fix runs at once on a throwaway repository under braid-testing/, with the operator's go; the store reused after the next deploy; the braid-check-shared-caches lines, osfd doctor, and the run folders' tmp/. Plus the first real slim and re-install: a finished run's footprint reading dependencies 0, and the two log lines above on a resume.

Part 4 shipped in PR 468 (main `ddfd8a93`), archived as `batch-2026-11-09b` (PR 469). Parts 1 to 3 are production recordings and stay open here. **What changed.** - The finished-run tidy also deletes the environments the working copy's `.osf/deps.json` records as installed (`node_modules`, `.venv`), only when the working copy is inside the run's folder, each environment's parent inside the working copy, and past `unsafe_to_delete`; a link goes as a link. Nothing else of the working copy is touched. The footprint's `dependencies` part shows what is left. - A re-opened run (resume, redo, accept) whose recorded environments are missing is held by the scheduler while `$OSF_PYTHON -m osf.pipeline.deps install --root . --changed` runs in the working copy with the dependencies step's own environment. `--changed` now counts a recorded environment the disk lacks as changed, so an unchanged manifest is installed again. A failed install is logged and the steps run anyway. The hold shows only in the log (`installing backend/.venv again before its steps run`, then `environments installed again in N s`). **Evidence on the laptop.** - Unit tests: `tests/engine/test_slim_finished_run.py` (what the tidy lists, and what it refuses), `tests/pipeline/test_deps_missing_environments.py` (the record read; the `--changed` pass reinstalls a missing environment and runs nothing when it is there), `tests/engine/test_restore_environments.py` (the hold, the release, the command and its environment). - Sim `slimmed-run-resumes`: a quick-fix run on `hello-venv` is stopped after its first turn, resumed with `after: slimmed` (a new operator option that waits until the footprint measures no dependencies), and merges with `backend/.venv/bin/python` back and `script.dependencies` run once. Its osfd log: `tidied finished run …: 126635 bytes (dependencies 32955, caches 93680)`, `installing backend/.venv again before its steps run`, `environments installed again in 2 s`, and at the end `tidied finished run …: (dependencies 11050892, caches 93680, temp 16974364)`. `./tools/sim.sh --smoke`: 7 of 7. **Left for production (parts 1 to 3):** two quick-fix runs at once on a throwaway repository under `braid-testing/`, with the operator's go; the store reused after the next deploy; the `braid-check-shared-caches` lines, `osfd doctor`, and the run folders' `tmp/`. Plus the first real slim and re-install: a finished run's footprint reading `dependencies 0`, and the two log lines above on a resume.
Author
Owner

The first slim on production, at the boot of ddfd8a9 (2026-11-09, 20:41 UTC). boot: tidying 14 finished run(s)' caches and temporary files, then one line per run; 13 had environments, run 187 (stopped during propose, before its install) had nothing to delete:

Run Freed (dependencies)
run_01M6K8CA… 86.5 MB
run_01M6Q9G6… 96.5 MB
run_01M6QT8M…, run_01M6S9C8…, run_01M6SF1D…, run_01M6SN4Y… 105.0 MB each
run_01M6SNC6…, run_01M6SY0G… 129.6 MB, 129.9 MB
run_01M6W1NT… 261.7 MB
run_01M6WZ8V…, run_01M6Y92E… 87.7 MB, 88.1 MB
run_01M70CPH… (Greeter 185) 18.7 MB
run_01M70M69… (Greeter 186) 82.2 MB

About 1.40 GB in all, the whole of the dependencies part of every finished run on the ledger; their footprints now read dependencies 0. Nothing else was touched (each line's bytes are all dependencies). The re-install on a re-open has not yet happened on production: the next resume or redo of one of these runs will show installing node_modules again before its steps run in osfd's log.

**The first slim on production**, at the boot of `ddfd8a9` (2026-11-09, 20:41 UTC). `boot: tidying 14 finished run(s)' caches and temporary files`, then one line per run; 13 had environments, run 187 (stopped during propose, before its install) had nothing to delete: | Run | Freed (dependencies) | |---|---| | run_01M6K8CA… | 86.5 MB | | run_01M6Q9G6… | 96.5 MB | | run_01M6QT8M…, run_01M6S9C8…, run_01M6SF1D…, run_01M6SN4Y… | 105.0 MB each | | run_01M6SNC6…, run_01M6SY0G… | 129.6 MB, 129.9 MB | | run_01M6W1NT… | 261.7 MB | | run_01M6WZ8V…, run_01M6Y92E… | 87.7 MB, 88.1 MB | | run_01M70CPH… (Greeter 185) | 18.7 MB | | run_01M70M69… (Greeter 186) | 82.2 MB | About 1.40 GB in all, the whole of the `dependencies` part of every finished run on the ledger; their footprints now read `dependencies 0`. Nothing else was touched (each line's bytes are all `dependencies`). The re-install on a re-open has not yet happened on production: the next resume or redo of one of these runs will show `installing node_modules again before its steps run` in osfd's log.
Author
Owner

Parts 1 to 3, read from production on 2026-11-11 (f27905f, idle). Closing: part 4 shipped and slimmed production on 11-09 (above), and these are the recordings.

3. Production checks

The deploy log of f27905f has all seven braid-check-shared-caches lines ok:

  • npm: two runs ran npx of one package at once, each into its own _npx
  • npm: two runs installed one package at once
  • npm: the shared store holds the package once, and no run a copy of its own
  • npm: a run's npm cache clean --force left the shared store's 5 files
  • uv: two runs installed one wheel at once from one cache, every file their own (UV_LINK_MODE=clone)
  • uv: an edit in one run's .venv reached neither the other run nor the cache
  • uv: two runs ran uv run --with of one package at once

osfd doctor on production: 0 failed, 1 warned (the sandbox, braid#203). "the shared package caches are writable" and "runs get UV_LINK_MODE=clone" are both ok.

Run folders, all 19 on disk:

  • every home/.npm/_cacache is a link to the shared store, and no run has a file in cache/;
  • the 15 succeeded runs' tmp/ is empty; the 4 stopped runs keep theirs, as designed;
  • /home/braid holds its three shell dotfiles and nothing else: no .npm, no .cache.

2. The store reused after a deploy

Seven installs of one project (scratch-vllm-fp8, and run 192's copy of it), 11-06 to 11-11. The last four each ran after a new deploy.

Runs npm ci in script.dependencies
175, 176, 177 added 125 packages in 2.3, 2.2 and 2.5 s
187, the first with 130 packages 4.1 s
188, 189, 192 2.6, 2.5 and 2.4 s

Runs 178 and 179 on brooke: 100 packages in 3.1 and 2.7 s. Installs that take seconds come from the store.

Not measured: the store's size before and after one such install. The shared cache is about 2.8 GB today (npm 2.2 GB, uv 0.6 GB), against 840 MB on 10-18. Nothing prunes it.

1. Two runs installing at once

Not run as written (two quick-fix runs started together on one repository). What there is instead:

  • the two "at once" lines above, checked on production's own image at every deploy;
  • runs 177 and 178 alive together on 11-07, their script.dependencies a minute apart (3.2 s and 3.7 s), both succeeded, and their folders read as in 3.

No two script.dependencies steps have overlapped to the second in the 23 runs on the ledger.

Part 4, one thing still not seen on production: a re-opened slimmed run installing again. No slimmed run has been resumed since 11-09, so that rests on the sim slimmed-run-resumes. Today's boot tidied 19 finished runs and found nothing left to delete.

An install that breaks with two runs at once, or a resume that fails to reinstall, would be a new ticket.

**Parts 1 to 3, read from production on 2026-11-11** (`f27905f`, idle). Closing: part 4 shipped and slimmed production on 11-09 (above), and these are the recordings. **3. Production checks** The deploy log of `f27905f` has all seven `braid-check-shared-caches` lines `ok`: - npm: two runs ran npx of one package at once, each into its own `_npx` - npm: two runs installed one package at once - npm: the shared store holds the package once, and no run a copy of its own - npm: a run's `npm cache clean --force` left the shared store's 5 files - uv: two runs installed one wheel at once from one cache, every file their own (`UV_LINK_MODE=clone`) - uv: an edit in one run's `.venv` reached neither the other run nor the cache - uv: two runs ran `uv run --with` of one package at once `osfd doctor` on production: 0 failed, 1 warned (the sandbox, braid#203). "the shared package caches are writable" and "runs get UV_LINK_MODE=clone" are both ok. Run folders, all 19 on disk: - every `home/.npm/_cacache` is a link to the shared store, and no run has a file in `cache/`; - the 15 succeeded runs' `tmp/` is empty; the 4 stopped runs keep theirs, as designed; - `/home/braid` holds its three shell dotfiles and nothing else: no `.npm`, no `.cache`. **2. The store reused after a deploy** Seven installs of one project (`scratch-vllm-fp8`, and run 192's copy of it), 11-06 to 11-11. The last four each ran after a new deploy. | Runs | `npm ci` in `script.dependencies` | |---|---| | 175, 176, 177 | added 125 packages in 2.3, 2.2 and 2.5 s | | 187, the first with 130 packages | 4.1 s | | 188, 189, 192 | 2.6, 2.5 and 2.4 s | Runs 178 and 179 on `brooke`: 100 packages in 3.1 and 2.7 s. Installs that take seconds come from the store. Not measured: the store's size before and after one such install. The shared cache is about 2.8 GB today (npm 2.2 GB, uv 0.6 GB), against 840 MB on 10-18. Nothing prunes it. **1. Two runs installing at once** Not run as written (two quick-fix runs started together on one repository). What there is instead: - the two "at once" lines above, checked on production's own image at every deploy; - runs 177 and 178 alive together on 11-07, their `script.dependencies` a minute apart (3.2 s and 3.7 s), both succeeded, and their folders read as in 3. No two `script.dependencies` steps have overlapped to the second in the 23 runs on the ledger. **Part 4, one thing still not seen on production:** a re-opened slimmed run installing again. No slimmed run has been resumed since 11-09, so that rests on the sim `slimmed-run-resumes`. Today's boot tidied 19 finished runs and found nothing left to delete. An install that breaks with two runs at once, or a resume that fails to reinstall, would be a new ticket.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#192
No description provided.