A node added to a built-in pipeline stalls runs already in flight #102
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Built-in pipelines are built from Braid's code, not read from the run's base commit. So after a deploy, a run admitted earlier loads the new graph, and its step rows were made from the old one.
If the new graph has a node the run has no row for (for example
openspec.repairfromvalidation-repair, orscript.plan-tick),_promotenever promotes anything that needs it (if any(need not in by_node for need in node.needs): continue). The run waits for ever with no reason shown.No run has hit this yet; it was noticed while shipping
validation-repair, and the deploys were timed so no run was mid-graph.Options:
osf.run.startedand loading that on recovery and redo.Option 1 keeps a run's process fixed for its life, which is what named-pipelines already promises for repository pipelines.
There's a second symptom of the same cause, seen on run #29 (git-flow on
cmoriarty/scratch). After the1d596b7(#107) and3346eae(#109) deploys, five of its finished steps were flagged "inputs changed", and the console offered "Re-run 5 stale steps":git.commit.proposalscript.budgetscript.gitignore.proposalopenspec.apply[1–3]script.review-outcomescript.depsagent.test.unitagent.reconcile-tasksscript.deps.testsrecompute_inputs_digestsrecomputes each succeeded step's inputs digest from the dependencies its node has in the graph loaded now. The run has no rows for the new nodes, so the digest comes out different, and the step is marked stale although nothing it read changed.agent.reconcile-tasks, whose needs didn't change, wasn't flagged.Re-running would have repeated about an hour of agent work for nothing. It also couldn't have unstuck the run:
git.commit.featurehad come to needagent.code-review(#108), for which the run had no row either, so it stayed pending until the run was abandoned.Option 1, pinning the graph a run was admitted with, fixes both symptoms.
Fixed in
dab6969with option 1, as the OpenSpec changepin-run-pipeline. It's archived in75bb426, updating the named-pipelines requirement "The run records its pipeline and recovery replays it".osf.run.started, projected onto the newrun.pipeline_yaml(migration 8). After a restart, and on redo, recovery and resume,Scheduler._pipelineloads that text first. A deploy that adds, removes or rewires built-in steps then changes new runs only. A run in flight keeps its graph, its finished steps keep their inputs, and nothing stalls.Verified:
git.commit.feature. It finished its feature commit on its recorded graph, with no step flagged stale, while a run started on the new build got the new step.dab6969: a run admitted after the deploy has its pipeline text recorded (12,552 characters). #34, admitted before it, has none, as expected.