The test lint and scenario coverage decide nothing, so a run can merge flagged tests and untested scenarios #138
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Since #137,
agent.test.unitandagent.test.e2ecomplete onlane-passed, which is the suite's exit code on the current code. The lane report still computes the test lint (B1–B7), scenario coverage and mutation, and the step's summary shows them, but none of them decides anything. #137 left that to a follow-up, to be judged by the measurement it took.What the measurement says
These are runs 22 to 41 on production, recorded in
openspec/changes/archive/2026-10-03-test-lanes-hold-on-the-verdict/design.md.expect(parseState(…)).not.toBeNull()as the only check of "accepts an empty seat"To decide first: does a test a run only edited owe a tag?
B6 judges the tests a run added or changed. In runs 36 to 39, 27 of its 62 findings were older tests that the run only edited, for instance by adding a prop to every
render. There are two options:Proposal
ban. It should name each finding with its file, line and rule, as it already does for a malformed report's schema errors.agent.test.unitandagent.test.e2etosuites-green:<lane>. That check already re-derives the lint over the run's own lines (lint_base), and checks coverage and mutation. A failing verdict is tried again twice, briefed with every reason, then goes to a person.Related: #137, and #125 (the verification steps).
Thinking on the open question (should a test a run only edited still need a tag?), from a review of the hello world baseline in #139.
What a tag is for. A
// spec:tag gives traceability: which scenarios have no test, and which tests belong to no scenario. It does not show that a test is meaningful. A test that asserts nothing useful carries a tag as easily as a good one, and a rule that checks for a tag rewards adding one. That is the same proxy trap #139 found with the "107#on screen" oracle, which passed seven times without checking anything real.So:
One-time audit of the older untagged tests, separate from the rule. Run mutation on them first, so "meaningful" is measured, then review the survivors and decide each one:
The audit needs to say that deleting and "keep but flag" are acceptable outcomes. Otherwise the easy way to quiet the lint is to tag everything.
Not yet checked: how many of the older untagged tests fall into each group, and whether this repository's own tests have the same gap as the scratch app's.
Shipped as two changes, re-scoped because #139 removed
agent.test.unitandagent.test.e2efrom the built-in pipelines. Tag matching belongs to #170'sscenario-tags-match.weak-tests-fail-the-check: the lint decides, in the fix loop.def test_to the end of the file.lint.blockingandlint.recordedinverify.json. A B1 or B2 finding fails the round as a red lane does, and so does B6 in a change. B3–B5 and B7 are recorded only: they had no findings to measure, and B5 and B7 have false-positive shapes in the code.pull-request-says-what-tests-cover: tags are a report. The pull request gets a "Tests" section: how many of the change's scenarios have a tagged test, which don't, and what the lint recorded without failing.Verified. The fast and full lanes passed on the integrated batch (3,388 backend tests, 681 vitest, 249 browser specs with the slow and live ones, 29 of 29 simulated runs), and it is live on production as
57971c2(deploy run 138, self-checkbraid-selfcheck: ok). Archived ina6ae0dc. The new simulated run islint-weak-test. On production, in a throwaway repository, an added test asserting onlytoBeTruthy()failed round 1 on B2, and passed once it asserted the value. The pull request body read "1 of 2 scenarios ofadd-lobbyhas a tagged test" and named the missing one.Deferred: