Build steps write tests without spec tags, so round 1 of every fix loop only adds them #271

Closed
opened 2026-10-28 11:54:57 -04:00 by cmoriarty · 1 comment
Owner

What happens
In 6 of the 11 build runs (123–125, 127–129), the fix loop's first round failed on the test lint ("missing // spec: tag") before the judge ever looked. A fix round added the tags, and the tests ran again. The untagged tests came mostly from openspec.apply, which writes unit tests while it builds but never runs the lint.

Fix

  • The build step runs the test lint before it finishes, as it runs the tests.
  • Or the lint's message is part of the build's own check, so the step fixes it in place.

Done when
Round 1 of a fix loop reaches the judge in a run whose build wrote new tests.

From the #260 re-run's transcripts.

**What happens** In 6 of the 11 build runs (123–125, 127–129), the fix loop's first round failed on the test lint ("missing `// spec:` tag") before the judge ever looked. A fix round added the tags, and the tests ran again. The untagged tests came mostly from `openspec.apply`, which writes unit tests while it builds but never runs the lint. **Fix** - The build step runs the test lint before it finishes, as it runs the tests. - Or the lint's message is part of the build's own check, so the step fixes it in place. **Done when** Round 1 of a fix loop reaches the judge in a run whose build wrote new tests. _From the #260 re-run's transcripts._
Author
Owner

Fixed in #275 (PR #277, live on production). The apply brief asks for a test tagged with each scenario and for ban-check on each new test file, and an apply step doesn't finish while its own tests have a blocking lint finding (B1, B2, B6).

Run 133: apply ran ban-check itself and tagged every scenario (6 of 6), so agent.write-spec-tests was skipped and round 1's lanes passed with no lint finding. The judge didn't run, but only because no UI file changed, not because of the lint. One side effect is filed as #278: rule B7 judges a test by its file name, and the agent renamed a test twice to please it.

Fixed in #275 (PR #277, live on production). The apply brief asks for a test tagged with each scenario and for `ban-check` on each new test file, and an apply step doesn't finish while its own tests have a blocking lint finding (B1, B2, B6). Run 133: apply ran `ban-check` itself and tagged every scenario (6 of 6), so `agent.write-spec-tests` was skipped and round 1's lanes passed with no lint finding. The judge didn't run, but only because no UI file changed, not because of the lint. One side effect is filed as #278: rule B7 judges a test by its file name, and the agent renamed a test twice to please it.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#271
No description provided.