A failed CI job's log reaches the fix round as the job's tail, mostly post-step cleanup: the error itself is cut to one line #180

Closed
opened 2026-10-16 18:35:53 -04:00 by cmoriarty · 1 comment
Owner

Seen on run 67 (production 74e261c), 2026-10-16. The CI lane (#163) gives a failed job's log to the fix round as the last 3,000 characters of the job's log. That tail is almost all the job's post steps: "⭐ Run Post Check out the repository", the git config cleanup, "✅ Success - Post Check out the repository".

The failure itself survives only as a cut line at the very top:

  • Round 1: ts,line=227,column=3::Error: Test timed out in 5000ms. … ❯ test/server.test.ts:227:3.
  • Round 2: …: undefined, … file: 'pg_type.c', routine: 'TypeCreate' }. That's a Postgres error whose message and test name were cut off; most likely concurrent CREATE TABLE in CI's shared database.

So the fix round works half blind on exactly the failures it can't see locally, which makes more rounds, and more cap failures (#178), likely.

Wanted

  • The red lane's tail is the failing step's own output: the step the lane already names ("failed at step Unit tests"), up to its end. Post steps and the job's set-up are left out.
  • Where the runner marks errors (::error, the test runner's summary), keep those whole even when the tail is cut.
  • A test: a job log with a long post-step section still yields the failing step's error lines in the lane.
Seen on run 67 (production 74e261c), 2026-10-16. The CI lane (#163) gives a failed job's log to the fix round as the last 3,000 characters of the **job's** log. That tail is almost all the job's post steps: "⭐ Run Post Check out the repository", the git config cleanup, "✅ Success - Post Check out the repository". The failure itself survives only as a cut line at the very top: - **Round 1:** `ts,line=227,column=3::Error: Test timed out in 5000ms. … ❯ test/server.test.ts:227:3`. - **Round 2:** `…: undefined, … file: 'pg_type.c', routine: 'TypeCreate' }`. That's a Postgres error whose message and test name were cut off; most likely concurrent `CREATE TABLE` in CI's shared database. So the fix round works half blind on exactly the failures it can't see locally, which makes more rounds, and more cap failures (#178), likely. **Wanted** - The red lane's tail is the failing **step's** own output: the step the lane already names ("failed at step `Unit tests`"), up to its end. Post steps and the job's set-up are left out. - Where the runner marks errors (`::error`, the test runner's summary), keep those whole even when the tail is cut. - A test: a job log with a long post-step section still yields the failing step's error lines in the lane.
Author
Owner

Shipped in bb5d87a (change ci-log-is-the-failing-step, commit 609debb), live since 00:07Z on 2026-10-17, and archived after the verification run (run_01M53R7621CJW6PCZG7S6R94HS, thorough on cmoriarty/braid-combined-test, vLLM, succeeded and merged at 03:49Z).

  • The excerpt is cut at Forgejo runner's bare exitcode '1': failure line, so it ends where the failing step ended, not after the post steps.
  • Colour codes are stripped before the end of the output is taken.
  • Each ::error annotation, which names the failing test, its file and line, and its message, is kept whole and decoded at the top.
  • Verified on run 67's two real failed job logs (braid-combined-test Actions runs 10 and 11), kept as fixtures. Both excerpts start with the annotation ("type "links" already exists", "Test timed out in 5000ms") and end with the test summary, with no post-step line and no escape code.
  • Not yet seen on production: in the verification run, CI wasn't asked in round 1 (a local lane was red) and passed all 6 jobs in round 2.

Closing; the next run whose CI fails will show it.

Shipped in bb5d87a (change `ci-log-is-the-failing-step`, commit 609debb), live since 00:07Z on 2026-10-17, and archived after the verification run (`run_01M53R7621CJW6PCZG7S6R94HS`, thorough on cmoriarty/braid-combined-test, vLLM, succeeded and merged at 03:49Z). - The excerpt is cut at Forgejo runner's bare `exitcode '1': failure` line, so it ends where the failing step ended, not after the post steps. - Colour codes are stripped before the end of the output is taken. - Each `::error` annotation, which names the failing test, its file and line, and its message, is kept whole and decoded at the top. - Verified on run 67's two real failed job logs (braid-combined-test Actions runs 10 and 11), kept as fixtures. Both excerpts start with the annotation ("type \"links\" already exists", "Test timed out in 5000ms") and end with the test summary, with no post-step line and no escape code. - Not yet seen on production: in the verification run, CI wasn't asked in round 1 (a local lane was red) and passed all 6 jobs in round 2. Closing; the next run whose CI fails will show it.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#180
No description provided.