We need to design a baseline test that we can reuse perpetually #489

Closed
opened 2026-11-11 15:25:24 -05:00 by cmoriarty · 1 comment
Owner

Currently, I come up with ideas for throwaway games and apps when I need to test braid end to end. This works well enough, but I'd like to put more thought into it. Mainly, we should devise a test that creates basic implementations of an array of different types of applications.

Some ideas:

  • Games [2d, 3d, pixelated, web, mobile-friendly, games like Freedoku and Scratch]
  • Applications [database application, static web applications, data driven visualizations, apps like Soundcheck]
  • Services [services such as Sports Breaker]

This requires some research and design work to round out the types of things we want Braid to be good at. The end result should be a single brief that builds one project and tests all of these types of projects we want to judge Braid on, as well as testing as many of Braids features as possible.

Currently, I come up with ideas for throwaway games and apps when I need to test braid end to end. This works well enough, but I'd like to put more thought into it. Mainly, we should devise a test that creates basic implementations of an array of different types of applications. Some ideas: - Games [2d, 3d, pixelated, web, mobile-friendly, games like Freedoku and Scratch] - Applications [database application, static web applications, data driven visualizations, apps like Soundcheck] - Services [services such as Sports Breaker] This requires some research and design work to round out the types of things we want Braid to be good at. The end result should be a single brief that builds one project and tests all of these types of projects we want to judge Braid on, as well as testing as many of Braids features as possible.
Author
Owner

Shipped in cmoriarty/braid#490 (main 799b2f05), archived in cmoriarty/braid#491 as standard-test-run. The capability is standard-run.

  • The brief: docs/test-briefs/midway.md. Midway is a small arcade with a scoreboard, in eight parts that do not depend on each other: a static lobby, a 2D pixel-art game, a number puzzle, a 3D game, a game for two people, a database application with an API, a page of charts, and a background worker. Each asks for a phone-width screen. Nothing calls an outside service, and everything random or timed takes a seed or a clock.
  • Starting it: ./tools/standard-run.sh <server> <n> starts a New Project run in braid-testing/midway-<n>: the thorough pipeline in one run, groups side by side, auto-approve on, Use CI off, the brief's back end as the run's services.
  • The record: docs/testing.md lists the rows written down for every standard run, and says when to use Midway, Greeter or Scratch.
  • Test: the brief keeps a part for each kind, and the script's request is one the API accepts.

Design choices are in the archived change's design.md, among them why one arcade and not eight programs, and why the parts are independent (side-by-side groups need work that does not chain). Not covered yet: a planned, several-run variant; a run with CI on; a Free run. Each is this brief with one setting changed.

The first run is next; its record and findings go on its own issue.

**Shipped in cmoriarty/braid#490** (main `799b2f05`), archived in cmoriarty/braid#491 as `standard-test-run`. The capability is `standard-run`. - **The brief:** `docs/test-briefs/midway.md`. Midway is a small arcade with a scoreboard, in eight parts that do not depend on each other: a static lobby, a 2D pixel-art game, a number puzzle, a 3D game, a game for two people, a database application with an API, a page of charts, and a background worker. Each asks for a phone-width screen. Nothing calls an outside service, and everything random or timed takes a seed or a clock. - **Starting it:** `./tools/standard-run.sh <server> <n>` starts a New Project run in `braid-testing/midway-<n>`: the thorough pipeline in one run, groups side by side, auto-approve on, Use CI off, the brief's back end as the run's services. - **The record:** `docs/testing.md` lists the rows written down for every standard run, and says when to use Midway, Greeter or Scratch. - **Test:** the brief keeps a part for each kind, and the script's request is one the API accepts. Design choices are in the archived change's `design.md`, among them why one arcade and not eight programs, and why the parts are independent (side-by-side groups need work that does not chain). Not covered yet: a planned, several-run variant; a run with CI on; a Free run. Each is this brief with one setting changed. The first run is next; its record and findings go on its own issue.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
cmoriarty/braid#489
No description provided.