Testing — what goes where

Loom has many test tiers because it generates code for eleven targets (five backends, six frontends) and the interesting failures live at different altitudes. This doc is the placement guide: given a change, which tier proves it, and where a new test belongs. For the exact run commands and env-var gates, CLAUDE.md → "Build & test commands" is the authoritative list; the CI column here maps each tier to its workflow.

The mental model

Two axes decide where a test lives:

• Structural vs behavioral. Structural tests assert the generator emits the right source — parse/validate the .ddd, lower to IR, and string-match or typecheck the output (expect(out).toContain(...), tsc --noEmit). Behavioral tests assert the generated code runs correctly — boot it and exercise real round-trips.
• Docker-free vs docker. Most tiers boot in-process (PGlite, app.fetch, headless Chromium). The cross-backend stack (.NET/Java/Phoenix/Python over real Postgres) needs docker and runs nightly.

Per-PR coverage is mostly structural + the docker-free behavioral tiers; the heavy cross-backend/runtime behavioral coverage is nightly.

The tiers

Per-PR column verified against the workflow on: blocks 2026-10-05, after wave C3 packet 3e moved the last never-gating runtime legs pre-merge (docs/ci-gating.md → "The never-gating legs, re-measured (2026-10-05)"); wave G1 of docs/new-plan/verification-waves-2026-09.md had promoted the obs, native-OIDC, SPA-smoke and VO legs before it. Nothing re-tests on push: main since 2026-10-20 (docs/ci-gating.md → "Post-merge net").

Choosing where a new test goes

Walk these in order; stop at the first match.

1. New/changed grammar, validator, macro, lowering, or IR shape → fast vitest suite (test/{language,macro,ir,...}). Add a parsing test

   • a negative validator test where relevant. After a grammar edit, re-run langium:generate and commit the output.

2. A backend/frontend now emits something new or different → fast vitest suite, one generator test per affected target (test/generator/<platform>/), string-matching the emitted source. This is the default home for generator changes — it's fast and exact. If the emitted shape is captured in a baseline fixture (test/fixtures/baseline-output/, regenerated via scripts/capture-baseline-fixture.mjs), regenerate it.

3. The emitted code might not compile (a new import, type, or dependency) → the matching per-backend / generated-frontend build gate. Run at least one LOOM_TS_BUILD=1 / LOOM_REACT_BUILD=1 pass locally.

4. The emitted code must behave — domain logic, an api round-trip, a page form → a behavioral tier:

   • api or pure-domain on the Hono backend → behavioral api/unit (it's already exercised if the example is in test/behavioral/corpus.json; add a test e2e … against <node> or aggregate test to the .ddd).
   • a React page/form round-trip → behavioral ui (add a test e2e … against <react>; the corpus case needs "ui": true).
   • cross-backend behaviour (.NET/Java/Phoenix/Python) → it rides the conformance-full docker leg; keep the assertion in the emitted test e2e so every backend runs it.

5. Cross-backend contract (OpenAPI / wire shape) → conformance parity (per-PR). It's mostly automatic from wireShape; a new contract dimension goes in the parity harness.

6. Something only observable in the real browser playground (worker IO, service-worker handoff, iframe synthesis, IDB, in-browser npm bundle) → playground e2e (web/e2e/). Remember it's post-merge, network-gated, and self-skips when the sandbox can't reach the registry — so it's a signal, not a per-PR gate.

Behavioral corpus constraint

The docker-free behavioral tiers (test/behavioral/) require each corpus system to have exactly one platform: node (Hono) deployable, so the host-agnostic, path-matched dispatch is unambiguous. Multi-backend systems (examples/showcase.ddd, examples/acme.ddd) stay in the docker conformance-full leg. See test/behavioral/README.md for the full runner mechanics (one-origin serving, async-spawn, the DoD rollup).

Why so many gates

Structural tests prove code is emitted; they can't catch a generated form that submits a malformed value, a backend that 500s on a real round-trip, or a duplicate-React bundle. Those only fail when the code runs — which is what the behavioral/runtime/conformance tiers exist for. Conversely, booting a docker stack to assert a string appears in a file would be absurdly slow. Put each assertion at the lowest altitude that can actually catch the failure.

In a fresh git worktree, run npm install first. test/platform/packaging-split-fs-discovery.test.ts and packaging-split-core-pkg.test.ts call discoverBackendsFs(repoRoot), which resolves the node_modules/@loom/backend-hono-v4|v5 workspace symlinks; a worktree with no node_modules fails both although the code is fine.

Proving a toolchain change neutral — scripts/compare-toolchains.mjs

A refactor that claims "no behaviour change" (a typing-pass cutover, an IR reshape, an emitter dedup) proves it against the models the repo already has:

node scripts/compare-toolchains.mjs                       # base = merge-base with origin/main
node scripts/compare-toolchains.mjs --match examples/     # a subset, for a quick look
node scripts/compare-toolchains.mjs --only diagnostics    # skip the generate leg

It builds the base in a detached worktree, then compares two legs over every tracked .ddd. The first is ddd parse --json diagnostics; the second is ddd generate system output, byte for byte, for each model the base validates. Each side reads its own copy of the sources, because generate system writes <dir>/.loom/<model>.migration-history.json next to the model and reads it back on the next run. Exit 0 means identical, 1 means a difference (listed in <out>/report.json). The full run takes about an hour on 4 cores. Put the result in the PR body; if the change is meant to move some output, list the intended differences there.

Running any CI gate locally — the reverse index

Every CI gate runs locally. Do not push a commit just to see whether a gate passes — that burns the shared ~20-slot runner pool and turns a 3-minute local check into an hour of queue. This table is the reverse index: given a workflow (check) name, the local command that runs the same thing. Prerequisites: docker = start the daemon first (dockerd >/tmp/dockerd.log 2>&1 &, then wait for docker info); pg = a Postgres on 127.0.0.1:5432 with user/password postgres (one docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=postgres postgres:18-alpine serves every pg row below, or the suite starts its own sidecar where noted); mirror = LOOM_HEX_MIRROR=1 when hex.pm is unreachable through the proxy (docs/tools.md → "Compiling generated backends in Docker" has the full per-toolchain write-ups).

Completeness is pinned by test/system/local-run-mapping.test.ts — every workflow file must have a row here, so a new gate without a local recipe fails the fast suite.