Loom
← Back to home

Migrations

MigrationsIR is the schema delta a generated stack applies to bring its Postgres database in line with the .ddd source. It is the only secondary IR in the compiler — every backend consumes LoomModel directly for everything else, but schema migration is the one place where (a) the table layout is a pure function of the enriched IR that every DB backend needs identically, and (b) the output is stateful (it diffs against the schema the committed migration chain already builds). Deriving it once in phase ⑨ and handing the same MigrationsIR[] to Drizzle, EF Core, Ecto, Flyway, and the Python runner means a schema change is bit-for-bit equivalent across all five backends — cross-backend consistency falls out of having one canonical source rather than five hand-kept emitters.

Builder: src/system/migrations-builder.ts. Type: src/ir/types/migrations-ir.ts. SQL renderer: src/generator/sql-pg.ts.

The shape

One MigrationsIR is produced per generation of an owning module — one per entry of the migration chain:

interface MigrationsIR {
  module: string;        // owning module
  storageName: string;   // physical storage binding (pg, analyticsDb), or ""
  baseline: SchemaSnapshot | null;   // the module's schema before; null for its Initial
  next: SchemaSnapshot;              // the module's schema after
  steps: MigrationStep[];            // ordered ops baseline → next
  version: string;       // chain version — unique across modules, strictly increasing
  name: string;          // "Initial", or the `migrate add` name ("AddOrderStatus")
}

A SchemaSnapshot is an alphabetically-sorted list of TableShapes ({ name, schema?, columns, primaryKey, foreignKeys, indexes, checks? }). MigrationStep is a closed union of createTable / dropTable / renameTable / addColumn / dropColumn / renameColumn / alterColumnNullable / alterColumnType / alterColumnDefault / addIndex / dropIndex / renameIndex / addCheck / dropCheck / sqlComment / backfillColumn / sqlExec. Backends only translate steps to native syntax — they never re-derive the schema from the IR.

Every delta step carries a schema? — the owning bounded context's Postgres schema, exactly as createTable carries it on the nested TableShape. Without it an ALTER/DROP/CREATE INDEX on a schema-qualified system targets the wrong relation or fails (ALTER TABLE "sales"."orders" …, not a bare orders). The SQL renderer and the Ecto emitter both qualify the relation from the step's schema.

Phase ⑨ derivation

buildMigrations(sys, snapshots) (called by ddd migrate add and by the system orchestrator, not by ir/) walks every module:

  1. Skip modules with no migrationsOwner (frontend-only subdomains emit nothing).
  2. Build next = schemaFromModule(module, shapeOf, schemaOf) — the table list the current source describes.
  3. Read baseline from the SnapshotStore — the module's schema at the head of the migration chain (headSnapshots), or nothing when the module has no entry.
  4. steps = diffSchema(baseline, next) — a pure diff. Same snapshot in ⇒ empty steps.

The result is a MigrationDelta — a MigrationsIR without a version. Only the chain numbers a generation, when ddd migrate add records it (see The migration chain).

diffSchema emits an FK-safe global order: rename tables first (the diff pairs a renamed table with its new self, so every later step — a dropped CHECK, column or index included — names it by its NEW name) → drop indexes/columns (which unblocks table drops) → drop tables in reverse-topological (child-first) order so a parent is never dropped while a child still references it → create tables in topological (parent-first) order (Kahn's sort with an alphabetical tiebreak) so an inline REFERENCES never points at a not-yet-created table → add columns (their FK targets now exist) → alter columns → add indexes. CHECK constraints bracket the column work — dropped first (a Postgres constraint keeps its columns alive, so it must go before they can be dropped or renamed) and added last (every column it names then exists under its final name). Tables are matched by schema-qualified name (sales.orders ≠ billing.orders); an old snapshot whose tables predate schema-qualification (no schema field) is reconciled by bare name against a single same-named next table, so a format bump reads as "same table, now qualified" rather than drop+recreate.

Between the diff and the emitted steps, applyDestructivePolicy classifies the delta and enforces the destructive-change gate (see below).

Destructive changes — the --allow-destructive gate

A delta can silently destroy data. applyDestructivePolicy classifies each step and, unless ddd migrate add is run with --allow-destructive, aborts with a loom.migration-destructive error naming the offending steps. A module's Initial is always exempt — nothing pre-exists to destroy. The decision is made once, when the generation is recorded, and the chain entry records it ("allowDestructive": true); generate system replays the chain and takes no such flag.

ddd migrate add drop-legacy app.ddd                       # aborts on a destructive delta
ddd migrate add drop-legacy app.ddd --allow-destructive   # records it (drops; NOT-NULL → 3-step)

The inferred rename announces itself — loom.migration-rename-inferred

The three contrary signals above narrow the guess; they cannot eliminate it, because nothing can. A rename and an unrelated drop+add of the same shape reach applyDestructivePolicy as a byte-identical diff, so whichever way the heuristic jumps, one of the two authors is not being served. Collapsing is the right default — the author who did rename would otherwise lose the column — but it leaves a residual: the author who did not rename gets the old column's rows under the new column's name.

What made that residual dangerous was not the guess, it was the silence. The collapse used to fire and the run to report 0 error(s), 0 warning(s), so nothing ever told that author to look. It now emits loom.migration-rename-inferred every time it fires, naming the table, both column names, and the way to declare the intent either way:

loom.migration-rename-inferred warning: migration for module "Cat": inferred a RENAME of
  shop.products.title -> description (one dropped column, one added column, same type and
  nullability, no backfill and no default). If these are unrelated columns, the old column's
  data will land under the new name. Declare the intent either way:
    migration "rename-…" { <Aggregate>.<oldField> -> <newField> }   // yes, a rename
    migration "backfill-…" { <Aggregate>.<newField> = <value> }     // no — a new column, and this is what its existing rows get

It is a warning, deliberately, and it does not gate the exit code: the inference is load-bearing, so an error would break every model that relies on it, and the migration is written. Declaring the intent — either way — silences it, because both declarations are contrary signals the collapse already yields to:

A scalar-literal field default on the added column (contrary signal 3) silences it for the same reason as the backfill. A nullability difference (signal 1) never reaches the warning either — it raises loom.migration-ambiguous-rename instead, which is an error.

Table steps — renames and cross-context moves

One step shape keeps a table's rows when the declaration that owns it is renamed, or moved to another bounded context. The left side names the declaration as it was, the right side as it is now:

migration "evolve" {
  Order -> PurchaseOrder             // rename in place (an aggregate)
  Tracker -> Watcher                 // a workflow's correlation-state table
  OrderTotal -> OrderSum             // a materialized projection's read model
  Leases.LeaseLedger -> Ledgers.LeaseLedger   // move to another context
  A.Note -> B.Memo                   // move and rename at once
}

The qualified form qualifies both sides (Ctx.Name -> Name reads as a column rename). The right side may be an aggregate, a workflow or a projection, declared in any file of the project; it is resolved on the merged model, and a bad one is a phase-⑦ error:

What each target takes along:

A move renders as ALTER TABLE … SET SCHEMA (creating the target schema first, like a create-table) and then a rename in the new schema when the name changes too. Postgres carries the table's indexes, constraints and owned sequences along. The renameTable step records the old schema in fromSchema:

migration "move-note" { A.Note -> B.Memo }

CREATE SCHEMA IF NOT EXISTS "b";
ALTER TABLE "a"."notes" SET SCHEMA "b";
ALTER TABLE "b"."notes" RENAME TO "memos";
CREATE SCHEMA IF NOT EXISTS "b";
ALTER TABLE "a"."note_charges" SET SCHEMA "b";
ALTER TABLE "b"."note_charges" RENAME TO "memo_charges";
CREATE SCHEMA IF NOT EXISTS "b";
ALTER TABLE "a"."lines" SET SCHEMA "b";
ALTER TABLE "b"."lines" RENAME COLUMN "note_id" TO "memo_id";
ALTER TABLE "b"."memo_charges" RENAME COLUMN "note_id" TO "memo_id";
ALTER INDEX "b"."lines_note_id_idx" RENAME TO "lines_memo_id_idx";
ALTER INDEX "b"."note_charges_note_id_idx" RENAME TO "memo_charges_memo_id_idx";

execute "CREATE SCHEMA IF NOT EXISTS b"
execute("ALTER TABLE \"a\".\"notes\" SET SCHEMA \"b\"")
rename table(:notes, prefix: "b"), to: table(:memos, prefix: "b")

A move between modules is recorded in the target module's chain — the module whose schema gains the tables. Its generation (an Initial when the module is new) carries the SET SCHEMA steps; on Ecto, a first generation that takes tables over writes them to a <module>_take_over_tables file ahead of its per-table creates. The source module records a generation too, holding one comment per table it hands off, so its schema in the chain stops listing them; it never drops them. A move can only find tables that a module still in the model records, so keep the source module (or its other contexts) in the model until the move is recorded.

Without the step, a rename or a move is a drop + create, and the destructive gate refuses it. Its message names the table step as the remedy whenever a table is dropped. The step is ledger-inert like a column rename: once the baseline holds the new relation and not the old, it emits nothing.

Data migrations (M-T2.3)

Structural DDL moves the schema; the migration block's data steps move the data — through the same migration chain, versioned and history-tracked like every DDL step. Two steps exist, both deliberately minimal:

Backfill — Agg.field = <expr> (keyword-free, like the -> rename). The expression is validated against a SQL-renderable subset (literals, enum values, sibling scalar fields as column refs, arithmetic/comparison/boolean operators, ?:, ??, now(), and datetime ± days/hours/minutes(n) — the full table is below; codes loom.migration-expr-unsupported, loom.backfill-type-mismatch, loom.backfill-target-invalid) and fires whenever the diff adds that column or flips it NOT NULL:

aggregate Order { placedAt: datetime  qty: int  status: string  rank: int }   // status, rank added this generation
migration "order-status" { Order.status = "pending"  Order.rank = qty + 1 }

ALTER TABLE "sales"."orders" ADD COLUMN "status" TEXT NOT NULL DEFAULT 'pending';
ALTER TABLE "sales"."orders" ALTER COLUMN "status" DROP DEFAULT;
ALTER TABLE "sales"."orders" ADD COLUMN "rank" INTEGER NULL;
UPDATE "sales"."orders" SET "rank" = ("qty" + 1) WHERE "rank" IS NULL;
ALTER TABLE "sales"."orders" ALTER COLUMN "rank" SET NOT NULL;

The two shapes differ on purpose. A constant value — one that reads no column: a literal, an enum value, now(), constant arithmetic — is the same for every row, so the column is added carrying it as its DEFAULT and the default is dropped again in the same migration (the field-default pair). On PG11+ that ADD is catalog-only: existing rows read the stored default without being rewritten, and the column is NOT NULL from the moment it exists. An expression that reads a sibling column needs a real per-row UPDATE, so the column is added nullable, filled, then set NOT NULL. See § Locks on a populated table for what each costs on a live table. On a NULL → NOT NULL flip of a column that already exists, every backfill is an UPDATE.

A duration is admitted in exactly one shape — a direct days(n) / hours(n) / minutes(n) operand of +/- on a datetime — and renders as Postgres interval arithmetic, which adds the same absolute span every backend's runtime adds. A bare duration, duration ± duration, or a duration on a non-datetime stays refused (there is no duration column to write it to):

migration "sla-deadline" { Ticket.dueAt = createdAt + hours(24) }

UPDATE "ticketing"."tickets" SET "due_at" = ("created_at" + make_interval(hours => 24)) WHERE "due_at" IS NULL;

A required column needs a non-NULL value. The type check runs before anything is recorded, and it knows that NULL propagates through every SQL operator. A backfill into a required column that can return an optional sibling's NULL is refused with loom.backfill-type-mismatch, because the UPDATE would write the NULL and the SET NOT NULL after it would fail at boot. Give it a fallback with ??, which renders as a CASE. A presence-tested ternary (label == null ? "untitled" : label) works the same way:

aggregate Contract { label: string?  title: string }   // title added this generation
migration "titles" { Contract.title = label ?? "untitled" }

UPDATE "legal"."contracts" SET "title" = (CASE WHEN ("label" IS NULL) THEN 'untitled' ELSE "label" END) WHERE "title" IS NULL;

Inherited fields are backfilled on the base. Write the step on the aggregate that declares the field, and it reaches every table that stores the field. For a TPH base that is the one shared table. A TPC (inheritanceUsing: ownTable) base has no table, so the step fans out to every concrete's own table, and each UPDATE reads that table's sibling columns. Moving owner up from Laptop to the TPC base Asset:

migration "owner-moves-up" { Asset.owner = "it" }

ALTER TABLE "hr"."phones" ADD COLUMN "owner" TEXT NOT NULL DEFAULT 'it';
ALTER TABLE "hr"."phones" ALTER COLUMN "owner" DROP DEFAULT;

laptops already has the column, so it gets nothing. The concrete spelling (Phone.owner = …) is refused with loom.backfill-unknown-field, and the message names the base to write it on.

The backfill subset is a permanent limit. A backfill runs as one SQL UPDATE per row, on every backend, so its expression is limited to what src/generator/sql-pg-expr.ts renders to Postgres. These are admitted:

Shape Renders as
literals: string, int/long/decimal/money, bool, null, now() the SQL constant (now() stays a call)
enum value its stored text
a sibling scalar field (primitive / enum / X id) the quoted column
( … ), unary - / ! the same, ! as NOT
+ - * / %, comparisons, && / || SQL operators. String + is ||, int / int is cast to numeric, and == null / != null is IS [NOT] NULL
c ? a : b, a ?? b CASE WHEN … THEN … ELSE … END
datetime ± days/hours/minutes(n) make_interval(…) arithmetic

Everything else is refused with loom.migration-expr-unsupported, which names the offending kind. That covers calls and intrinsics, member access (value-object leaves included), constructors, list and object literals, match, conversions, id, this, and a bare duration. The workaround is a raw sql step. Widening is not on a roadmap: a new shape is added only when a real model needs it, and it comes with a migration-evolution e2e for that shape.

Naturally ledger-inert like renames: once the column is baked into the baseline snapshot the step matches nothing. That inertness is the only silent outcome a backfill has: if the column is not in the baseline — so this migration is adding it — and nothing consumed the step, the derivation aborts with loom.migration-backfill-discarded rather than writing a migration in which the author's declared value never runs. (That is a compiler invariant, not a model error: it is what the rename heuristic used to produce when it swallowed the addColumn the backfill was waiting for.) Backfills target single scalar columns only — value-object leaves are excluded (Phoenix stores a VO as one :map column, so a leaf UPDATE would not be portable).

Raw SQL — sql "…" — the escape hatch for one-shot DML the backfill subset can't express. Emitted exactly once: the snapshot records each step's "<block>#<index>" key in appliedDataMigrations (raw SQL has no structural condition to be naturally inert against). Runs after the generation's structural steps, in declaration order. On Ecto both data steps render through execute/1; everywhere else they ride the same migration file as the DDL (renderBackfillSql is shared, so the DML is bit-identical).

migration "clear-legacy-notes" { sql "UPDATE sales.orders SET note = '' WHERE note IS NULL" }

sql before "…" — the opt-in modifier that emits the step ahead of the generation's structural steps instead. It exists for the fix-up the new DDL would otherwise reject — the canonical case is a dedupe that must run before a new unique (…) index is created over a populated table. Same exactly-once ledger, same declaration order among the before steps, same file (Ecto: execute/1 first in change/0). On a module's Initial generation there is nothing to precede — none of its tables exist yet — so before keeps the trailing position there, on every backend. before is a placement word, not a reserved keyword — it stays usable as a field, workflow or let name — and any other placement is rejected (loom.migration-sql-placement).

aggregate Account { number: string  unique (number) }   // unique added this generation
migration "dedupe-account-numbers" {
  sql before "UPDATE banking.accounts SET number = number || '-' || left(id::text, 4) WHERE id IN (SELECT id FROM (SELECT id, row_number() OVER (PARTITION BY number ORDER BY opened_at) rn FROM banking.accounts) x WHERE rn > 1)"
}

UPDATE banking.accounts SET number = number || '-' || left(id::text, 4) WHERE id IN (…);
--> statement-breakpoint
CREATE UNIQUE INDEX "accounts_number_uq" ON "banking"."accounts" ("number");

Raw SQL is verbatim — schema-qualify every relation (sales.orders, not orders): the generated DDL is schema-qualified, and the migration runners' search_path does not include context schemas, so an unqualified name fails at apply time (Flyway/Drizzle/psql alike).

Moving data between tables is a raw sql step. A backfill fills one column from its own row, so a reshape that moves rows elsewhere — splitting a contained collection into a new aggregate, say, or turning internal comments into InternalNote rows — has no keyword form. Declare the new table, then copy into it with INSERT … SELECT in the same generation (the raw step runs after the generation's DDL, so the target exists), and drop the old column or table in a later, destructive-gated generation:

migration "split-internal-notes" {
  sql "INSERT INTO support.internal_notes (id, ticket_id, body) SELECT gen_random_uuid(), ticket_id, body FROM support.comments WHERE internal"
}

This is the intended pressure valve, not a gap awaiting syntax (D-MIG-DSL-STEPS): a cross-table move is one-shot DML whose shape is specific to the model, and the raw step already runs once, in order, on all five backends.

A raw step is pinned to the module its block's other steps name; a block with only sql steps targets the system's single owner module, and an ambiguous scope (a block spanning modules, or no affinity in a multi-module system) aborts with MigrationSqlScopeError — split the block per module.

No transform clause — expand→migrate→contract. A type change with a value mapping needs no dedicated syntax, because backfills may reference sibling columns: generation 1 adds the new column and fills it from the old (Order.totalCents = total * 100); generation 2 drops the old column (destructive-gated, deliberate). An in-place alterColumnType keeps the blind USING col::type cast — Postgres fails loudly on impossible casts.

TPH concrete renames. Renaming a TPH concrete changes no DDL, but its rows keep the old kind discriminator — the rename cascade emits a ledgered one-shot fix-up:

migration "vendor-to-sponsor" { Vendor -> Sponsor }   // Vendor is TPH under Party

UPDATE "parties" SET "kind" = 'Sponsor' WHERE "kind" = 'Vendor';

A spent block says so — loom.migration-block-applied

A migration block is never re-run — renames and backfills are inert once the baseline reflects them, and sql steps are ledgered — so it may stay in the source forever. Once every step is already recorded in the baseline snapshot of every module it touches, generate system prints a suggestion (it never gates the exit code and is not counted as a warning):

loom.migration-block-applied suggestion: migration "sla-deadline" is fully applied: every step is
  already recorded in the schema snapshot (its renames and backfills are reflected in the schema,
  its sql steps are in the applied ledger), so it has no further effect. You may remove it from
  the source; the migration files already emitted keep its SQL.

What counts as recorded is read off the snapshot, never guessed: a column rename's new column exists and its old one does not; a table rename's new relation exists and its old one does not; a backfill's column exists and this generation emits no UPDATE for it; a sql step's key is in appliedDataMigrations. It stays silent on a first run, in the generation that runs the block, and while any one step still has work to do. Removing the block is safe because the emitted migration files are immutable and already carry its SQL — a database that has not applied them yet still gets it.

A unique index on a populated table — loom.migration-unique-on-existing-table

A diff cannot see the data. Adding unique (…) to an aggregate whose table already exists in the baseline emits a CREATE UNIQUE INDEX CONCURRENTLY (§ Locks on a populated table) that fails at boot if any stored rows collide — the generation's non-transactional unit fails and stays unrecorded, the service does not come up, and every later generation queues behind it. So generate system warns, at derivation time, beside the other phase-⑨ advisories:

loom.migration-unique-on-existing-table warning: migration for module "Retail": adds unique index accounts_number_uq on existing table banking.accounts (number). Rows already stored may violate it — … dedupe them AHEAD of the index in this same generation (declaring one silences this warning):
    migration "dedupe-…" { sql before "UPDATE <schema>.<table> SET … WHERE …" }
1 warning(s) in derived migrations.

Declaring any sql before step for the module in the same generation is read as "handled" and silences it; a plain (trailing) sql step does not — it runs after the index. Not raised for a table created this generation (empty), a module's first generation, or a rebuild of a unique index the baseline already had over the same columns (the drop + add a column rename causes, or a full unique narrowed to a partial one). A NEW partial unique (softDeletable's WHERE is_deleted = false) still warns — it is a uniqueness claim over the live rows. A warning never changes the exit code.

An enum that loses a member — loom.migration-enum-narrowed

Deleting (or renaming) a member of an enum that a persisted field uses narrows the column's value-set CHECK. The re-added constraint is NOT VALID, so the migration itself always applies — but the rows already holding the removed value stay, and the app cannot live with them: python, java, .NET and elixir map the stored string onto the enum when they load a row, so one such row fails the whole list route that reads it; node serves it off-contract; and any UPDATE of the row is refused by the new check.

The rule is the one a unique (…) on a populated table follows, because it is the same situation — a new constraint that rows already stored may violate, and a diff that cannot see the rows. It is a warning, not a destructive-change refusal: nothing is dropped and nothing fails at apply time (that is what puts a NULL→NOT NULL flip behind --allow-destructive), and the fix is a data step the author writes, not a flag:

enum EmploymentType { permanent }          // was { permanent, contractor }
aggregate Employee { kind: EmploymentType }

loom.migration-enum-narrowed warning: migration for module "Hr": enum check employees_kind_enum on existing table employees no longer admits 'contractor' in column kind. Rows already stored may still hold a removed value. … Remap the removed values AHEAD of the check in this same generation (declaring one silences this warning):
    migration "remap-…" { sql before "UPDATE employees SET kind = <one of 'permanent'> WHERE kind IN ('contractor')" }

migration "remap-contractor" {
  sql before "UPDATE employees SET kind = 'permanent' WHERE kind = 'contractor'"
}

UPDATE employees SET kind = 'permanent' WHERE kind = 'contractor';
ALTER TABLE "employees" DROP CONSTRAINT IF EXISTS "employees_kind_enum";
-- NOT VALID: already-stored rows are not re-checked. To verify them, run ALTER TABLE "employees" VALIDATE CONSTRAINT "employees_kind_enum"
ALTER TABLE "employees" ADD CONSTRAINT "employees_kind_enum" CHECK ("kind" IN ('permanent')) NOT VALID;

As with the unique-index warning, any sql before step for the module in the same generation silences it, and a trailing sql step does not (it runs after the check). Not raised for a widening (members only added), a module's first generation, or a table created this generation. The check is matched by name, so a narrowing that lands in the same generation as a rename of its table or column is not compared. Turning a plain string column into an enum adds a check with no baseline value set to compare against, so it does not warn either.

A unique index follows a softDeletable flip (partial ↔ full)

Under softDeletable the unique (…) index is partial (WHERE is_deleted = false), so a live row may reuse a soft-deleted row's key. The index name (<table>_<cols>_uq) depends only on the table and the columns, so flipping the capability keeps the name and changes only the predicate. The index diff compares the whole index (columns after renames, uniqueness, predicate, opclasses), not only the name: a same-named index whose shape changed is rebuilt, as a DROP INDEX + CREATE … INDEX, because Postgres cannot alter an index predicate. The drop rides the leading index-drop bucket and the create the trailing index-add bucket, so a predicate that names is_deleted is created after that column's ADD COLUMN, and dropped before its DROP COLUMN. All five backends render it: Ecto as drop index(…) + create index(…, where: …), the others as shared Postgres SQL.

// generation 1: aggregate Account with crudish { number: string  unique (number) }
aggregate Account with softDeletable, crudish { number: string  owner: string  unique (number) }

DROP INDEX "ledger"."accounts_number_uq";
--> statement-breakpoint
ALTER TABLE "ledger"."accounts" ADD COLUMN "is_deleted" BOOLEAN NOT NULL DEFAULT false;
--> statement-breakpoint
ALTER TABLE "ledger"."accounts" ALTER COLUMN "is_deleted" DROP DEFAULT;
--> statement-breakpoint
ALTER TABLE "ledger"."accounts" ADD COLUMN "deleted_at" TIMESTAMP WITH TIME ZONE NULL;
--> statement-breakpoint
CREATE UNIQUE INDEX "accounts_number_uq" ON "ledger"."accounts" ("number") WHERE is_deleted = false;

Before this, the diff paired indexes by name only. The flip emitted no index step, and the database kept whichever index the first generation created.

The migration-evolution tier runs both directions against a populated database on every SQL backend (runUniqueIndexEvolutionGate, npm run test:migration-evolution{,-python,-java,-dotnet,-elixir}):

Amending a generation that never applied

A recorded generation is frozen: it is a file in migrations/, and the next migrate add diffs against the schema it leaves. If that generation then FAILS at boot (the unique-index case above), recording the fix as a new generation puts it BEHIND the one that fails, so it never runs. A failed generation was never recorded by the migrator — each backend applies a generation inside a transaction on Postgres (Drizzle, EF Database.Migrate(), Flyway, Ecto, the python runner's __loom_migrations), so a failure leaves no history row — and it is therefore safe to replace it, as long as no database has applied it:

  1. ddd migrate remove app.ddd — once per entry the failed generation recorded (one per module it changed); each run deletes the newest.
  2. Amend the .ddd (e.g. add the sql before dedupe).
  3. ddd migrate add <name> again, then ddd generate system.

If any database DID apply the generation, do not replace it — record a new one instead. The other way out — running the fix by hand (docker compose exec db psql -U postgres -d <db> -c "UPDATE …") and restarting the service so the frozen generation re-runs — works, but leaves the fix outside the model.

A failed concurrent index is the exception. An index added to an existing table is built in the generation's second, non-transactional unit (§ Locks on a populated table), so when that is what fails — the unique-index case on an existing table — the generation's transactional half has already committed and is recorded. Do not migrate remove it. Dedupe by hand and restart: the concurrent unit is unrecorded, so it runs again, and its leading DROP INDEX CONCURRENTLY IF EXISTS clears the INVALID index the failed build left behind before rebuilding it.

Down migrations are no-op by decision (D-MIG-NO-DOWN): operators roll forward; recovery is backup + roll-forward. And data steps live in the DSL, not in per-backend stub files (D-MIG-DSL-STEPS) — one source, five backends.

Locks on a populated table

Every backend's runner applies a generation inside a transaction, so a lock a statement takes is held until that transaction commits. Two statement shapes used to make that a full-table outage on a live system: at 1M rows, an UPDATE backfill plus two plain CREATE INDEX statements held the table for ~25 s, and every concurrent read and write of it waited. Both are now rendered so they hold no long lock:

Step Rendered as Lock held while it runs
Column add backfilled with a constant (also a field default, an empty collection, softDeletable's false) ADD COLUMN … DEFAULT <c> NOT NULL + ALTER COLUMN … DROP DEFAULT ACCESS EXCLUSIVE for milliseconds: on PG11+ a non-volatile default is stored in the catalog, and no row is rewritten
Column add backfilled with an expression ADD COLUMN … NULL → UPDATE … WHERE … IS NULL → SET NOT NULL ACCESS EXCLUSIVE for the whole UPDATE and the SET NOT NULL scan. Batched backfills are not implemented
Index added to a table that already exists (resource … index:, unique (…), an FK's covering index) DROP INDEX CONCURRENTLY IF EXISTS + CREATE [UNIQUE] INDEX CONCURRENTLY, in a unit of its own none that blocks reads or writes. The build waits for transactions already open on the table
Index on a table created in the same generation inline plain CREATE INDEX none worth avoiding: the table is empty 
aggregate Order { ref: string  status: OrderStatus  priority: int }   // priority + the index added this generation
resource storeState { for: Store, kind: state, use: primary, index: [Order.status] }
migration "priority" { Order.priority = 1 }

-- <version>_<module>_priority.sql: applied in one transaction
ALTER TABLE "store"."orders" ADD COLUMN "priority" INTEGER NOT NULL DEFAULT 1;
ALTER TABLE "store"."orders" ALTER COLUMN "priority" DROP DEFAULT;
-- <version>_<module>_priority_concurrent.sql: applied outside any transaction
DROP INDEX CONCURRENTLY IF EXISTS "store"."orders_status_idx";
CREATE INDEX CONCURRENTLY "orders_status_idx" ON "store"."orders" ("status");

The non-transactional unit. Postgres refuses CONCURRENTLY inside a transaction block, and a multi-statement query counts as one. So the steps that need it (addIndex with concurrently, chosen once by src/ir/util/migration-steps.ts) leave the generation's file and become a second migration of the same generation. It is ordered right after the first, holds one statement per command, and goes into the runner's ledger on its own:

Backend Second unit How the runner applies it
Hono / Drizzle db/migrations/<tag>_concurrent.sql, first line -- loom:no-transaction, its own journal entry the generated db/migrate.ts (Drizzle's migrator with the same drizzle.__drizzle_migrations ledger and watermark) runs a marked file statement by statement outside a transaction, then writes its ledger row. Consecutive unmarked files still share one transaction. npm run db:migrate runs the same runner out of band (db/migrate-cli.ts); drizzle-kit migrate, which wraps everything in one transaction, is not wired
Python / FastAPI migrations/<tag>_concurrent.sql, same marker run_migrations() applies a marked file on an AUTOCOMMIT connection, then records its tag in __loom_migrations. Consecutive unmarked files share one transaction
Java / Flyway V<v>.<n>.1__<Mod>_<Name>Concurrent.sql Flyway runs a script that holds only CREATE/DROP INDEX CONCURRENTLY statements without a transaction. It refuses a script that mixes them with transactional ones, which is why the unit is its own script
.NET / EF Core Migrations/<slug>_Concurrent.cs one migrationBuilder.Sql(…, suppressTransaction: true) per statement
.NET / Dapper Migrations/<slug>_Concurrent.cs, InTransaction => false DbSchema.EnsureAsync runs its statements without a transaction, then records its id
Phoenix / Ecto <version + 1>_<module>_<name>_concurrent.exs with @disable_ddl_transaction true and @disable_migration_lock true ecto.migrate; drop_if_exists index(…, concurrently: true) + create index(…, concurrently: true)

Ecto orders and records files by an integer version, so the migration chain reserves version + 1 after any generation that has such a unit: the next migrate add numbers its entry at least two above it, and a chain whose entry takes the reserved version is refused when it is read.

Recovery. A concurrent build that fails (duplicate keys under a new unique index, a cancelled deploy) leaves an INVALID index of that name. It enforces nothing, but it would make a plain retry fail with already exists. The unit starts with DROP INDEX CONCURRENTLY IF EXISTS, and it is recorded only after its last statement, so the next boot drops the leftover and rebuilds. The generation's transactional half is already committed at that point; see § Amending a generation that never applied.

Recorded generations do not change. The concurrently flag is recorded with the step in migrations/. A generation recorded before the flag existed keeps its plain CREATE INDEX inside its one file, and keeps its UPDATE backfill, so a migration a database has already applied renders byte-identically and no Flyway checksum or Drizzle watermark moves.

Reshape detection — the loom.migration-shape-change gate (M-T2.4)

Flipping an aggregate's shape: relational|embedded|document or its TPH/TPC inheritance strategy reshapes the physical table — a full-table data move the column diff can't express (a relational row becomes (id, data, version); one shared TPH table becomes one table per concrete). To detect it, the snapshot TableShape carries two optional stamps: savingShape (aggregate root) and inheritance ({strategy, base} on TPH-shared / TPC-concrete tables). Both are optional — schemaVersion stays 1, and an unstamped pre-M-T2.4 baseline is simply not compared (the flip falls to the generic destructive gate, exactly as before — backward-compatible grace).

detectReshapes finds a saving-shape flip (matched table, savingShape changed) and an inheritance-strategy flip (TPH↔TPC, whose table names change, grouped by root base across the drop/create sets). A flip raises the dedicated MigrationShapeChangeError (loom.migration-shape-change) instead of the generic destructive listing. Under --allow-destructive there is no drop: the old table is renamed to <table>__pre_reshape (kept for the data move), the new shape is created empty, and a -- TODO reshape … recipe marks the INSERT … SELECT the operator runs. The backup is recorded in the written snapshot, so its cleanup drop is destructive-gated the next generation.

The same goes for every owned child table the new shape no longer has. A relational aggregate keeps its parts, value collections and X id[] join tables in tables of their own; shape: document / embedded folds them into the root, so they fall out of the next schema. Each one holding an ON DELETE CASCADE FK to a backed-up table (directly, or through another child) is renamed to <table>__pre_reshape beside the root, with its own TODO recipe; the gate's listing names them. Its FK follows the root's rename (Postgres and Ecto carry an FK across RENAME), so the backups still join to each other for the copy, and the next generation's cleanup drops the children before the root. An X id reference is a restrict FK, not ownership, so another aggregate removed in the same generation is still an ordinary DROP TABLE.

aggregate Cart shape: document {   // was relational
  owner: string
  lines: Line[]                    // value collection → cart_lines
  contains items: Item[]           // part → items
  entity Item { name: string }
}

-- relational Cart → `shape: document` Cart, under --allow-destructive
DROP INDEX "cart_lines_cart_id_idx";
DROP INDEX "items_cart_id_idx";
ALTER TABLE "carts" RENAME TO "carts__pre_reshape";
ALTER TABLE "cart_lines" RENAME TO "cart_lines__pre_reshape";
ALTER TABLE "items" RENAME TO "items__pre_reshape";
CREATE TABLE "carts" ("id" UUID NOT NULL, "data" JSONB NOT NULL, "version" INTEGER NOT NULL, PRIMARY KEY ("id"));
-- TODO reshape (carts: saving shape relational → document): copy data from carts__pre_reshape into the new shape (e.g. INSERT … SELECT), then drop the backup under --allow-destructive
-- TODO reshape (carts: saving shape relational → document): copy data from cart_lines__pre_reshape into the new shape (e.g. INSERT … SELECT), then drop the backup under --allow-destructive
-- TODO reshape (carts: saving shape relational → document): copy data from items__pre_reshape into the new shape (e.g. INSERT … SELECT), then drop the backup under --allow-destructive

The migration chain — migrations/ beside the source

Every generation a model has recorded lives beside its entry .ddd, in migrations/, one JSON file per (generation, module): src/system/migration-chain.ts owns the format and every rule, src/cli/migrate.ts only moves the files. Each entry carries the backend-neutral steps and the module's schema after them, so generate system renders every backend's migration files — the whole chain, every run — from migrations/ alone. The output tree holds no migration state, can be deleted, and regenerates byte-identically anywhere: CI, a fresh clone, a second environment.

app.ddd
migrations/
  20261025141500_sales_initial.json
  20261026141500_billing_initial.json
  20261102093000_sales_rename_order_qty.json

ddd migrate add initial app.ddd             # first recorded generation: each module's Initial
ddd migrate add rename-order-qty app.ddd    # every later schema change, kebab-case name
ddd migrate status app.ddd [--json]         # each module's head, and what is not recorded yet
ddd migrate remove app.ddd                  # delete the newest entry (one no database has applied)
ddd migrate rebase app.ddd                  # after a merge leaves two heads, re-record one on top of the other
ddd generate system app.ddd -o out          # renders the chain; never records anything

Commit migrations/. It is the record a database's migration history is checked against.

The playground keeps the same files in its workspace, beside the entry .ddd. Its generate renders the chain and refuses drift the same way. Its Migrations tab shows the status and records new entries (migrate add); see playground.md.

A model with no migrations/ has nothing recorded. generate system then emits each owning module's initial migration, derived from the model and numbered from BASE_VERSION, with nothing to drift from — every example and fixture in this repository works that way. The first migrate add starts the chain, and from then on generate system refuses a model that has moved past the chain's head (MigrationDriftError, naming the modules) until the change is recorded.

Versions. An entry's version is the UTC clock at migrate add (YYYYMMDDHHMMSS), moved above the head when the clock is behind it, and above head + INITIAL_RESERVATION (one "day", 1_000_000) when the head is a module's Initial — one initial expands into many files on Ecto, which numbers one per table from the generation's version. So a version is unique across modules and strictly increasing along the chain, and every backend's migrator (Drizzle's watermark, Flyway's out-of-order refusal, Ecto's per-file versions) sees a new generation above everything it already applied.

One list. Every entry names the entry recorded just before it, of any module, by id (<version>_<module>_<name>, its file name without .json) — an id rather than a version, because two branches recording on top of the same head can draw the same version. Two entries naming the same predecessor are two heads: what two branches that each recorded a migration look like once merged. generate system and migrate status then refuse the chain (MigrationChainError) and point at ddd migrate rebase, which deletes the later-recorded branch's entries (--move <version> picks the other) and records the merged model's changes again on top of the remaining head, under the moved migration's name (--name to change it) and with its --allow-destructive decision. The same error covers a missing predecessor, an entry numbered at or below the one it follows, a file name that does not match its contents, two entries of one module sharing a name, and unparseable JSON (an unresolved merge conflict).

migrationsOwner — one backend per module owns schema

Schema emission is assigned to exactly one deployable per module so two backends sharing a module don't both create the tables. assignMigrationsOwner (enrichments.ts) walks sys.deployables in declaration order and picks the first deployable that hosts any context of the module AND whose platform owns a database (PlatformSurface.needsDb, read via descriptorFor — no hardcoded platform list). Failing that, migrationsOwner stays undefined and no migrations are emitted.

At distribution time (migrationsForDeployable in system/index.ts), every needsDb deployable that hosts a context of the module still receives that module's MigrationsIR (each compose service has its own database, so each must run the migrations) — but only the migrationsOwner derivation gates whether the slice is built at all.

How IR maps to tables

schemaFromModule collects every aggregate in the module (alphabetical) and produces tables. The shape depends on the aggregate's saving shape, persistence mode, and inheritance:

Aggregate → table. aggregate Order → table orders (plural(snake(name))), id PK typed from idValueType, one column per scalar/enum/X id field. The spelling lives in one place, tableNameFor (src/util/storage-names.ts), with the per-node accessors (aggregateTableName, partTableName, projectionTableName, associationTables) in src/ir/enrich/storage-names.ts; every backend's ORM mapping and the migrations read them, and the aggregate's route segment uses the same spelling (routeSegmentFor). The order matters for a name that ends in an acronym: ProductSKU → product_skus, never product_sk_us.

context Sales {
  aggregate Order {
    placedAt: datetime
    customer: Customer id
    status: OrderStatus
  }
}

CREATE TABLE orders (
  id UUID NOT NULL,
  placed_at TIMESTAMP WITH TIME ZONE NOT NULL,
  customer_id UUID NOT NULL,
  status TEXT NOT NULL,
  PRIMARY KEY (id),
  FOREIGN KEY (customer_id) REFERENCES customers ON DELETE RESTRICT
);
CREATE INDEX orders_customer_id_idx ON orders (customer_id);

A Customer id field becomes a customer_id FK (ON DELETE RESTRICT) plus a covering index. Type mapping: int→INTEGER, long→BIGINT, decimal→DECIMAL (unbounded), money→DECIMAL(19, 4) (the canonical bound every ORM layer also declares — see money-scale.ts), string→TEXT, bool→BOOLEAN, datetime→TIMESTAMP WITH TIME ZONE, guid→UUID, json→JSONB. Bounds are part of the column's identity in the diff: a bare-DECIMAL baseline against a bounded target emits an alterColumnType (see § Destructive changes).

Containments → child tables. A contained entity part gets its own table with a <parent>_id FK (ON DELETE CASCADE) back to the owner. The FK-dependency sort guarantees the parent is created first even though pipelines sorts before projects alphabetically.

Value objects → flattened columns. A price: Money field destructures into price_amount, price_currency (recursively for nested VOs), each tagged with a voGroup so Phoenix/Ecto can regroup them into a single :map. Relational backends emit the columns as-is.

Renaming a value-object field moves its json key. Where a value object is ONE json column (Phoenix's :map, Dapper's jsonb), a leaf rename (street → road) renames no column, but every stored document still holds the old key. The delta therefore rewrites the key in every row — snake on Phoenix ('street'), the PascalCase property name on Dapper ('Street') — and, on Dapper, in every element of a value-object list's jsonb array:

valueobject Address { street: string  city: string }   // v1
valueobject Address { road: string    city: string }   // v2
aggregate Person with crudish { name: string  home: Address  past: Address[] }

rename table(:person_past, prefix: "c"), :street, to: :road
execute("DO $$ BEGIN IF EXISTS (SELECT 1 FROM \"c\".\"persons\" WHERE jsonb_typeof(\"home\") = 'object' AND \"home\" -> 'street' IS NULL AND \"home\" -> 'road' IS NULL) THEN RAISE EXCEPTION '…'; END IF; END $$")
execute("UPDATE \"c\".\"persons\" SET \"home\" = (\"home\" - 'street') || jsonb_build_object('road', \"home\" -> 'street') WHERE jsonb_typeof(\"home\") = 'object' AND \"home\" -> 'street' IS NOT NULL")

The key is read off the column name, which cannot tell a field of a nested value object (home.geo.lat) from a flat one (home.geoLat), or recover an acronym's casing. So the first statement checks the data: when a stored value has neither the old key nor the new one, the migration stops with an error rather than moving nothing. Write that rename as a sql step instead.

An OPTIONAL value object carries a null-consistency CHECK. Flattening an optional VO makes N independent nullable columns, but every backend reads the group atomically — java's record compact constructor runs the VO's invariant, node asserts each leaf non-null, python passes each leaf straight in. A partially-null row therefore crashes the load, and nothing else in the schema forbids one (a hand-written UPDATE, a bad backfill, a future partial-update path). The builder emits one CheckShape per optional VO node:

valueobject Geo  { lat: decimal  lng: decimal }
valueobject Addr { line1: string  line2: string?  city: string  geo: Geo? }
aggregate Order { shipTo: Addr? }

CONSTRAINT "orders_ship_to_null_consistent" CHECK (
  ("ship_to_line1" IS NULL     AND "ship_to_city" IS NULL) OR
  ("ship_to_line1" IS NOT NULL AND "ship_to_city" IS NOT NULL)),
CONSTRAINT "orders_ship_to_geo_null_consistent" CHECK (
  ("ship_to_geo_lat" IS NULL     AND "ship_to_geo_lng" IS NULL) OR
  ("ship_to_geo_lat" IS NOT NULL AND "ship_to_geo_lng" IS NOT NULL))

The grouping rules, each of which exists to keep the constraint from rejecting valid data:

On an EXISTING database the ALTER path renders NOT VALID: stored rows may already violate the constraint (nothing enforced it before), so validating retroactively would abort the migration on exactly the database that most needs the guard. Every future INSERT/UPDATE is enforced immediately; the emitted comment names the ALTER TABLE … VALIDATE CONSTRAINT … an operator runs to back-check the old rows deliberately. A createTable carries its checks inline, where there is nothing to validate.

Phoenix emits none of this kind — Ecto stores a value object as one :map cell, so the group cannot be half-written and the leaf columns the constraint names do not exist. It does emit the enum kind below.

Checks are derived for the domain tables only: the aggregate root, its TPH shared table, its contained-part tables, and its value-collection child table — the rows a repository hydrates back into a real value object. The projection read-model and workflow-state tables also flatten, but both make their non-key columns nullable on purpose (a fold or a workflow step upserts only the fields the event it is handling carries), so a half-filled row there is the designed state and a constraint could fail on legitimate data.

An enum column carries its VALUE SET as a CHECK. An enum persists as TEXT on every backend, so without this the set of legal values is not part of the schema at all — and two ordinary evolutions were therefore complete no-ops, emitting no migration, no constraint and no warning:

change before now
skill: string → skill: Skill column type unchanged (TEXT → TEXT) ⇒ empty diff addCheck
enum Skill { physio, gp, dentist } → { physio, gp } the enum is not in the schema ⇒ empty diff dropCheck + addCheck

In both cases the database went on accepting a value the model had just outlawed, while the same generation's OpenAPI $ref-ed the field to the enum and the same generation's zod / pydantic client refused it — the server serving data its own published contract forbids.

enum Skill { physio, gp, dentist }
aggregate Appt { skill: Skill  backup: Skill? }

CONSTRAINT "appts_skill_enum"  CHECK ("skill" IN ('physio', 'gp', 'dentist')),
CONSTRAINT "appts_backup_enum" CHECK ("backup" IS NULL OR "backup" IN ('physio', 'gp', 'dentist'))

The same NOT VALID rule as above is what makes this safe to land on a populated database: the ALTER leaves already-stored rows alone (so narrowing an enum can never fail a deploy) while refusing every future write of a removed value; the emitted comment names the VALIDATE CONSTRAINT an operator runs to audit the old rows deliberately. The deploy surviving is not the app surviving, though: a row still holding a removed value fails to load on most backends, so a narrowing on a populated table raises loom.migration-enum-narrowed until a sql before remap is declared.

Covered: a scalar enum column, at either nullability, on the aggregate root, a TPH shared table (where a concrete's column is forced nullable, so it gains the null arm), a part table, and an enum nested inside a flattened value object (shipTo: Address? with Address { country: Country } → ship_to_country). Not covered, each because the constraint is a different expression rather than a missing case: an enum array (skills: Skill[] → TEXT[], which needs <@ ARRAY[…]), an enum inside a VO the lookup cannot resolve (it collapses to one json cell — no column to constrain), and the projection / workflow-state tables, excluded for the same partial-upsert reason as above.

All five backends carry this kind, Phoenix included — it names one real column that exists everywhere, and an elixir app that skipped it would be the single backend whose database still accepted a deleted value.

Reference collections → join tables. A X id[] field never produces a column. Enrichment derives one AssociationIR per such field, and the builder lays down a join table keyed by (owner_fk, target_fk) with both FKs ON DELETE CASCADE, and nothing else: X id[] is a set, so the composite primary key is the whole row and there is no ordinal column. Read-back order is a read-time projection — every backend orders the rows by the target FK id.

aggregate Order { items: Product id[] }

CREATE TABLE order_items (
  order_id UUID NOT NULL,
  product_id UUID NOT NULL,
  PRIMARY KEY (order_id, product_id),
  FOREIGN KEY (order_id) REFERENCES orders ON DELETE CASCADE,
  FOREIGN KEY (product_id) REFERENCES products ON DELETE CASCADE
);
CREATE INDEX order_items_product_id_idx ON order_items (product_id);

A VO array (charges: Money[]) produces an id-less child table (<parent>_<field>) of flattened VO columns keyed by (parent_fk, ordinal). Phoenix creates the same child table plus a synthetic id uuid primary key (the Ecto has_many row identity cast_assoc needs) and no timestamps — the same table whether it arrives in the initial generation or in a later delta. Dapper has no child table: the list is one jsonb column on the parent.

Inheritance. TPC (inheritanceUsing: ownTable): the abstract base emits no table, each concrete is standalone with the merged field set. TPH (sharedTable): one shared table named for the base, with id, a kind discriminator (TEXT NOT NULL), the base's own columns, then every concrete's own columns forced nullable and de-duplicated by name. A TPH concrete's contained parts FK to the shared base table. See inheritance.md.

Other persistence shapes. shape: document → a (id, data jsonb, version int) triple; shape: embedded → a queryable root row with one JSONB column per containment; persistedAs: eventLog and eventSourced workflows → an append-only <name>_events stream keyed by (stream_id, version). Correlation-bearing workflows get a state table; a durable channel adds the shared __loom_outbox table.

Optimistic concurrency (versioned). Every aggregate is versioned by default — the macro layer splices the versioned capability into each one that is not event-sourced (persistedAs: eventLog, whose (stream_id, version) stream already is its concurrency control) — so a plain aggregate gains a version INTEGER NOT NULL DEFAULT 1 state column; writing with versioned is redundant. Every backend's save path emits a guarded UPDATE … WHERE id = $1 AND version = $2 (bumping version) and returns HTTP 409 when zero rows match — a lost-update guard. See capabilities.md.

Manual indexes. A resource index: [Entity.col, Entity.(a, b)] binding (D-INDEX-INFRA) lands each entry as a non-unique CREATE INDEX on the named entity's table — the manual companion to the auto-derived FK covering indexes above. The advisory loom.index-suggestion lint (D-INDEX-SUGGEST) points at filtered columns that lack one; it never emits an index on its own. See resources.md.

Schema qualification. Each table is stamped with its owning context's Postgres schema (snake(context.name) by default, or the dataSource schema: override). The SQL renderer emits CREATE SCHEMA IF NOT EXISTS and qualifies table/FK/index references. undefined means the unqualified public schema.

The Postgres SQL renderer

renderPgStep(step) (src/generator/sql-pg.ts) translates one MigrationStep to Postgres DDL. It is shared by the TS, .NET (EF Core and Dapper), Python, and Java backends — calling the same renderer means a migration written by any of them is bit-for-bit equivalent SQL. Phoenix is the exception: it stays in Ecto DSL (its output is Elixir, not SQL) so ecto.migrate keeps working, and translates MigrationStep itself in elixir/migrations-emit.ts.

It also exports renderSeedRowInsert (the raw seed-path INSERT, shared so seed SQL is bit-identical cross-backend).

Where output lands, and boot-time application

Backend Output path Applied at boot via
Hono / Drizzle db/migrations/<tag>.sql + meta/_journal.json migrate() from the generated db/migrate.ts (Drizzle's migrator plus non-transactional files, § Locks on a populated table)
.NET / EF Core Migrations/<slug>.cs (migrationBuilder.Sql(...)) Database.Migrate() → __EFMigrationsHistory
.NET / Dapper Migrations/<slug>.cs (a LoomMigration with its Statements) DbSchema.EnsureAsync → public.__loom_migrations (see below)
Phoenix / Ecto priv/repo/migrations/<ts>_create_<table>.exs ecto.migrate
Java / Spring src/main/resources/db/migration/V<v>.<n>__<Mod>_<Name>.sql Flyway
Python / FastAPI migrations/<tag>.sql run_migrations() from the FastAPI lifespan; __loom_migrations tracking table

Files use a module-qualified tag (<version>_<module>_<name>): the version is already unique, but every module's first generation is named "Initial", and the module keeps a filename saying which module it migrates (Ecto's migration module is <Module><Name> for the same reason — its names are app-wide). Down migrations are no-ops everywhere (operators roll forward, not back).

The fixed companion tables — the transactional outbox (__loom_outbox), the command-audit log (audit_records) and the provenance history (provenance_records) — are ordinary MigrationsIR tables, derived once by outboxTableShape / auditTableShape / provenanceTableShape in src/system/migrations-builder.ts and gated on the module actually using the feature. They are machinery, not domain: cross-context (rows carry a target_type), so they take no context schema. Emitting them from one shape is deliberate — each was once hand-written per backend, and both consolidations turned up live defects the duplicated copies had been hiding.

The per-aggregate half of provenance is ordinary schema too: each provenanced root field's co-located <field>_provenance column (nullable JSONB) is derived onto its aggregate's root table by withLineageColumns in the same file. No backend hand-emits provenance or audit DDL, and nothing sits at a fixed far-future version outside the chain. Like any other column, the lineage column is versioned, diffed and recorded, so the rules are the usual ones:

A database whose Flyway history still holds the retired 29991231235959 provenance migration has a "future" applied version. Flyway skips that row, but it still treats every later chain version as out of order, so that database needs spring.flyway.out-of-order=true once. On Drizzle the retired entry's odd journal slot sits below the next generation's even slot, so nothing is hidden. The applied-set runners (EF, Dapper, Ecto, the python runner) just ignore the stale row.

persistence: dapper — the same chain, on the jsonb layout (M-T2.17)

Until D-DAPPER-ALTER was built, the Dapper adapter applied no migrations: its DbSchema.cs ran CREATE TABLE IF NOT EXISTS (in public) on every boot, so after the first deploy every model change was silently ignored — added and renamed columns, backfills, sql / sql before steps, index flips. It now applies the same MigrationsIR chain EF applies (dotnet/emit/dapper-migrations.ts):

Compose and k8s need nothing for the runner. The per-deployable db-bootstrap service (#3141) only ever creates the database; the schema inside it is the app's to migrate at boot, on every adapter. The chart runs no migration job either: each app migrates itself at boot, and the advisory lock makes that safe with more than one replica.

The Drizzle journal's ordering key

Drizzle is the only backend in the table above whose migrator is a watermark rather than an applied-set: migrate() reads the single highest __drizzle_migrations.created_at and applies a journal entry only when lastApplied.created_at < entry.when, strictly. EF, Flyway, Ecto and the __loom_migrations runner (Python, and .NET on persistence: dapper) all record each applied migration by identity, so for them an inserted migration is simply pending and nothing can renumber.

That makes when load-bearing in a way a filename is not. Three requirements:

  1. strictly increasing down the journal — a tie or a decrease silently skips an entry, and its tables are never created;
  2. never changing for an entry already applied — a bumped when re-runs it, and the re-run dies on already exists;
  3. strictly greater, for a newly appended entry, than for every entry already applied — otherwise the watermark hides it forever.

when is therefore the entry's position in chain order: when = epochMillis(BASE_VERSION) + ordinal * 2. The chain only grows at its head, with a version above every existing one, so an entry's position never moves (2) and a new entry always lands last (1, 3). Entries take the even slots. The factor of two stays because deployed databases already carry these values, and a database that applied the retired out-of-chain provenance migration holds it at the odd slot just above its then-newest entry, which the next generation's even slot still clears. The runtime gate is the journal-ordering leg of migration-evolution-e2e.

Migration-evolution gate — proving migrations evolve on data (M-T2.13)

The tiers above prove a migration emits (test:k8s, corpus compile) and applies to a fresh database (the obs / tenancy e2e migrate an empty DB at first boot). Neither proves a migration evolves correctly on data that already exists — the silent-data-loss class. The migration-evolution-e2e.yml gate closes that, with two runtime checks per SQL backend against a real Postgres:

  1. migrate-vs-create equivalence. The domain schema after applying the full chain (Initial v1 → delta v2) must be byte-for-byte equal to a fresh create of the head (v2) model. A drift means an ALTER diverged from what a CREATE of the same shape produces. The comparison is an order-independent per-column/PK fingerprint (via host psql on information_schema): physical column order legitimately differs — a rename leaves a column in place while an add appends — so columns are sorted by name, and each backend's migration-runner bookkeeping table (__drizzle_migrations, __loom_migrations, __EFMigrationsHistory, flyway_schema_history, schema_migrations) is excluded so only the domain schema is compared.
  2. Populated forward-migrate. Seed a v1 row through the REST surface, record the .ddd's v2 with ddd migrate add and regenerate the same tree (so the chain carries the forward migration), apply it (at boot for node/python/dotnet/java; an explicit mix ecto.migrate for vanilla Phoenix), and assert the row survives with correct values — the renamed column keeps its value, the back-filled NOT-NULL column is populated, the nullable add is NULL. A destructive/lossy change the derive should have gated shows up here as a lost or wrong row.

The v1→v2 delta (test/e2e/fixtures/migration-evolution/{base,evolved}.ddd) is deliberately gate-clean (explicit rename intent + a backfill make it non-destructive, so no --allow-destructive), exercising the three data-preserving move classes together: a value-preserving renameColumn, a NOT-NULL add backfilled with a constant (add-WITH-DEFAULT → DROP DEFAULT), and a plain nullable add. The unique-index leg (unique-evo-base.ddd) rebuilds an index on a populated table, so it is the runtime proof of every runner's non-transactional CREATE INDEX CONCURRENTLY unit.

The shared harness (test/e2e/support/migration-evolution-harness.ts) owns the Postgres lifecycle (one server, chain + fresh databases), the fingerprint, and both assertion sequences; a thin per-backend BackendDriver supplies only the boot mechanics (mirroring the tenancy-isolation recipes). Opt-in, per backend: npm run test:migration-evolution{,-python,-java,-dotnet,-dapper,-elixir} (each gated on LOOM_MIGRATION_E2E[_…]=1; needs docker or LOOM_MIGRATION_PG_URL plus host psql). The gate runs per-PR (path-scoped) and in the merge queue; see ci-gating.md. The dapper leg runs every gate the dotnet leg runs on persistence: dapper.

Semantic changes that emit no migration — exact decimal arithmetic (RS-37)

Not every change to what lands in a column is a schema change. When RS-37 made decimal arithmetic exact on node and python (M-T5.22 — the other three backends already computed it exactly), the column type did not move, so phase ⑨ derives no migration and none is needed. What moved is the value those two backends compute and then write: a derived x: decimal = a + b, or an operation's total := total + x, now stores 0.3 where it stored 0.30000000000000004.

Historical rows are not rewritten. A row node or python persisted before the change keeps its binary-float value, so an unchanged input can read back 0.30000000000000004 from an old row and 0.3 from a new one — and a find … where equality against a stored computed value can match one and miss the other. A stored derived column can be brought into line by re-saving the row through the regenerated backend (a save recomputes every derived from the row's own fields) or by a one-off UPDATE … SET x = <the same expression in SQL> — Postgres numeric arithmetic is exact, so the SQL answer is the new answer. A value an OPERATION wrote cannot be recomputed that way — the inputs it was computed from are gone — so it keeps its old spelling unless the deployment rounds it itself. Loom emits no backfill for either: it has no record of which stored values were computed rather than entered.

A column type that moved without a migration — elixir datetime (RS-38)

RS-38 (wave C5 moment 5b, ledger F2-W-06) moved the Phoenix backend's declared datetime columns from :utc_datetime — Ecto's timestamp(0), SECOND precision — to timestamptz, the microsecond column the other four backends already used, read through the Loom.Datetime Ecto type (milliseconds on the wire). The MigrationsIR did not change: the column is still a datetime, and phase ⑨ derives no migration step for it. What changed is how the elixir migration emitter renders that column type, so:

Relationship to .loom/ and wire-spec.json

Two change records sit side by side and are easy to conflate:

ddd diff <file.ddd> [--baseline <old.ddd> | --git <ref>] reads both at once and classifies each change: a wire change that moves the contract under an existing client is breaking-for-clients; a migration the destructive gate above refuses (including a reshape) is breaking-for-data. That verdict comes from the gate itself, through buildMigrations' destructiveRefusals sink, which records every refusing module and derives it as under --allow-destructive instead of throwing on the first. Everything else is compatible. A rename declared in a migration block is one rename on both sides: a compatible RENAME COLUMN, and a single property-renamed wire change, which is still breaking for clients. With no flag the committed chain is the schema baseline. The command exits 1 on a breaking change; see tools.md § diff.

See loom-artifacts.md for the full .loom/ bundle and technical.md § Phase ⑨ for the orchestration walk-through.

Language-version migrations

The rest of this page is about schema migrations the compiler derives. This section is about the other kind: a language change that moves what an unchanged .ddd means, and the mechanical edit that keeps a project's behaviour across it.

The enforcement: default flip (M-T3.1)

What changed. An auth { … } block that writes no enforcement: used to be in enforcement: opt — only members that declare a requires gate were gated, and every other client-reachable command and read served any authenticated caller. It is now in enforcement: denyByDefault: every client-reachable command (operations, creates, destroys, workflow starters and handlers, route-bound command/query handlers) and every declared read (named finds, projections, workflow-instance reads, audited history) on an auth: required deployable must carry a requires gate, or the build fails with loom.default-deny-ungated / loom.audit-history-ungated (auth.md lists every arm and the two warning-only exceptions).

What did not change. A system with no auth { … } block has no enforcement posture and is not checked, before or after. enforcement: has no effect on the generated code — denyByDefault is a compile-time refusal, not a runtime filter — so a model that validates under both postures emits byte-identical output under either. A gated route answers 403 exactly as it did.

Keeping the old behaviour. Write enforcement: opt into the auth block. The codemod does it for every block that names no enforcement::

node scripts/codemod-enforcement-opt.mjs .            # rewrite every *.ddd under .
node scripts/codemod-enforcement-opt.mjs --list .     # print what it would rewrite
node scripts/codemod-enforcement-opt.mjs --check .    # exit 1 while any block still relies on the default

auth {
  enforcement: opt          // inserted by the codemod
  provider: keycloak
  oidc { issuer: env("OIDC_ISSUER") clientId: env("OIDC_CLIENT_ID") }
}

It is a text edit — comments and formatting outside the inserted clause are untouched, a commented-out auth { is not a block, and a block that already names enforcement: (either value) is left alone, so it is safe to re-run.

Adopting the new default instead. Delete the enforcement: opt line (or never run the codemod), run ddd parse, and answer each loom.default-deny-ungated with a requires <expr> — requires true where a member is intentionally public. with crudish(requires: <Policy>) gates the members the macro emits.