Observability

Every generated backend emits one shared, machine-parseable log shape so a dashboard, alert, or jq query written once works across every deployable platform — Hono (Node.js), .NET (ASP.NET Core), Phoenix (Elixir), Java (Spring Boot), Python (FastAPI).

The wire-shape pattern, applied to logs. One catalog defines the events; per-backend renderers consume it; the framework's standard logger on each platform writes the lines.

The envelope

Every emitted line is a single JSON object on one line. Four keys are always present:

Two further carrier ids ride on every line emitted inside a request frame (absent on boot-time lines), so a log line joins to the audit / provenance rows written in the same frame:

Both come from the ambient execution-context carrier — see request-context.md. Each backend binds them at its native logging seam (pino mixin, ASP.NET BeginScope, Java MDC, the Python contextvar formatter, Elixir Logger.metadata), read at log time so per-frame accuracy holds.

Catalog-specific structured fields ride alongside the envelope as their own top-level keys (method, path, status, duration_ms, aggregate, workflow, …) — not nested under a data field. This keeps jq '.event == "request_end" | .duration_ms' working on every backend without dialect differences.

The catalog

src/generator/_obs/log-events.ts is the single source of truth. Every event pins its event name, level, and field set. Per-backend renderers (src/generator/_obs/render-{hono,dotnet,phoenix}.ts, plus the Java and Python renderers under their backend dirs — src/generator/java/emit/observability.ts and src/generator/python/emit/obs.ts) consume it; a typo at any generator call site is a typecheck error, not a runtime missing-event surprise.

Stability promise: additive only. New events / new optional fields won't break consumers. Renaming or removing requires a downstream migration.

Levels are concepts, not verbosity tiers

Filter to warn and you see only faults. Filter to info and you see the domain narrative without the noise. The --trace switch is the only way to inject trace lines into domain methods (kept off by default so the default artefact stays pure).

Every backend reads the LOG_LEVEL env var at boot (compose sets info; accepted trace/debug/info/warn/error, case-insensitive, anything else falls back to info). Phoenix has no trace level, so it maps to :debug; warn/warning map to :warning. In a release the level is set in config/runtime.exs, so Ecto's per-statement SQL logging (:debug) stays out of the stream unless LOG_LEVEL=debug.

Catalog excerpt

The full list lives in src/generator/_obs/log-events.ts. Highlights:

Lifecycle bracket (every backend): server_starting → server_listening → server_shutdown → server_drained

Request bracket (every backend): request_start → request_end — with method, path, status, duration_ms.

Domain narrative (info — emitted on every domain action): aggregate_created, operation_invoked, event_dispatched, workflow_started, workflow_completed.

Workflow failure (error): workflow_failed — the terminal event of a run that did NOT reach workflow_completed, carrying workflow and error. It is what makes workflow_started readable: without a terminal event on the failure path, "started but never finished" is indistinguishable from "still running", and any started/completed pairing leaks one row per failure. Emitted by node (from the workflow router's onError, naming the workflow off the request context), java (a catch on the child-frame try), python (an except around the route body) and elixir (the {:error, _} arm of the result, which is result-tuple shaped rather than exception shaped). .NET does not emit it yet — its handler body is one assembled string whose non-transactional path would have to be re-indented to sit inside a try, and --sourcemap anchors that body's statement fragments by exact-text search, so the indent would silently drop them from the map. The gap is recorded by a test in test/generator/workflow-lifecycle-log.test.ts.

Domain faults (warn — recoverable): domain_error, forbidden, not_found.

Framework faults (warn — the request never reached the domain): client_error. Emitted where the FRAMEWORK refuses a request before any handler runs — an unmatched route, a verb the path does not serve, a body it could not read. Kept distinct from domain_error because no aggregate was resolved, so a dashboard can separate "the client called something that isn't there" from "the client called something that refused".

System faults (error): internal_error, extern_handler_threw, migration_failed, db_error.

Database lifecycle (info / warn / error): db_connecting, db_connected, db_disconnected (warn), db_pool_exhausted (warn), db_error (error), db_connect_retry (warn — attempt, max_attempts, delay_ms, error).

Every backend retries its boot-time migration run while the database is not reachable yet (connection refused, DNS not resolvable, "the database system is starting up") with one shared policy — 10 attempts, 500 ms doubling to a 10 s cap (src/generator/_obs/boot-db-retry.ts) — logging db_connect_retry per wait; a SQL failure is never retried, and the last attempt still aborts boot. Hono, Python, .NET and Phoenix run the loop in-process; Java hands the same wall-clock budget to Hikari's initialization-fail-timeout (which retries silently, 1 s apart). The system docker-compose.yml marks every app service restart: unless-stopped, so a backend that exhausts the budget, or crashes later, is brought back by compose.

Migrations (info / error): migrations_starting, migration_applied, migrations_complete, migration_failed (error). The error field of migration_failed names the database's reason, not just the failing statement: Hono walks drizzle's wrapped error chain and appends the Postgres code / detail / constraint (e.g. Key (number)=(T-1) is duplicated. for a unique index over existing rows); .NET spells out the PostgresException's SQLSTATE, message and constraint (Npgsql redacts detail unless the connection string sets Include Error Detail); Python's str(exc) and Phoenix's Exception.message/1 already carry the driver's detail. Java's Flyway callback has no access to the failure, so its line says only migration failed; Spring Boot's startup-failure trace right after it carries the PSQLException with its detail.

Auth and audit (info / debug): auth_enabled, audit_recorded (debug), provenance_recorded (debug).

Transactions and child sync (trace): tx_begin, tx_commit, tx_rollback, child_synced.

Health (info / debug): health_ok, health_degraded.

Extern lifecycle (debug): extern_handlers_registered.

This list is an excerpt of what the catalog ships; the file src/generator/_obs/log-events.ts is authoritative.

Domain trace (opt-in via --trace): invariant_evaluated, precondition_evaluated, value_computed.

Per-backend implementation

The renderers in src/generator/_obs/ keep the per-backend differences local. A call site like renderHonoLogCall("requestEnd", […]) emits the right req.log.info(…) for Hono, the right _log.LogInformation(…) for .NET, the right Logger.info(…) for Phoenix — all carrying the same catalog identity.

Consuming the stream

# Pretty-print everything (jq).
docker compose logs -f api | jq -C .

# Just failures and their messages.
docker compose logs api | jq 'select(.level == "warn" or .level == "error") | {ts, event, message}'

# Slowest 10 requests.
docker compose logs api | jq -c 'select(.event == "request_end") | {path, duration_ms, status}' | sort -t: -k2 -rn | head

# Correlate a single request across every log line.
RID=01J6...
docker compose logs api | jq "select(.request_id == \"$RID\")"

The same queries work against the .NET, Phoenix, Java, and Python deployables.

Metrics (Prometheus)

Alongside the log stream, every backend deployable — Hono, Python, .NET, Java, and Phoenix — exposes a Prometheus scrape target at GET /metrics, the standard "monitoring" surface a dashboard or alert rule consumes.

Same design as the log catalog: one platform-neutral source of truth, src/generator/_obs/metrics.ts, pins every metric's stable name, type, help, label set, and (for histograms) bucket bounds. A per-backend renderer consumes it, so a PromQL query written once works everywhere. The client library differs per platform but the exposition matches: pino's backend uses prom-client (Hono), prometheus_client (Python), prometheus-net (.NET), Actuator + Micrometer (Java), and telemetry_metrics_prometheus_core fed by the Phoenix endpoint :telemetry event (Phoenix).

What's exposed:

The two HTTP metrics are recorded at the same request seam as the request_start/request_end log lines (the request-id middleware's finally block). The route label is the matched route template (/api/carts/:id), never the raw path — labelling by raw path would explode cardinality on every id.

The two domain counters are recorded at the domain seams: an operation increment at operation_invoked / aggregate_created (the constructor counts as op="create"), and a fault increment at the app-wide error handler alongside the matching fault log line. domain_faults_total is labelled by kind only (not aggregate) — several backends handle faults in a central seam where the aggregate isn't in scope, so an aggregate label wouldn't be uniform cross-backend; per-aggregate throughput lives on domain_operations_total{aggregate,op} instead. On the four manual-increment backends (Hono / .NET / Java / Python) a record* helper does the bump; on Phoenix the counters are declared in the Telemetry supervisor and fed by [:loom, :domain, :*] :telemetry events at the seams.

# Scrape once.
curl -s localhost:3000/metrics

# Request rate over 1m (PromQL).
rate(http_requests_total[1m])

# 5xx error ratio.
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# p95 latency.
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# Domain error rate — faults over operations (business RED).
sum(rate(domain_faults_total[5m])) / sum(rate(domain_operations_total[5m]))

# Busiest operations.
topk(5, sum(rate(domain_operations_total[5m])) by (aggregate, op))

Stability: the metric catalog is a wire contract, same governance as the log catalog and wireShape — additive changes (new metrics, new optional labels) are safe; renaming a metric/label or changing a histogram's buckets breaks dashboards + recording rules and requires a consumer migration.

Metrics collector (Prometheus)

Every generated backend also exposes a Prometheus scrape target at GET /metrics (M-T7.1). The system composer wires a collector into an opt-in overlay, so one command gives a running monitoring surface with zero setup — and a plain docker compose up stays the app and its database, nothing else (M-FT.13):

docker compose -f docker-compose.yml -f docker-compose.obs.yml up

• docker-compose.obs.yml adds a prometheus service (UI on :9090) that mounts monitoring/prometheus.yml — a scrape config with one job per backend deployable, hitting its /metrics on the compose network. Pure static frontends emit no metrics and get no job.

• The overlay also merges the OTLP export env onto each backend service (see Tracing below). Both halves of the bundle switch together, which is why this is an overlay rather than a compose profiles: block — Compose can gate a whole service on a profile, but not a single environment: key, so a profile would leave every backend exporting spans into a void.

• On Kubernetes (generate system --k8s, and the Helm chart), each backend pod carries the standard scrape-discovery annotations, so a cluster Prometheus finds them automatically:

  prometheus.io/scrape: "true"
  prometheus.io/port: "3000"
  prometheus.io/path: /metrics
  

Both are additive — a system with no backend deployable emits neither the overlay nor the scrape config, and the base compose does not advertise an overlay that is not there.

/metrics is UNAUTHENTICATED, deliberately — it sits on every backend's auth-bypass list beside /health and /ready, on all five backends, whatever the deployable's auth: setting. The reason is that both halves of the scrape come out of the same generator: monitoring/prometheus.yml (and the k8s prometheus.io/* annotations) carry no credentials, so a gated /metrics would answer every scrape Loom itself configured with a 401 — a monitoring stack that cannot monitor (finding F-022). The alternative — minting a static bearer token into both the generated app and the generated scrape job — puts a long-lived shared secret in the repository and still protects nothing once the port is published. Treat /metrics as an operations surface: keep the backend port on the internal network (compose already publishes it only to localhost), or front it with an authenticating proxy. It carries counters, histograms and process gauges — never request bodies, principals or tenant data.

Tracing (OpenTelemetry)

Alongside the log stream and the Prometheus metrics, every backend deployable opens an OpenTelemetry SERVER span per request (M-T7.1) — the distributed-tracing layer, projected onto the same execution-context frame the logs and audit rows already ride.

The design principle (from execution-context.md): the execution-context backbone mints the correlationId (request id) and per-frame scopeId; tracing projects that frame onto a span. So the carrier ids ride the span as attributes, and the span's own ids ride back onto every request-scoped log line:

• On the span — the Loom ids as loom.correlation_id / loom.scope_id / loom.actor_id (namespaced clear of the OTel semantic-convention keys), plus the HTTP semantic attributes http.request.method / http.route (route template, low cardinality) / url.path / http.response.status_code, and the span name {METHOD} {route-template}.
• On the logs — the span's trace_id / span_id are stamped onto every request-scoped line via the same ambient-carrier mechanism that stamps scope_id (pino mixin, .NET BeginScope, Java MDC, the Python contextvar formatter, Elixir Logger.metadata). That's log↔trace correlation: pivot from a slow request_end line to its trace, or from a trace to every log line it produced.

Always created, conditionally exported. A span is created on every request — so trace_id is always present on the logs — but spans are exported (OTLP/HTTP) only when the standard OTEL_EXPORTER_OTLP_ENDPOINT env var is set. No endpoint, no exporter, no export attempt: a local boot without a collector stays quiet and cheap. The neutral descriptor pinning the attribute keys, env names, and log-field names is src/generator/_obs/tracing.ts — the log-catalog / metric-catalog pattern applied to tracing.

Per-backend client library (the standard OTel SDK for each platform):

Bundled collector (opt-in). The system composer wires a Jaeger all-in-one service into docker-compose.obs.yml (OTLP ingest on :4318, query UI on :16686 — the trace twin of the Prometheus UI on :9090), and the same overlay merges each backend's OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_SERVICE_NAME onto its service, so one command gives a running trace surface with zero setup.

The base docker-compose.yml sets neither var. That is deliberate and it is what makes the lean stack quiet: with no endpoint the SDK builds no exporter at all, so a plain docker compose up still stamps trace_id on every log line without a single failed export (M-FT.13, finding G7). On Kubernetes each backend carries OTEL_SERVICE_NAME plus an overridable (empty) OTEL_EXPORTER_OTLP_ENDPOINT that an operator points at their cluster collector via the chart's env: overlay — export stays off until then.

# Start the stack WITH metrics + traces, then open the bundled Jaeger UI.
docker compose -f docker-compose.yml -f docker-compose.obs.yml up
open http://localhost:16686

# Repoint at any OTLP-compatible backend (Tempo, an OTel Collector, a vendor).
OTEL_EXPORTER_OTLP_ENDPOINT=http://my-collector:4318 \
  docker compose -f docker-compose.yml -f docker-compose.obs.yml up

Stability: the span attribute keys / env names / log-field names are a wire contract — same governance as the log + metric catalogs. Additive changes are safe; renaming one breaks trace queries + log↔trace joins.

Verification

Five runtime end-to-end suites boot the generated server against a real postgres, hit /health, and assert the JSON stream carries the full lifecycle + request bracket with the catalog envelope. Opt-in via env vars (kept out of npm test because they're slow):

LOOM_OBS_E2E=1            npx vitest run test/e2e/observability-events.test.ts
LOOM_OBS_E2E_DOTNET=1     npx vitest run test/e2e/observability-events-dotnet.test.ts
LOOM_OBS_E2E_PHOENIX=1    npx vitest run test/e2e/observability-events-phoenix.test.ts
LOOM_OBS_E2E_JAVA=1       npx vitest run test/e2e/observability-events-java.test.ts
LOOM_OBS_E2E_PYTHON=1     npx vitest run test/e2e/observability-events-python.test.ts

Each suite:

1. Generates the backend project from a fixture.
2. Spins up a throwaway postgres:18-alpine sidecar.
3. Boots the server.
4. Waits for server_listening to appear on stdout.
5. Hits /health.
6. SIGTERMs the process group; waits for exit.
7. Parses the JSON stream and asserts the catalog envelope + lifecycle order.

All five obs-e2e suites additionally scrape GET /metrics and assert the Prometheus exposition carries the http_requests_total / http_request_duration_seconds series for the /health request (route-template + status labels), plus the default runtime metrics on the backends whose client library ships them (Hono/Python/.NET/Java).

They also assert tracing: request_start / request_end carry a canonical 32-char-hex trace_id (16-char span_id), shared across the request bracket — proving the SERVER span is created and its ids are threaded onto the log stream (log↔trace correlation), independent of whether a collector is configured.

.github/workflows/{hono,dotnet,phoenix,java,python}-obs-e2e.yml run their respective suites on every PR that touches the matching generator, the shared catalog, or the renderer. Each opts in via the backend-specific env var (LOOM_OBS_E2E*) and skips locally when not enabled.

Prerequisites:

• Hono: Node only — runs in pure Node, no sidecar required (the generated pg pool is lazy).
• .NET: docker (for the postgres sidecar) + the net10.0 SDK — not on the sandbox host, so build in mcr.microsoft.com/dotnet/sdk:10.0 (see tools.md, including the proxy-CA step NuGet needs).
• Phoenix: docker + mix + Erlang/OTP.
• Java: docker (for the postgres sidecar) + a Java 25 toolchain with Gradle 9.1+ — the sandbox host's JDK 21 / Gradle 8.14 cannot build the emitted project; use gradle:9-jdk25.
• Python: docker (for the postgres sidecar) + uv.

When the env var is set but a prereq is missing, the suite fails loudly with an actionable message rather than skipping silently — so a misconfigured CI surfaces as a real failure pointing at what to add to the workflow.

Extending the catalog

Add an event:

// src/generator/_obs/log-events.ts
myEvent: { event: "my_event", level: "info", fields: ["foo", "bar"] },

Then emit it at the right seam in each backend that should fire it:

// In a Hono builder
renderHonoLogCall("myEvent", [
  { name: "foo", valueExpr: "foo" },
  { name: "bar", valueExpr: "bar" },
])

// In a .NET emitter
renderDotnetLogCall("myEvent", [
  { name: "foo", valueExpr: "foo" },
  { name: "bar", valueExpr: "bar" },
])

// In a Phoenix emitter
renderPhoenixLogCall("myEvent", [
  { name: "foo", valueExpr: "foo" },
  { name: "bar", valueExpr: "bar" },
])

The Java (src/generator/java/emit/observability.ts) and Python (src/generator/python/emit/obs.ts) backends consume the same catalog through their own log-call seams; emit the new event there too for full five-backend coverage.

The renderer enforces field-set correctness against the catalog at generate time. A LOOM_OBS_E2E_*=1 run then verifies the line actually arrives on stdout with the expected envelope at runtime.

For domain-injected trace events (gated by --trace), set domain: true in the catalog entry — the renderer routes through the domain-log seam (DomainLog.Current in .NET, Logger directly in Phoenix where :telemetry carries it).

Further reading

• docs/old/proposals/observability.md — design rationale, level-as-concept analysis. (proposals/ isn't deployed to the docs site; link points at GitHub.)
• docs/traceability.md — separate concern; audited operations are append-only DB rows, distinct from the structured log channel here.