Local Development

Running the orchestrator and web service against a local Postgres so you can exercise the platform without touching staging or production. Targets contributors working on the services side of this repo — not agency engineers writing transforms (see the Guide for that).

The bin/ scripts give every git worktree its own isolated database and ports, sharing one Postgres container — so you (or a coding agent) can run several checkouts of this repo concurrently without them clobbering each other's data. This is the agent-dev-workflow pattern; see plans/agent-dev-workflow.md for the design rationale and .agents/skills/agent-dev-workflow/ (vendored) for the deep reference.

Prerequisites

bun for the TypeScript services (bin/setup also runs asdf install)
docker — bin/ manages a shared Postgres container directly (see §Auxiliary services below for what, if anything, still uses docker compose)
Optionally gcloud with application-default login if you want to exercise the GCS-backed code paths (asset version uploads, validator full-report uploads)

One-time setup

bin/setup

Idempotent — ensures the shared Postgres container, creates this worktree's database (schema-only; framework migrations don't seed data), installs dependencies for services/orchestrator, services/web, and services/web/ui, and migrates. It prints the derived config as KEY=VALUE lines on stdout (DATABASE_URL, CG_DATABASE, GRPC_PORT, PORT, VITE_PORT, ORCHESTRATOR_URL) — human status goes to stderr, so an orchestrator (Conductor and similar) can capture just the env lines.

Framework migrations are schema-only (since the migrations-replayability refactor): a fresh database has the full schema but zero environments, pipelines, or asset registrations. Client seed data lives in the client repo — e.g. sound-transit-gtfs-pipeline/seed.sql wires the st-schedule-production / st-realtime-production pipelines and PIMS assets. To exercise the pipeline flow below, apply it once:

bin/seed ../sound-transit-gtfs-pipeline/seed.sql

Run the stack

bin/dev

Starts the orchestrator (gRPC), the web service (Fastify REST, AUTH_DISABLED=1 by default), and the Control Console's Vite dev server together, on ports picked for this worktree — printed to stderr on startup. Ctrl-C stops all three.

Run interactively with no flags, bin/dev first offers the fixture menu — the named datasets under fixtures/ (see §Fixtures) plus "none (empty database)" — with this worktree's last explicitly-chosen fixture as the Enter-default, so a fresh boot lands on a populated Console instead of an empty one. The menu only appears when nothing is running: a flagless interactive bin/dev that finds a session (with or without --restart) sticks with that session's fixture instead of asking. The prompt requires stdin and stderr to be a TTY; scripts, agents, and CI are never prompted (flagless non-TTY means no fixture, exactly as before, and never touches the remembered default). Pass --fixture <name> to skip the menu, or --fixture none to demand an empty boot without being asked (none is reserved — a fixture can't take that name).

bin/dev is a per-worktree singleton. If a healthy session is already running for this worktree, a second bin/dev doesn't start anything — it prints the running session's endpoints (as KEY=VALUE on stdout) and exits 0. That's the intended flow for a second terminal or a coding agent: reuse the running stack, don't race it. A session another bin/dev is still booting (postgres ensure, migrations, fixture seed) is waited on and then attached, not torn down. Attaching refuses (exit 2) when the running session doesn't match what was asked for — a different --fixture, or explicit DATABASE_URL/GRPC_PORT/PORT/VITE_PORT overrides it wasn't started with — rather than exiting 0 with settings you didn't ask for. (One interactive exception: a flagless TTY bin/dev adopts whatever fixture the running session was started with — attaching to it, or, with --restart, rebooting with the same fixture, so a bounce never silently changes what data the stack serves — "no flag" from a human means "whatever's there", while a non-TTY caller keeps the strict refusal.) A stale or half-dead session (any of the three children gone, recycled PIDs after a reboot, a dead fixture fake-gcs container) is torn down and replaced automatically; bin/dev --restart bounces whatever is there (pass --fixture <name> or --fixture none alongside it to change datasets on the way through).

Session state lives in .dev/ (gitignored). Each service's full output is written to .dev/logs/{orchestrator,web,vite}.log — fresh per session, kept after exit — so anyone (including an agent that didn't start the session) can inspect or follow the stack's output without owning its terminal. If any child dies, the whole stack tears down loudly and names the dead service, instead of leaving the survivors serving 14 UNAVAILABLE from still-bound ports.

Logs are size-capped for long-lived sessions: past ~20 MB per service (override: CG_DEV_LOG_CAP_BYTES) the most recent half is kept in <service>.log.old and the live file starts over — so a session left running for weeks tops out around 30 MB per service instead of growing forever.

bin/dev status   # health-checked KEY=VALUE contract; exit 0 running / 1 stopped / 2 unhealthy / 3 starting
bin/dev stop     # stop the recorded session, from any terminal
bin/dev logs     # follow all three logs (or: bin/dev logs orchestrator|web|vite)

AUTH_DISABLED=1 is the zero-setup, IdP-less local dev mode — the web service refuses to boot without either it or a full OIDC configuration (auth is never optional; see specs/authentication.md §Local Development Mode in the repo — the specs tree isn't part of this docs site). In this mode every request runs as a synthetic local admin, the server binds 127.0.0.1 only, and the Console SPA renders signed-in with zero clicks. No API key, session, or CSRF header is needed on any request. bin/dev never sets HOST — that loopback binding is the mode's containment guarantee, and every worktree instance is already loopback + a distinct port. To exercise the real OIDC flow locally instead, set the OIDC_* vars against a dev-tenant IdP before running bin/dev (specs/configuration.md §Web Service Environment Variables) — a shared, opt-in local IdP for this is tracked as a follow-up to plans/agent-dev-workflow.md, not yet built.

Running two worktrees' bin/dev at once (e.g. two coding-agent sessions) lands each on its own database and ports automatically — nothing to configure.

Port band

One shared Postgres container plus three per-worktree ranges (see bin/_common.sh for the authoritative claim):

Service	Default	Worktree range
Postgres (shared)	`40000`	n/a — one container for every worktree
Orchestrator gRPC (`GRPC_PORT`)	`40001`	`40001`–`40099`
Web (`PORT`)	`40101`	`40101`–`40199`
Vite / Console UI (`VITE_PORT`)	`40201`	`40201`–`40299`

Every value has an env override (GRPC_PORT, PORT, VITE_PORT, CG_DATABASE, CG_PG_PORT) if you want explicit control instead of the derived pick.

Other bin/ commands

Command	Does
`bin/setup`	Ensure Postgres + this worktree's DB + deps + migrations (idempotent)
`bin/dev`	Run orchestrator + web + Vite together (singleton per worktree — reuses a healthy running session); flagless on a TTY offers the fixture menu
`bin/dev --fixture <name>`	Boot with a named dataset from `fixtures/` — see §Fixtures (`--fixture none` = empty DB, no prompt)
`bin/dev status`	Health-check the recorded session; `KEY=VALUE` contract on stdout (exit 0 running / 1 stopped / 2 unhealthy / 3 starting)
`bin/dev stop`	Stop the recorded session from any terminal
`bin/dev logs [svc]`	Follow the session logs (`.dev/logs/{orchestrator,web,vite}.log`)
`bin/test`	Run the orchestrator test suite against the dedicated `continuous_gtfs_test` database (never dev data)
`bin/db [SQL]`	Run SQL against this worktree's DB, or open an interactive `psql` shell with no args
`bin/reset-db`	Drop + recreate + migrate this worktree's DB
`bin/seed <file.sql>`	Apply a client seed file (e.g. the ST repo's `seed.sql`) into this worktree's DB
`bin/cleanup`	Drop this worktree's DB (refuses the canonical dev DB without `--force`); stop a running `bin/dev` session
`bin/gc`	Sweep merged agent worktrees — see §Sweeping merged worktrees

services/web and services/web/ui have no DB-backed tests — run those directly (cd services/web && bun test, cd services/web/ui && bun test); bin/test only wraps the orchestrator, the one suite that touches Postgres.

Auxiliary services

docker-compose.yml at the repo root now covers only shared auxiliary services — started once per machine, not per worktree — currently just the MobilityData GTFS validator API:

docker compose up -d
export VALIDATOR_URL=http://localhost:8081

Every worktree's bin/dev can point at this one shared instance (VALIDATOR_URL isn't wired into bin/dev automatically — export it before running bin/dev, or add it to your shell profile). Postgres is not in compose — bin/_common.sh owns the shared Postgres container directly, because compose ties operations to the directory it was first run from, which breaks the moment an agent worktree using it gets deleted (see .agents/skills/agent-dev-workflow/references/gotchas.md).

Useful env vars

Var	Default	When to set
`DATABASE_URL`	—	`bin/` emits this; set it yourself to point a service at a non-`bin/`-managed Postgres. Wins over the discrete `DB_*` vars below when set.
`DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USER`, `DB_PASSWORD`	`localhost` / `40000` / `continuous_gtfs` / `continuous_gtfs` / `localdev`	ignored once `DATABASE_URL` is set; Cloud Run keeps using these directly
`GRPC_PORT`	`40001` (via `bin/dev`)	port collision
`VALIDATOR_URL`	unset	`http://localhost:8081` after starting the aux compose stack — wires the validator integration
`ANALYSIS_BUCKET`	`continuous-gtfs-analysis`	only if you want full validator reports uploaded somewhere you own (otherwise the upload errors are caught and logged; summaries still land in the run row)
`AUTH_DISABLED` (web only)	`1` (via `bin/dev`)	unset it (and set the `OIDC_*` vars) to exercise real OIDC login instead
`CONTINUOUS_GTFS_API_KEY` (web only)	unset	enforced-mode only: the transitional shared `X-API-Key` the authz gateway accepts as an operator-equivalent service principal until its retirement; irrelevant under `AUTH_DISABLED`
`PORT` (web only)	`40101` (via `bin/dev`)	port collision
`ORCHESTRATOR_URL` (web only)	`localhost:<GRPC_PORT>` (via `bin/dev`)	only if running the orchestrator elsewhere

Fixtures

A fixture is a named, self-contained dataset under fixtures/<name>/: an idempotent seed.sql plus, when the scenario needs feed bytes, a committed gcs/ object tree served by a per-worktree fake-gcs emulator — so everything renders fully offline, with no GCP credentials and no real bucket ever touched. Fixture mode migrates and seeds before the services boot, and each fixture's README documents its known contract (exact rows, counts, and diffs) so tests and screenshots can assert against it.

bin/dev --fixture demo

Fixture	Populates
`demo`	Everything schedule-side, coherently: environments, pipelines, asset versions with real bytes (Compare works), run history with step detail, published outputs, and admin users/service accounts. The "boot a full Console to click around" dataset.
`compare-schedule`	Schedule Compare only — one asset, two versions with a precisely documented 3-file diff.
`schedule-runs`	Runs pages only — three completed runs with per-step detail, DB rows only (no bytes, no emulator).

The focused fixtures exist because they're stable contracts — a test asserting "exactly 3 files modified" must not break when someone enriches the demo dataset. Load one fixture per session; switching (bin/dev --restart --fixture <other>) reseeds on top of the existing database, so run bin/reset-db first when moving between fixtures to avoid leftover rows. Authoring conventions, seeding recipes, and the synthetic-feed generator pattern live in the platform-fixtures skill (.claude/skills/platform-fixtures/).

Preview features (speculative UI)

The Console dev build can render speculative surfaces — pages, panels, or fields whose backend doesn't exist yet — with fake data served at the network boundary by MSW and an unmissable "Preview" marker separating them from real features. Each feature is an entry in services/web/ui/src/preview/registry.ts with its own plan, toggled per-feature from the dev-only Previews menu in the Console header (state persists in localStorage, per origin — worktrees don't share toggles). A page-scale preview lives at its real intended URL via the src/routes/(preview)/ route group; toggled off it shows a placeholder, not a 404.

Preview features are dev-mode only by decision: production builds exclude preview routes from route-tree generation, stub .preview. modules to null components, and ship no MSW. bin/dev is the demo surface. Unmatched requests always pass through to the real local backend, so previews coexist with fixture data. Conventions, the feature lifecycle, and the graduation ritual live in the speculative-ui skill (.claude/skills/speculative-ui/).

Exercise the schedule pipeline

Requires seed data — the asset keys and pipeline ids below come from the client seed (sound-transit-gtfs-pipeline/seed.sql, see §One-time setup); on an unseeded schema-only database these curls 404.

The PIMS fetcher fires immediately on startup and will 401 against the real PIMS endpoints — that's expected noise. You don't need PIMS credentials to drive a pipeline run locally. Push your own bytes instead.

Any real GTFS schedule zip works; the Sound Transit agency repo's data/pims-gtfs.zip is a convenient fixture. The examples below use $PORT — the value bin/setup/bin/dev printed for the web service (40101 in the main checkout, something else in a worktree).

# Push a fresh asset version → orchestrator creates a run and (if VALIDATOR_URL
# is set) enqueues input validation. No worker is needed for this part — the
# dispatch just gets queued for later.
curl -X POST \
  -H "Content-Type: application/zip" \
  --data-binary @path/to/schedule.zip \
  "http://localhost:${PORT}/api/v1/assets/pims%2Fproduction%2Fschedule/versions"

# Find the run id (most-recent first)
curl -s "http://localhost:${PORT}/api/v1/runs?pipeline_id=st-schedule-production&limit=1" | jq

# Single-run detail — includes result.validatorReports.{input,output} when
# the validator integration is wired
curl -s "http://localhost:${PORT}/api/v1/runs/<run_id>" | jq

# Manual trigger (re-dispatch the pipeline against current inputs without
# pushing new bytes)
curl -X POST -H "Content-Type: application/json" \
  -d '{"reason":"my-test"}' \
  "http://localhost:${PORT}/api/v1/pipelines/st-schedule-production/trigger"

Output validation (the validatorReports.output half) requires a running worker that consumes the dispatch and streams an artifact back. Beyond the scope of this guide; see pipeline/ for the worker.

Expected local noise vs real bugs

The following messages are normal — don't chase them:

Fetch pims/...: HTTP 401 Unauthorized — PIMS rejects unauthenticated requests; the orchestrator logs and moves on. Set the pims-prod / pims-qa secrets if you have them, or ignore.
Probe pims/...: probe HTTP 401 Unauthorized — falling through to full fetch — same root cause, different code path.
Asset cache hydrated: 0 loaded from GCS (on a fresh DB) — no asset versions registered yet. Resolves once you push one.
Queued dispatch ... — no schedule worker; will retry on next registration — no worker is running. The run exists in the DB, validator hooks still fire for the input.
Cannot sign data without 'client_email' from the /validator-reports/:kind endpoint — V4 signed URL generation needs service-account credentials, which gcloud auth application-default login user-mode creds don't provide. Works in Cloud Run where the orchestrator runs as a proper SA with the token-creator self-binding. Test signed URLs in staging instead.

These are real bugs:

Anything tagged error in the orchestrator log that isn't one of the above
PostgresError of any kind
500 Internal Server Error from the web service for routes that aren't /validator-reports/:kind

Teardown

# Stop the stack (Ctrl-C in bin/dev's terminal), or from another terminal:
bin/dev stop

# Stop the stack AND drop this worktree's database:
bin/cleanup

# Wipe just this worktree's database (keeps the shared Postgres container up
# for other worktrees):
bin/cleanup   # then bin/setup again for a fresh one

# Stop the shared aux services (validator) — only if nothing else needs them:
docker compose down

Sweeping merged worktrees

Agent worktrees are alive while their PR is in review and become garbage the moment it merges — but merges happen on the reviewer's schedule, with no local session listening. bin/gc closes that gap with a lazy, state-based sweep:

bin/gc            # sweep: remove every worktree PROVEN merged (applies by default)
bin/gc --dry-run  # print the same verdicts, mutate nothing

It enumerates this repo's linked worktrees under any */.claude/worktrees/ path and removes only the ones it can prove disposable: a clean, unlocked worktree (never the main checkout or the one you're running from) whose HEAD is contained in origin/develop — by ancestry, by patch equivalence (git cherry, which catches PRs rebased remotely before merging), or by a merged PR on GitHub (gh, best-effort, which catches squash merges). For each proven worktree it stops any recorded bin/dev session, drops the derived per-worktree database, removes the worktree, deletes the local branch, and prunes stale remote-tracking refs. Everything unproven is skipped with a one-line reason.

A SessionStart hook runs bin/gc --nudge (no network, no mutation) and prints a one-line reminder when merged husks have accumulated; it stays silent when there's nothing to sweep.

Manual flow (no `bin/`)

For the rare case you want to run a service by hand outside the bin/ scripts — e.g. debugging a bin/dev issue itself — point at the shared Postgres container's default port and pass ports/env explicitly:

# Orchestrator
cd services/orchestrator
DATABASE_URL="postgres://continuous_gtfs:localdev@localhost:40000/continuous_gtfs" \
  GRPC_PORT=40001 bun run index.ts

# Web
cd services/web
AUTH_DISABLED=1 PORT=40101 ORCHESTRATOR_URL=localhost:40001 bun run index.ts

# Console UI
cd services/web/ui
VITE_PORT=40201 VITE_API_TARGET=http://localhost:40101 bun run dev

This talks to the main worktree's database (continuous_gtfs) on the shared container's default port (40000) — bin/setup must have run at least once to create it. There's no manual-flow equivalent for worktree isolation; use bin/dev for that.