Skip to content

Local Development

Running the orchestrator and web service against a local Postgres so you can exercise the platform without touching staging or production. Targets contributors working on the services side of this repo — not agency engineers writing transforms (see the Guide for that).

Prerequisites

  • bun for the TypeScript services
  • docker compose for Postgres + optional dependencies
  • Optionally gcloud with application-default login if you want to exercise the GCS-backed code paths (asset version uploads, validator full-report uploads)

Start the dependencies

The compose file owns external dependencies — Postgres always, validator opt-in:

# Postgres only (default — what most local work needs)
docker compose up -d

# Postgres + MobilityData GTFS validator (for validator-integration work)
docker compose --profile validator up -d

Postgres auto-applies every migration in services/orchestrator/migrations/ on first boot, so a fresh volume comes up with the schema, the PIMS asset definitions, and the st-schedule-production / st-realtime-production pipelines already wired (see migrations 002 and 003).

Start the services

Two TypeScript services run on the host, not in compose, so you get HMR and a real debugger.

# Terminal 1 — orchestrator (gRPC on :50051)
cd services/orchestrator
bun install                # one-time
bun run index.ts

# Terminal 2 — web (Fastify REST on :3000)
cd services/web
bun install                # one-time
PORT=3000 ORCHESTRATOR_URL=localhost:50051 bun run index.ts

Both services pick sensible defaults for local dev — Postgres connection defaults match docker-compose.yml (continuous_gtfs / continuous_gtfs / localdev), Cloud Run features (leader-check, Secret Manager) are skipped when K_REVISION / GCP_PROJECT are unset, and the validator integration is opt-in via VALIDATOR_URL.

Useful env vars

Var Default When to set
DB_HOST, DB_PORT, DB_NAME, DB_USER, DB_PASSWORD match docker-compose.yml only if you point at a non-compose DB
GRPC_PORT 50051 port collision
VALIDATOR_URL unset http://localhost:8081 after starting the validator profile — wires the validator integration
ANALYSIS_BUCKET continuous-gtfs-analysis only if you want full validator reports uploaded somewhere you own (otherwise the upload errors are caught and logged; summaries still land in the run row)
CONTINUOUS_GTFS_API_KEY unset set to require X-API-Key on the POST endpoints; leave unset for unauth local dev
PORT (web only) 8080 set to 3000 so you don't collide with the orchestrator's 8080 Cloud Run convention
ORCHESTRATOR_URL (web only) localhost:50051 only if running the orchestrator elsewhere

Exercise the schedule pipeline

The PIMS fetcher fires immediately on startup and will 401 against the real PIMS endpoints — that's expected noise. You don't need PIMS credentials to drive a pipeline run locally. Push your own bytes instead.

Any real GTFS schedule zip works; the Sound Transit agency repo's data/pims-gtfs.zip is a convenient fixture.

# Push a fresh asset version → orchestrator creates a run and (if VALIDATOR_URL
# is set) enqueues input validation. No worker is needed for this part — the
# dispatch just gets queued for later.
curl -X POST \
  -H "Content-Type: application/zip" \
  --data-binary @path/to/schedule.zip \
  "http://localhost:3000/api/v1/assets/pims%2Fproduction%2Fschedule/versions"

# Find the run id (most-recent first)
curl -s "http://localhost:3000/api/v1/runs?pipeline_id=st-schedule-production&limit=1" | jq

# Single-run detail — includes result.validatorReports.{input,output} when
# the validator integration is wired
curl -s "http://localhost:3000/api/v1/runs/<run_id>" | jq

# Manual trigger (re-dispatch the pipeline against current inputs without
# pushing new bytes)
curl -X POST -H "Content-Type: application/json" \
  -d '{"reason":"my-test"}' \
  "http://localhost:3000/api/v1/pipelines/st-schedule-production/trigger"

Output validation (the validatorReports.output half) requires a running worker that consumes the dispatch and streams an artifact back. Beyond the scope of this guide; see pipeline/ for the worker.

Expected local noise vs real bugs

The following messages are normal — don't chase them:

  • Fetch pims/...: HTTP 401 Unauthorized — PIMS rejects unauthenticated requests; the orchestrator logs and moves on. Set the pims-prod / pims-qa secrets if you have them, or ignore.
  • Probe pims/...: probe HTTP 401 Unauthorized — falling through to full fetch — same root cause, different code path.
  • Asset cache hydrated: 0 loaded from GCS (on a fresh DB) — no asset versions registered yet. Resolves once you push one.
  • Queued dispatch ... — no schedule worker; will retry on next registration — no worker is running. The run exists in the DB, validator hooks still fire for the input.
  • Cannot sign data without 'client_email' from the /validator-reports/:kind endpoint — V4 signed URL generation needs service-account credentials, which gcloud auth application-default login user-mode creds don't provide. Works in Cloud Run where the orchestrator runs as a proper SA with the token-creator self-binding. Test signed URLs in staging instead.

These are real bugs:

  • Anything tagged error in the orchestrator log that isn't one of the above
  • PostgresError of any kind
  • 500 Internal Server Error from the web service for routes that aren't /validator-reports/:kind

Teardown

# Stop the orchestrator + web (Ctrl-C in their terminals)
# Stop the dependencies
docker compose --profile validator down

# Wipe the local DB if you want a fresh seed on next boot
docker compose down -v