Initial Environment Setup: staging → production

This runbook sets up the two-environment topology — a mutable staging environment feeding a promote-only production environment — on a Continuous GTFS deployment. It covers both cases:

• Existing single-env deployment (pipelines already live in production): the bootstrap creates a staging environment carrying production's pipelines and slot bindings, then flips production to promote-only, with no consumer-visible gap.
• Fresh deployment: create staging with its pipelines and bindings, then create production as a promotion environment sourced from staging. Production is born with no bindings of its own, so nothing needs de-slotting.

The procedure was first executed against the Sound Transit deployment on 2026-06-11; the worked examples below (run IDs, tags) come from that live run. That run created the environment and copied its bindings with direct SQL, which the Control Console has since replaced — this page describes the current path and keeps the parts of the original that are still the only way.

Environment CRUD is self-service

Creating, editing, and deleting environments on a running deployment is an admin operation in the Control Console — the Environments admin page, or POST/PATCH/DELETE /api/v1/envs (environments.md §Environment Administration (CRUD)). The create form registers the environment's pipelines and binds their input slots in the same operation, and slot bindings are edited afterwards on the Rules page (asset-registry.md §Pipeline Slots). This runbook is the one-time staging→production topology bootstrap/cutover — standing up staging as a copy, removing production schedule's bindings, flipping production to promote-only, and verifying the first promotion — which is more than creating an environment. The only step that still needs direct SQL is removing production schedule's bindings on an existing deployment (see step 3); a fresh deployment needs no SQL at all.

Background reading: environments.md §Environment Mutability defines the promote-only model; pipeline-execution.md §Promotion Runs defines what promotion copies per feed type.

Prerequisites

All of these must be true before you start:

1. Framework v0.12.3 or later deployed (orchestrator + web). Four issues hit during the original cutover were fixed in v0.12.1–v0.12.3 (see Known gotchas); deployments at ≥ v0.12.3 get all of them for free. Most load-bearing: the 64 MiB gRPC message limits — schedule dispatches over 4 MiB fail outright on older versions. The create form's bindings section and the Rules page's slot set additionally need a release in which workers report their INPUTS manifest at registration (declared-input slots); on an older release the environment is still created the same way, but bindings are set on the Rules page afterwards.
2. Database migrations applied through 018:
   • 016-environments-promotes-from.sql — the promotes_from mutability primitive
   • 017-pipeline-runs-kind.sql — the kind = 'promotion' run discriminator
   • 018-pipelines-image-tag.sql — per-pipeline image_tag
3. Maintenance-window DB access tooling — existing deployments only. Removing production schedule's bindings has no API path and is applied directly against Cloud SQL. The sysadmin skill's maintenance-window script (temporarily enable public IP, apply, remove) is the sanctioned path; the same window used for migrations 016–018 can carry it. A fresh deployment skips this.
4. A worker pool running and registered — GET /api/v1/workers shows at least one connected worker. Worker registration is also what tells the platform which pipelines and inputs the image declares: the create form's prefilled pipeline rows and bindings section, and the Rules page's slot set, are empty until a worker on the image has registered. If the deployment just cut over to orchestrator-managed pools, populate pipelines.image_tag first (see Populating image_tag): after the cutover removes the static tofu-managed pool, no pools exist until image_tag is populated.
5. Console credentials: a signed-in admin for the Environments admin page or POST /api/v1/envs (environment CRUD is admin-only, and service accounts are never admin, so this step is done by a person), and the Control Console API key (CONTINUOUS_GTFS_API_KEY) for the trigger and promote calls.

Pre-checks

Before changing anything:

API=https://api.<tenant>.continuousgtfs.io
FEEDS=https://feeds.<tenant>.continuousgtfs.io

# Current consumers are healthy (note last-modified for later comparison)
curl -sI $FEEDS/production/schedule.zip | grep -i 'HTTP\|last-modified'
curl -sI $FEEDS/production/trip_updates.pb | grep -i 'HTTP\|last-modified'

# Workers connected, advertising the expected image tag
curl -s $API/api/v1/workers | jq

# Snapshot current production output asset versions (rollback reference)
curl -s "$API/api/v1/runs?environment_id=production&limit=5" | jq '.runs[] | {id, status, completed_at}'

A stale feed here is a finding, not a blocker to investigate later

During the Sound Transit bootstrap, this pre-check surfaced that /production/schedule.zip had been stale since ~June 1 — schedule dispatches had been silently dying on gRPC's 4 MiB default message limit (the feed had grown to 4.66 MB). Root-cause anything stale before proceeding; the bootstrap's first-promotion verification depends on a working staging schedule run.

Populating image_tag (and what the reconciler does)

The orchestrator's pool reconciler derives the desired worker-pool set from the distinct non-NULL image_tag values across enabled pipelines. Setting the column is what makes pools exist. An environment created in ref-tracking mode resolves its own image_tag from the tracked ref's latest ready build; this manual write is for a pre-existing, manually pinned production whose column is still empty:

UPDATE pipelines SET image_tag = '<current pipeline image tag>'
WHERE environment_id = 'production';

Within one reconcile interval (default 60s) the orchestrator creates a Cloud Run worker pool named continuous-gtfs-pipeline-{first 8 chars of the sanitized tag} — dots and other characters invalid in Cloud Run resource IDs are mapped to dashes (tag v0.4.3 → pool continuous-gtfs-pipeline-v0-4-3). Pool↔tag identity is carried by the pool's container image reference, not the name, so sanitization collisions are harmless.

Confirm before proceeding:

• Orchestrator logs show Pool reconciler started and a pool-create action
• The pool's worker registers: GET /api/v1/workers lists a worker advertising the tag
• Allow ~30–60s for pool provisioning on top of the reconcile interval

The bootstrap

Steps 1–2 are done in the Control Console. Steps 3–4 apply only to an existing deployment whose production already runs pipelines directly; run them together in a single maintenance window and a single transaction. Pipeline IDs here are schedule and realtime; adjust if your agency repo names them differently.

1. Create the staging environment

On the Environments admin page (or with POST /api/v1/envs), create the environment:

• id staging, display_name "Staging".
• Image-source mode: leave promotes_from empty — that is what makes staging mutable (direct triggers, config edits, and pushes are accepted; staging is where all changes land first). Prefer ref-tracking (tracks_ref = the ref your pipeline image builds from, e.g. develop) so staging picks up new images as CI publishes them; use manual pin only if the deployment has no build reporting.
• Pipelines: register schedule and realtime (id, display_name, feed_type). In ref-tracking mode with a ready build for the ref, the form resolves the image and offers to prefill these rows from the pipelines the image declares.
• Slot bindings: each pipeline row carries a bindings section listing every input the image declares, required inputs marked as needing an asset and optional ones as empty, with the asset picker limited to assets of the declared content_kind. Bind each slot to the same asset production uses (production's current bindings are visible on the Rules page, or as the slots of its entries in GET /api/v1/pipelines), keeping the same debounce_ms. On a fresh deployment, bind the assets you registered for the agency's upstream feeds.

The equivalent API request:

POST /api/v1/envs
{
  "id": "staging",
  "display_name": "Staging",
  "tracks_ref": "develop",
  "pipelines": [
    {
      "id": "schedule", "display_name": "Schedule", "feed_type": "schedule",
      "slots": [
        {"slot_name": "<schedule input>", "asset_key": "<asset key>", "debounce_ms": 60000}
      ]
    },
    {
      "id": "realtime", "display_name": "Realtime", "feed_type": "realtime",
      "slots": [
        {"slot_name": "<rt input>", "asset_key": "<asset key>", "debounce_ms": 2000}
      ]
    }
  ]
}

The whole create is atomic — a rejected binding (an input the image doesn't declare, an asset of the wrong kind, an unknown asset) creates nothing and the 400 names the slot. Required inputs may be left unbound at create; they show as configuration errors on the Rules page, and the pipeline's runs fail, until bound. If the image's pipelines aren't offered (manual pin, no ready build for the ref, or no worker on the image has registered yet), type the pipeline rows by hand and bind on the Rules page after saving — see step 2.

With bindings in place, the orchestrator's fetcher-driven dispatch starts running staging pipelines automatically — staging auto-publishes to /staging/... CDN paths from its first run.

Ops controls are not part of the create

The create carries pipelines and bindings, not ops controls. If production has a non-default passthrough_on_failure or a disabled_steps list that staging should mirror, set them with PATCH /api/v1/pipelines/staging/{id} (or on the Rules page) after the create.

2. Verify staging's slot set on the Rules page

After saving, the console lands on the Rules page. Check both staging pipelines:

• Every slot reads bound. An unbound required input or a kind mismatch renders with the error treatment — these are configuration errors, and the worker refuses to run the pipeline until they are fixed. An unbound optional input is an empty slot with an informational label, not an error.
• Bind anything missing via the pipeline's Edit affordance — one Save issues one atomic replacement and, when a binding was added or changed, starts one run against current inputs.
• If the page notes that the slot set will be known once a worker on this image registers, the manifest hasn't been reported yet: wait for the worker (see prerequisite 4), then recheck.

3. De-slot production's schedule pipeline (SQL)

Existing deployments only. Production's schedule pipeline — and only schedule — must lose its bindings. Schedule promotion copies bytes (the validated output bundle), so production's schedule output must come exclusively from promotion; leaving slots bound would let fetcher-driven dispatch keep auto-publishing to /production/schedule.zip, racing promotions. The promote-only API gate doesn't cover this: automatic dispatch is not a direct API mutation. Removing the bindings is what actually stops direct production schedule output.

Production realtime keeps its bindings and keeps running: realtime promotion copies configuration, not bytes — the production RT pipeline must keep executing continuously to produce feeds at all.

This step has no API or console path, and is the reason the maintenance window exists: the binding editor cannot unbind a required input, and once production is promote-only (step 4) it rejects binding edits altogether.

BEGIN;

-- 3. De-slot production's SCHEDULE pipeline — and only schedule.
--    See the rationale above: schedule promotion copies bytes, so
--    production's schedule output must come exclusively from promotion.
--    Production REALTIME keeps its slots and keeps running.
DELETE FROM pipeline_slots
WHERE environment_id = 'production' AND pipeline_id = 'schedule';

4. Flip production to promote-only

In the same transaction:

-- 4. Flip production to promote-only. From this statement on, every
--    direct-mutation API path against production (trigger, pipeline
--    config edit, slot edit, output-asset push) rejects with HTTP 409;
--    production state changes only via promotion from staging.
UPDATE environments
SET promotes_from = 'staging'
WHERE id = 'production';

COMMIT;

The flip itself is also an admin edit — promotes_from on the Environments admin page or PATCH /api/v1/envs/production (environments.md §Edit); tracks_ref must be clear, since the two are mutually exclusive. Keeping it in the transaction with step 3 leaves no window in which production has neither bindings nor a promotion source.

After commit, until the first schedule promotion lands, /production/schedule.zip keeps serving the last auto-published bytes — content is continuous for consumers, there's just no fresh write path yet. Proceed to the first promotion promptly. On the Rules page, production's schedule pipeline now reads as promotion-fed rather than as a pipeline with missing bindings — promote-only pipelines are presented as intentional, not broken.

Fresh deployment instead of steps 3–4: create production on the Environments admin page in promotion mode (promotes_from = staging), registering the same schedule and realtime pipeline rows. The form offers no bindings section in this mode and the API rejects any slots — production's bindings arrive by promotion. Production realtime's bindings and image_tag are copied from staging by the first realtime promotion (step 3 below); production schedule never has bindings.

Topology decision table

What each environment keeps after the bootstrap, and why:

	pipelines row	slot bindings (pipeline_slots)	Runs directly?	Why
staging / schedule	yes	yes	yes — fetcher-driven + manual triggers	The validation surface. Output auto-publishes to /staging/schedule.zip
staging / realtime	yes	yes	yes — continuous ticks	The validation surface. Output auto-publishes to /staging/*.pb
production / schedule	yes (kept)	no (deleted)	no — promotion-fed only	The row is the promotion target (promotion runs are recorded against it). No bindings → no fetcher-driven dispatch → /production/schedule.zip changes only via promotion
production / realtime	yes (kept)	yes (kept)	yes — keeps running continuously	RT promotion copies config, not bytes. The production RT pipeline must keep executing to produce feeds; automatic dispatch is not a direct mutation, so the promote-only gate correctly doesn't block it

Staged code rollout (how a new version reaches staging)

On a ref-tracking staging environment this is automatic: when CI reports a new ready build for the tracked ref, staging advances to that image. For a manually pinned staging, pointing at a new pipeline image version is still a one-column write with no API path:

UPDATE pipelines SET image_tag = '<new tag>' WHERE environment_id = 'staging';

Either way, the reconciler notices the new distinct tag and creates a second pool alongside the existing one — environments pinned to different tags get distinct pools; environments converging on one tag share a single pool. Staging runs dispatch to the worker advertising the new tag; production runs keep dispatching to the old one until a realtime promotion copies the tag forward.

If the new image declares an input the previous one didn't, the new slot appears on the Rules page once the new pool's worker registers — unbound, and flagged as a configuration error if the input is required. Bind it there before expecting the pipeline to run. A binding for an input the new image no longer declares drops out of view but is retained; it comes back unchanged if a later image declares the name again.

Worked example from the Sound Transit bootstrap: staging image_tag was set to v0.4.3 while production sat on v0.4.0. The reconciler created continuous-gtfs-pipeline-v0-4-3 alongside continuous-gtfs-pipeline-v0-4-0, and staging schedule run 6955f9b8-0409-43eb-9339-bb1d408c784c completed on the new worker. After the realtime promotion copied v0.4.3 to production, v0.4.0 became unreferenced and its pool entered the lame-duck window (scaled to zero, deleted after the TTL).

First run and first promotions

All mutations require the API key. Triggers default to the production environment when environment_id is omitted — always pass it explicitly now that production is promote-only.

1. Trigger a staging schedule run

curl -s -X POST $API/api/v1/pipelines/schedule/trigger \
  -H "X-API-Key: $CONTINUOUS_GTFS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"environment_id": "staging", "reason": "env-bootstrap first staging run"}'

Wait for completion and capture the run id:

curl -s "$API/api/v1/runs?environment_id=staging&pipeline_id=schedule&limit=1" \
  | jq '.runs[0] | {id, status, completed_at}'

Confirm /staging/schedule.zip now serves the run's output (fresh last-modified). If the run failed before any step ran, naming an input, a required slot is unbound or bound to the wrong kind — fix it on the Rules page (step 2 above) and re-trigger.

2. Promote the schedule run to production

Schedule promotion adopts an explicit checkpoint from staging's timeline — you promote a specific validated state, not "latest". Read the staging head to get its checkpoint id (or pass the run id from step 1 as the legacy source_run_id alias — it resolves through that run's checkpoint):

curl -s $API/api/v1/pipelines/staging/schedule/timeline/head \
  -H "X-API-Key: $CONTINUOUS_GTFS_API_KEY"          # → .checkpoint.id

curl -s -X POST $API/api/v1/envs/production/pipelines/schedule/promote-schedule \
  -H "X-API-Key: $CONTINUOUS_GTFS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"checkpoint_id": "<staging head checkpoint id>"}'

Returns 200 with the created promoted event id and the adopted checkpoint id — adoption is synchronous, so the response is the completed fact. A 409 no_completed_evaluation means the checkpoint has no derived output yet (run it in staging first); a 409 no_active_service means the feed's calendars have all ended (the envelope names the last active service date) — this refusal has no override. Verify:

• Production's timeline head is the adoption: GET /api/v1/pipelines/production/schedule/timeline/head shows event.kind: promoted pointing at the same checkpoint id you named
• /production/schedule.zip last-modified advances; content hash matches the staging output it sourced

3. Promote realtime configuration to production

Realtime promotion takes no body — the target always adopts the source's current checkpoint, applying its stored config content (ops controls, image_tag, slot bindings) onto production's pipeline in one transaction:

curl -s -X POST $API/api/v1/envs/production/pipelines/realtime/promote-realtime \
  -H "X-API-Key: $CONTINUOUS_GTFS_API_KEY"

Returns 200 with the promoted event id + adopted checkpoint id. Verify production's realtime timeline head is the adoption (GET /api/v1/pipelines/production/realtime/timeline/head — its checkpoint.config is exactly what was applied), and that subsequent production RT runs complete on the promoted config (check GET /api/v1/runs?environment_id=production&pipeline_id=realtime — fresh runs completing).

4. Verify the promote-only gate

A direct trigger against production must now reject:

curl -s -X POST $API/api/v1/pipelines/schedule/trigger \
  -H "X-API-Key: $CONTINUOUS_GTFS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"environment_id": "production"}'

Expected — HTTP 409 with:

{"error": "env_promote_only", "env": "production", "promotes_from": "staging", "attempted": "trigger_pipeline"}

5. Verify the full topology

curl -s $API/api/v1/envs | jq                  # both envs listed
curl -s $API/api/v1/envs/production | jq       # promotes_from: "staging"

curl -sI $FEEDS/staging/schedule.zip | grep -i 'HTTP\|last-modified'
curl -sI $FEEDS/production/schedule.zip | grep -i 'HTTP\|last-modified'
curl -sI $FEEDS/production/trip_updates.pb | grep -i 'HTTP\|last-modified'   # RT fresh
curl -sI $FEEDS/schedule.zip | grep -i 'HTTP\|last-modified'                 # root-URL rewrite → /production/ intact

Rollback

If the bootstrap goes sideways (especially after step 4 but before the first schedule promotion, when production has no fresh write path):

BEGIN;

-- Production accepts direct mutations again; fetcher-driven dispatch
-- resumes wherever slots are bound.
UPDATE environments SET promotes_from = NULL WHERE id = 'production';

-- Re-seed production schedule's bindings from the staging copies.
INSERT INTO pipeline_slots (environment_id, pipeline_id, slot_name,
                            asset_key, debounce_ms)
SELECT 'production', pipeline_id, slot_name, asset_key, debounce_ms
FROM pipeline_slots
WHERE environment_id = 'staging' AND pipeline_id = 'schedule';

COMMIT;

This restores the pre-bootstrap behavior exactly: production schedule auto-publishes on the next upstream fetch. Once promotes_from is cleared, production is mutable again, so the bindings could equally be restored on the Rules page; the SQL keeps the rollback to one transaction. The staging env and its rows can be left in place (harmless — it just keeps publishing to /staging/...) or cleaned up separately. Historical production runs and output asset versions were never touched by the bootstrap, so nothing else needs restoring.

Known gotchas (from the first live run)

The 2026-06-11 Sound Transit bootstrap hit five live issues. All five are fixed in framework v0.12.1–v0.12.3 — a fresh deployment at ≥ v0.12.3 will not hit them — but they're recorded here because the symptoms are non-obvious:

1. Deploy SA missing roles/iam.roleAdmin — the cutover tofu apply (which creates the orchestrator's custom pool-management role) failed until the role was granted manually. Now part of documented deploy-SA requirements.
2. Orchestrator SA missing artifactregistry.reader — pool creation succeeded but pool instances couldn't pull the pipeline image. Manually granted during the run; the grant has since moved into the platform module as a project-level member.
3. Pool IDs reject dotted tags — Cloud Run worker_pool_id forbids dots, so release-style tags (v0.4.3) crashed pool creation. Fixed by name sanitization (dots → dashes).
4. Label values reject dotted tags — same character restriction on GCP label values broke the image-tag label. Fixed by deriving pool↔tag identity from the container image reference; the label is now ownership marking only.
5. gRPC 4 MiB default message limit — schedule dispatches at 4.66 MB died with RESOURCE_EXHAUSTED; this had silently broken the schedule pipeline since ~June 1. v0.12.3 raises all gRPC surfaces (worker channel, orchestrator server, web query client) to 64 MiB.

Additional operational notes:

• IAM propagation is slow — manual IAM grants took ~1–2 minutes to take effect. If a retry fails immediately after a grant, wait before digging deeper.
• Truly-fresh databases: migration 012 has a known clean-DB ordering issue (#113) — still open at the time of writing. A brand-new deployment running migrations from scratch should check that issue first; existing deployments that migrated incrementally are unaffected.
• Lame-duck pools after promotion are expected — when the realtime promotion converges both envs on one tag, the old tag's pool scales to zero and is deleted after POOL_LAMEDUCK_TTL_SECONDS (default 24h). Don't "clean it up" manually; re-referencing the old tag within the TTL is the instant-rollback path.