Initial Environment Setup: staging → production

This runbook sets up the two-environment topology — a mutable staging environment feeding a promote-only production environment — on a Continuous GTFS deployment. It covers both cases:

• Existing single-env deployment (pipelines already live in production): the bootstrap copies configuration into a new staging env and flips production to promote-only, with no consumer-visible gap.
• Fresh deployment: the same statements apply — the "copy from production" steps become plain inserts of your initial pipeline configuration into staging, and production rows are created empty of slots from the start.

The procedure was first executed against the Sound Transit deployment on 2026-06-11; the worked examples below (run IDs, tags) come from that live run.

Environment CRUD is self-service

Creating, editing, and deleting environments on a running deployment is an admin operation in the Control Console — the Environments admin page, or POST/PATCH/DELETE /api/v1/envs (environments.md §Environment Administration (CRUD)). This runbook is the one-time staging→production topology bootstrap/cutover — copying pipeline configuration, flipping production to promote-only, and verifying the first promotion — which is more than creating an environment. Use the admin path for routine environment changes.

Background reading: environments.md §Environment Mutability defines the promote-only model; pipeline-execution.md §Promotion Runs defines what promotion copies per feed type.

Prerequisites

All of these must be true before you start:

1. Framework v0.12.3 or later deployed (orchestrator + web). Four issues hit during the original cutover were fixed in v0.12.1–v0.12.3 (see Known gotchas); deployments at ≥ v0.12.3 get all of them for free. Most load-bearing: the 64 MiB gRPC message limits — schedule dispatches over 4 MiB fail outright on older versions.
2. Database migrations applied through 018:
   • 016-environments-promotes-from.sql — the promotes_from mutability primitive
   • 017-pipeline-runs-kind.sql — the kind = 'promotion' run discriminator
   • 018-pipelines-image-tag.sql — per-pipeline image_tag
3. Maintenance-window DB access tooling — the bootstrap SQL is applied directly against Cloud SQL. The sysadmin skill's maintenance-window script (temporarily enable public IP, apply, remove) is the sanctioned path; the same window used for migrations 016–018 can carry the bootstrap SQL.
4. A worker pool running and registered — GET /api/v1/workers shows at least one connected worker. If the deployment just cut over to orchestrator-managed pools, populate pipelines.image_tag first (see Populating image_tag): after the cutover removes the static tofu-managed pool, no pools exist until image_tag is populated.
5. The Control Console API key (CONTINUOUS_GTFS_API_KEY) for the trigger and promote calls.

Pre-checks

Before changing anything:

API=https://api.<tenant>.continuousgtfs.io
FEEDS=https://feeds.<tenant>.continuousgtfs.io

# Current consumers are healthy (note last-modified for later comparison)
curl -sI $FEEDS/production/schedule.zip | grep -i 'HTTP\|last-modified'
curl -sI $FEEDS/production/trip_updates.pb | grep -i 'HTTP\|last-modified'

# Workers connected, advertising the expected image tag
curl -s $API/api/v1/workers | jq

# Snapshot current production output asset versions (rollback reference)
curl -s "$API/api/v1/runs?environment_id=production&limit=5" | jq '.runs[] | {id, status, completed_at}'

A stale feed here is a finding, not a blocker to investigate later

During the Sound Transit bootstrap, this pre-check surfaced that /production/schedule.zip had been stale since ~June 1 — schedule dispatches had been silently dying on gRPC's 4 MiB default message limit (the feed had grown to 4.66 MB). Root-cause anything stale before proceeding; the bootstrap's first-promotion verification depends on a working staging schedule run.

Populating image_tag (and what the reconciler does)

The orchestrator's pool reconciler derives the desired worker-pool set from the distinct non-NULL image_tag values across enabled pipelines. Setting the column is what makes pools exist:

UPDATE pipelines SET image_tag = '<current pipeline image tag>'
WHERE environment_id = 'production';

Within one reconcile interval (default 60s) the orchestrator creates a Cloud Run worker pool named continuous-gtfs-pipeline-{first 8 chars of the sanitized tag} — dots and other characters invalid in Cloud Run resource IDs are mapped to dashes (tag v0.4.3 → pool continuous-gtfs-pipeline-v0-4-3). Pool↔tag identity is carried by the pool's container image reference, not the name, so sanitization collisions are harmless.

Confirm before proceeding:

• Orchestrator logs show Pool reconciler started and a pool-create action
• The pool's worker registers: GET /api/v1/workers lists a worker advertising the tag
• Allow ~30–60s for pool provisioning on top of the reconcile interval

The bootstrap SQL

Run the block below in a single maintenance window (and a single transaction). Each statement's rationale follows it. Pipeline IDs here are schedule and realtime; adjust if your agency repo names them differently.

BEGIN;

-- 1. Create the staging environment. promotes_from is left NULL,
--    which makes staging MUTABLE — direct triggers, config edits,
--    and pushes are accepted. Staging is where all changes land first.
INSERT INTO environments (id, display_name)
VALUES ('staging', 'Staging');

-- 2. Copy production's pipeline rows to staging. COPY, not move:
--    production keeps its rows because they are the promotion targets —
--    schedule promotion needs a production pipeline row to record the
--    promotion run against, and realtime promotion needs a production
--    row to copy configuration ONTO. The copy carries image_tag, so
--    staging starts on the same code version production is running.
INSERT INTO pipelines (environment_id, id, display_name, feed_type,
                       enabled, passthrough_on_failure, disabled_steps, image_tag)
SELECT 'staging', id, display_name, feed_type,
       enabled, passthrough_on_failure, disabled_steps, image_tag
FROM pipelines
WHERE environment_id = 'production';

-- 3. Copy the slot bindings to staging. With slots bound, the
--    orchestrator's fetcher-driven dispatch starts running staging
--    pipelines automatically — staging auto-publishes to /staging/...
--    CDN paths from its first run.
INSERT INTO pipeline_slots (environment_id, pipeline_id, slot_name,
                            asset_key, debounce_ms)
SELECT 'staging', pipeline_id, slot_name, asset_key, debounce_ms
FROM pipeline_slots
WHERE environment_id = 'production';

-- 4. De-slot production's SCHEDULE pipeline — and only schedule.
--    Schedule promotion copies BYTES (the validated output bundle), so
--    production's schedule output must come exclusively from promotion;
--    leaving slots bound would let fetcher-driven dispatch keep
--    auto-publishing to /production/schedule.zip, racing promotions.
--    The promote-only API gate doesn't cover this: automatic dispatch
--    is not a direct API mutation. Removing the slots is what actually
--    stops direct production schedule output.
--
--    Production REALTIME keeps its slots and keeps running: realtime
--    promotion copies CONFIGURATION, not bytes — the production RT
--    pipeline must keep executing continuously to produce feeds at all.
DELETE FROM pipeline_slots
WHERE environment_id = 'production' AND pipeline_id = 'schedule';

-- 5. Flip production to promote-only. From this statement on, every
--    direct-mutation API path against production (trigger, pipeline
--    config edit, slot edit, output-asset push) rejects with HTTP 409;
--    production state changes only via promotion from staging.
UPDATE environments
SET promotes_from = 'staging'
WHERE id = 'production';

COMMIT;

After commit, until the first schedule promotion lands, /production/schedule.zip keeps serving the last auto-published bytes — content is continuous for consumers, there's just no fresh write path yet. Proceed to the first promotion promptly.

Topology decision table

What each environment keeps after the bootstrap, and why:

	pipelines row	pipeline_slots	Runs directly?	Why
staging / schedule	yes	yes	yes — fetcher-driven + manual triggers	The validation surface. Output auto-publishes to /staging/schedule.zip
staging / realtime	yes	yes	yes — continuous ticks	The validation surface. Output auto-publishes to /staging/*.pb
production / schedule	yes (kept)	no (deleted)	no — promotion-fed only	The row is the promotion target (promotion runs are recorded against it). No slots → no fetcher-driven dispatch → /production/schedule.zip changes only via promotion
production / realtime	yes (kept)	yes (kept)	yes — keeps running continuously	RT promotion copies config, not bytes. The production RT pipeline must keep executing to produce feeds; automatic dispatch is not a direct mutation, so the promote-only gate correctly doesn't block it

Staged code rollout (how a new version reaches staging)

Pointing staging at a new pipeline image version is a one-column write:

UPDATE pipelines SET image_tag = '<new tag>' WHERE environment_id = 'staging';

The reconciler notices the new distinct tag and creates a second pool alongside the existing one — environments pinned to different tags get distinct pools; environments converging on one tag share a single pool. Staging runs dispatch to the worker advertising the new tag; production runs keep dispatching to the old one until a realtime promotion copies the tag forward.

Worked example from the Sound Transit bootstrap: staging image_tag was set to v0.4.3 while production sat on v0.4.0. The reconciler created continuous-gtfs-pipeline-v0-4-3 alongside continuous-gtfs-pipeline-v0-4-0, and staging schedule run 6955f9b8-0409-43eb-9339-bb1d408c784c completed on the new worker. After the realtime promotion copied v0.4.3 to production, v0.4.0 became unreferenced and its pool entered the lame-duck window (scaled to zero, deleted after the TTL).

First run and first promotions

All mutations require the API key. Triggers default to the production environment when environment_id is omitted — always pass it explicitly now that production is promote-only.

1. Trigger a staging schedule run

curl -s -X POST $API/api/v1/pipelines/schedule/trigger \
  -H "X-API-Key: $CONTINUOUS_GTFS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"environment_id": "staging", "reason": "env-bootstrap first staging run"}'

Wait for completion and capture the run id:

curl -s "$API/api/v1/runs?environment_id=staging&pipeline_id=schedule&limit=1" \
  | jq '.runs[0] | {id, status, completed_at}'

Confirm /staging/schedule.zip now serves the run's output (fresh last-modified).

2. Promote the schedule run to production

Schedule promotion takes an explicit source_run_id — you promote a specific validated run, not "latest":

curl -s -X POST $API/api/v1/envs/production/pipelines/schedule/promote-schedule \
  -H "X-API-Key: $CONTINUOUS_GTFS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"source_run_id": "<staging run id from step 1>"}'

Returns 202 with the promotion run id. Verify:

• The promotion run completes: GET /api/v1/runs?environment_id=production&pipeline_id=schedule&limit=1 shows a kind: promotion run with status: completed
• /production/schedule.zip last-modified advances; content hash matches the staging output it sourced

3. Promote realtime configuration to production

Realtime promotion takes no body — the source is always the env named by production's promotes_from, and it copies the current staging config (ops controls, image_tag, slot bindings) verbatim:

curl -s -X POST $API/api/v1/envs/production/pipelines/realtime/promote-realtime \
  -H "X-API-Key: $CONTINUOUS_GTFS_API_KEY"

Verify the promotion run record's result shows the copied source_pipeline_row and source_slots, and that subsequent production RT runs complete on the promoted config (check GET /api/v1/runs?environment_id=production&pipeline_id=realtime — fresh kind: transform runs completing).

4. Verify the promote-only gate

A direct trigger against production must now reject:

curl -s -X POST $API/api/v1/pipelines/schedule/trigger \
  -H "X-API-Key: $CONTINUOUS_GTFS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"environment_id": "production"}'

Expected — HTTP 409 with:

{"error": "env_promote_only", "env": "production", "promotes_from": "staging", "attempted": "trigger_pipeline"}

5. Verify the full topology

curl -s $API/api/v1/envs | jq                  # both envs listed
curl -s $API/api/v1/envs/production | jq       # promotes_from: "staging"

curl -sI $FEEDS/staging/schedule.zip | grep -i 'HTTP\|last-modified'
curl -sI $FEEDS/production/schedule.zip | grep -i 'HTTP\|last-modified'
curl -sI $FEEDS/production/trip_updates.pb | grep -i 'HTTP\|last-modified'   # RT fresh
curl -sI $FEEDS/schedule.zip | grep -i 'HTTP\|last-modified'                 # root-URL rewrite → /production/ intact

Rollback

If the bootstrap goes sideways (especially after step 5 of the SQL but before the first schedule promotion, when production has no fresh write path):

BEGIN;

-- Production accepts direct mutations again; fetcher-driven dispatch
-- resumes wherever slots are bound.
UPDATE environments SET promotes_from = NULL WHERE id = 'production';

-- Re-seed production schedule's slots from the staging copies.
INSERT INTO pipeline_slots (environment_id, pipeline_id, slot_name,
                            asset_key, debounce_ms)
SELECT 'production', pipeline_id, slot_name, asset_key, debounce_ms
FROM pipeline_slots
WHERE environment_id = 'staging' AND pipeline_id = 'schedule';

COMMIT;

This restores the pre-bootstrap behavior exactly: production schedule auto-publishes on the next upstream fetch. The staging env and its rows can be left in place (harmless — it just keeps publishing to /staging/...) or cleaned up separately. Historical production runs and output asset versions were never touched by the bootstrap, so nothing else needs restoring.

Known gotchas (from the first live run)

The 2026-06-11 Sound Transit bootstrap hit five live issues. All five are fixed in framework v0.12.1–v0.12.3 — a fresh deployment at ≥ v0.12.3 will not hit them — but they're recorded here because the symptoms are non-obvious:

1. Deploy SA missing roles/iam.roleAdmin — the cutover tofu apply (which creates the orchestrator's custom pool-management role) failed until the role was granted manually. Now part of documented deploy-SA requirements.
2. Orchestrator SA missing artifactregistry.reader — pool creation succeeded but pool instances couldn't pull the pipeline image. Manually granted during the run; the grant has since moved into the platform module as a project-level member.
3. Pool IDs reject dotted tags — Cloud Run worker_pool_id forbids dots, so release-style tags (v0.4.3) crashed pool creation. Fixed by name sanitization (dots → dashes).
4. Label values reject dotted tags — same character restriction on GCP label values broke the image-tag label. Fixed by deriving pool↔tag identity from the container image reference; the label is now ownership marking only.
5. gRPC 4 MiB default message limit — schedule dispatches at 4.66 MB died with RESOURCE_EXHAUSTED; this had silently broken the schedule pipeline since ~June 1. v0.12.3 raises all gRPC surfaces (worker channel, orchestrator server, web query client) to 64 MiB.

Additional operational notes:

• IAM propagation is slow — manual IAM grants took ~1–2 minutes to take effect. If a retry fails immediately after a grant, wait before digging deeper.
• Truly-fresh databases: migration 012 has a known clean-DB ordering issue (#113) — still open at the time of writing. A brand-new deployment running migrations from scratch should check that issue first; existing deployments that migrated incrementally are unaffected.
• Lame-duck pools after promotion are expected — when the realtime promotion converges both envs on one tag, the old tag's pool scales to zero and is deleted after POOL_LAMEDUCK_TTL_SECONDS (default 24h). Don't "clean it up" manually; re-referencing the old tag within the TTL is the instant-rollback path.