Skip to content

How Should Agencies Manage Transforms?

March 20, 2026 — Experiments: config-management + pipeline-dag

Updated

The transform framework described in this experiment was built into a production package in 010: Building the Transform Framework. Sound Transit's 4-environment config model was consolidated into a single canonical pipeline — the per-environment split was an artifact of their old system.

The Question

Sound Transit needs to control how their GTFS data gets transformed — renaming stops, filtering routes, updating metadata. We want them to be able to manage this themselves without touching our infrastructure. But we also need the system to be safe (bad config can't break production), version-controlled, and visible in the Control Console.

There's also a hosting constraint: the pipeline infrastructure needs to execute multiple pipeline versions simultaneously — a tagged release against production, a different tag against staging, and a branch build or live worktree against a dev environment. Whatever model we pick needs to support all of these without separate infrastructure per version.

What We Tried

First, we tested three separate configuration approaches:

  • Git-Tag deploy — agency writes Python, tags a version, selects which tag runs in each environment
  • Git-Branch deploy — each environment tracks a git branch, merging triggers a deploy
  • YAML-only — agency edits a YAML file with parameterized operations, no custom code

Then we threw all three away and built a unified model that combines the best parts.

What We Found

  1. YAML is safe but limiting — agency staff can only use pre-built operations. The moment they need something custom, they're stuck.

  2. Git-Branch deploy has no audit trail — you can't answer "what exactly was running in production last Tuesday?" because the branch HEAD keeps moving.

  3. The real insight: builtins and custom code don't need to be different systems. A builtin operation is just a Python class you instantiate. Custom code is just a Python function with a decorator. The framework treats them identically.

What It Looks Like

Here's what a Sound Transit engineer writes to set up a schedule pipeline:

# pipelines/schedule/feed_info.py

from continuous_gtfs.builtins.schedule import UpdateFeedInfo

update_feed_info = UpdateFeedInfo(
    publisher_name="Sound Transit",
    publisher_url="https://soundtransit.org",
    feed_lang="en",
)

That's it. The framework discovers this file, sees it's a Step, and adds it to the pipeline DAG. Here's a second file that mixes a builtin (with a dependency on the first step) and custom code:

# pipelines/schedule/stop_cleanup.py

from continuous_gtfs import step
from continuous_gtfs.builtins.schedule import MergeStops
from .feed_info import update_feed_info

# Builtin with a DAG dependency — runs after feed info is updated
merge_stops = MergeStops(
    similarity_threshold=0.9,
    after=[update_feed_info],
)

# Custom code — runs after merge, works on files the builtin doesn't cover
@step(files=["stops.txt", "stop_times.txt"], after=[merge_stops])
def remove_unused_stops(ctx):
    """Remove stops not referenced by any stop_time."""
    used = ctx.datasets["stop_times.txt"]["stop_id"].unique()
    ctx.datasets["stops.txt"] = ctx.datasets["stops.txt"].filter(
        pl.col("stop_id").is_in(used)
    )

The after=[update_feed_info] on merge_stops is a Python object reference — if someone renames or deletes update_feed_info, this file fails at import time, not at runtime. The DAG is built from these references automatically.

Both show up the same way in the Control Console — as nodes in a visual DAG:

flowchart LR A["Update Feed Info\n(builtin)\nfeed_info.txt"] --> B["Merge Stops\n(builtin)\nstops.txt"] B --> C["Remove Unused Stops\n(custom)\nstops.txt, stop_times.txt"]

No config file declares this pipeline. The folder IS the pipeline. Drop a .py file in, it appears in the DAG. Remove it, it disappears.

The Decision

Unified Step model. Pipeline = folder. Builtins and custom code are the same thing. Version control via git tags, instant rollback by pointing an environment at a previous tag. Infrastructure config (URLs, secrets, scaling) stays separate and locked down — agency staff never touch it.

What This Means

  • Sound Transit staff manage transforms by editing Python files in their own git repo
  • Common operations (rename, filter, merge) are one-line builtin instantiations
  • Custom logic is a decorated function — same visibility in the UI, same version control
  • We can optimize builtins behind the scenes without changing the agency's code
  • Switching between versions is instant — no rebuild, no redeploy
  • Multiple versions run simultaneously — production runs tag v1.2.0, staging runs v1.3.0-rc1, and a developer tests from a branch or local worktree. Same infrastructure, same framework, different pipeline folders loaded at different refs.

Open Questions

  • How do we handle the RT pipeline DAG? The schedule pipeline operates on DataFrames (Polars). The RT pipeline operates on protobuf messages. The Step model works for both, but the builtins and context shape are different. Should RT pipelines use the same ctx.datasets pattern, or do protobuf entities need a different interface?
  • What does the CI/CD flow look like for pipeline versions? Tag → build → artifact → deploy-to-environment is the conceptual flow, but the specifics (where does the artifact live? how does the pipeline service pick it up?) need design work.