Pipelines & Steps

A pipeline is an ordered collection of SQL transformations. A step is one transformation. BoltPipeline figures out the rest.

The basics

What is a pipeline?

A pipeline in BoltPipeline is a named collection of SQL transformations that move and shape data from source tables to target tables. Each pipeline has its own lifecycle — it is planned, certified, and operated as a single unit.

Pipelines are the unit of certification and promotion. When you certify a pipeline, all its steps are validated together. When you promote a pipeline to Production, all its steps run together. This means everything that logically belongs together should be in the same pipeline.

Good candidates for a single pipeline

All transformations for a Customer 360 model

An entire Finance reporting layer (Bronze → Silver → Gold)

A complete set of marketing attribution steps

All Supply Chain dimension and fact tables

Building blocks

What is a step?

A step is one SQL transformation — a single SELECT statement that reads from one or more source tables and writes into one target table. Each step has a name, a target table, and an SCD type that governs how BoltPipeline handles row updates.

Steps are the smallest unit of work in BoltPipeline. Each step is independently validated during certification and independently monitored for drift during operation.

Step example — dim_customerSCD Type 2

SELECT
  c.customer_id,
  c.full_name,
  c.email,
  c.region,
  c.customer_segment
FROM raw.crm_customers c
WHERE c.is_active = true

Target: analytics.dim_customer · SCD: Type 2 (full history) · Keys: customer_id

Execution order

Dependencies and execution order

When one step reads from a table that another step writes to, BoltPipeline automatically detects this dependency and ensures the upstream step runs first. You do not need to manually specify execution order — BoltPipeline builds the dependency graph from the table references in your SQL.

Example: a three-step pipeline

Step 1

bronze → silver.dim_customer

Step 2

bronze → silver.fact_orders

Step 3

silver.* → gold.customer_summary

Step 3 reads from tables produced by Steps 1 and 2. BoltPipeline detects this and always runs Steps 1 and 2 before Step 3 — even if you add them in a different order.

Circular dependencies

If Step A reads from a table written by Step B, and Step B reads from a table written by Step A, BoltPipeline detects the cycle and fails certification with a clear error. Circular dependencies are not allowed — your pipeline must form a directed acyclic graph (DAG).

Sizing guidelines

How many steps per pipeline?

BoltPipeline supports up to 50 steps per pipeline. In most cases, 50 well-designed steps can represent an entire analytical domain — all your finance transformations, your complete customer 360 pipeline, or a full supply chain data layer.

Design for cohesion

›Group steps that logically belong together (same domain, same data source)
›Steps that share upstream tables are natural pipeline mates
›Up to 50 steps before creating a second pipeline

Avoid fragmentation

›One pipeline per SQL file adds cost with no governance benefit
›Micro-pipelines (1–2 steps) are hard to manage at scale
›Splitting by team rather than domain leads to duplicated logic

Getting your SQL in

Three ways to build your plan

Available

Build in the console

Write your SQL directly in the BoltPipeline step editor. Add steps one at a time, test your logic, and certify when ready. Best for iterative design.

Available

Bring your own SQL

Paste in SQL you have already written or import SQL files from your existing scripts, dbt models, or version control. BoltPipeline accepts standard SQL.

Coming soon

AI-assisted design

Describe what you want in plain language. BoltPipeline uses your Enterprise Model metadata to generate SQL grounded in your actual schema — not hallucinated table names.